Investigating disagreement in the scientific literature

Abstract

Disagreement is essential to scientific progress but the extent of disagreement in science, its evolution over time, and the fields in which it happens remain poorly understood. Here we report the development of an approach based on cue phrases that can identify instances of disagreement in scientific articles. These instances are sentences in an article that cite other articles. Applying this approach to a collection of more than four million English-language articles published between 2000 and 2015 period, we determine the level of disagreement in five broad fields within the scientific literature (biomedical and health sciences; life and earth sciences; mathematics and computer science; physical sciences and engineering; and social sciences and humanities) and 817 meso-level fields. Overall, the level of disagreement is highest in the social sciences and humanities, and lowest in mathematics and computer science. However, there is considerable heterogeneity across the meso-level fields, revealing the importance of local disciplinary cultures and the epistemic characteristics of disagreement. Analysis at the level of individual articles reveals notable episodes of disagreement in science, and illustrates how methodological artifacts can confound analyses of scientific texts.

Keywords: citation analysis; disagreement; meta-research; metascience; natural language processing; none.

Figure 1.. Agreement and validity of different combinations of signal term and filter term.

Measures calculated from 50 randomly-sampled citances for each combination of signal term (vertical axis) and filter term (horizontal axis), annotated as valid or invalid instances of disagreement by two independent coders. (a) Percentage agreement, or the proportion of citances for which coders independently agreed on the label. (b) Percentage validity, or the proportion of citances which both coders labeled as valid. Averages for the various signal terms are shown in the left-most column; averages for the various filter terms are shown in the bottom row. (c) Percentage agreement (blue circles) and validity (red diamonds) of each signal/filter term combination, ordered from highest percent validity (top) to lowest percent validity (bottom). Numbers on the right are the total number of citances returned by querying using the signal/filter term combination, and are colored according to their log-transformed value. (d) Log-transformed count of citances returned by each query combination, colored by the (log-transformed) number of citances. Citance counts are non-exclusive, meaning that citances of the form debat* + studies will also be counted towards debat* _standalone_.

Figure 1—figure supplement 1.. Distribution of citances returned by signal/filter term queries.

Callouts (I, II, …, VIII) map to examples in Table S3. a. Distribution of all disagreement citances appearing in papers across five fields: Biomedical and Health Sciences, Life and Earth Sciences, Physical Sciences and Engineering, Social Sciences and Humanities, and Math and Computer Science. b–d. Percentage change between the actual number of citances per field and signal/filter term combination compared to the expected given the disciplinary distribution (from a). The red line corresponds to 0 percent increase between the actual and expected. White dots indicate that the citances for that signal/filter term are under-represented (lower than expected, ratio less than zero), whereas black dots indicate that citances are over-represented (more than expected). Shown aggregated across signal terms (b), filter terms (c), and for all signal/filter term combinations (d).

Figure 2.. Disagreement reflects a hierarchy of fields.

(a) Percent of all citances in each field that contain signals of disagreement, meaning they were returned by one of the 23 queries with validity of 80% or higher. Fields marked by lower consensus, such as in Soc & Hum, had a greater proportion of disagreement. (b) Percent of disagreement by field and over time, showing little change overall, but some changes by field. Text indicates the average percentage-point change per-year by field.

Figure 2—figure supplement 1.. Percent of all citances returned by each of the 23 queries with validity over 80%.

Each panel corresponds to the signal phrase, and lines within each panel to filter phrases.

Figure 3.. Heterogeneity in disagreement across meso-fields.

Fine-grained view across 817 meso-level fields, each a cluster of publications grouped and positioned based on their citation links derived from the Web of Science database (see Materials and methods), 2000–2015. The area of each point is proportional to the number of disagreement citances in that field. Overlapping points are an artifact of their position and size, and bear no additional meaning. Color maps to the log ratio of the share of disagreement citances given the mean share across all fields, truncated at 4 x greater and 4 x lower than the mean. Soc & Hum tends to have a greater proportion of disagreement citances, and Math & Comp the least. Other panels show the same data, but highlight the meso-fields in each high-level field. Meso-fields of interest are highlighted, and labels show a selection of journals in which papers in each field are published. Journals listed in labels are representative of each meso-field in the Web of Science, and is not limited to those represented in the Elsevier ScienceDirect data. An interactive version of this visualization is available online at https://tinyurl.com/disagreement-meso-fields.

Figure 3—figure supplement 1.. On average, older papers are less likely to receive a disagreement citance, though this trend does not hold for the “hard” sciences.

Percentage of disagreement citances by the relative age of the citing to the cited paper, in years, and high-level field, for papers published between 2000 and 2015. Intensity of color corresponds to the age category of the cited paper.

Figure 3—figure supplement 2.. Distribution of citances by their position in the text of the manuscript, and by field.

Shown for all citances (solid line) and disagreement citances (dotted line). For example, about 15% of disagreement citances in Physical Sciences and Engineering appear in the first 0%–5% of the sentences in documents.

Figure 3—figure supplement 3.. Little difference in disagreement between men and women.

Percentage of disagreement citances by gender of the citing-paper author, their authorship position (first or last), and the high-level field. Numbers above each bar corresponds to the ratio difference between the percentage of disagreement between women and men. The number below each bar corresponds to the number of disagreement citances. we infer a gender for the first and last authors of papers with a disagreement citance published after 2008, determined based on the author’s first name as in past work (Larivière et al., 2013).

Figure 3—figure supplement 4.. Authors disagree less when citing their own work.

Percentage of disagreement citances among instances of non-self and self-citation, 2000–2015. A citance is defined as a self-citation when the citing and cited paper have at least one name in common. Results are shown by field. Numbers below each bar are the number of disagreement citances. Overall, disagreement is 2.4 times more common for non-self citation than for self-citation, with variance between major fields.

Figure 4.. Full research articles with a disagreement citance are cited more.

The y-axis shows the difference in average citation counts for papers containing at least one disagreement citance, and for papers without. Positive values indicate that publications with disagreement received more citations than those without. Values are shown for the population of publications in each year following publication (x-axis). Shown here for only articles labeled in the Web of Science database as full research articles.