Evolution of international collaborative research efforts to develop non-Cochrane systematic reviews

Abstract

This research-on-research study describes efforts to develop non-Cochrane systematic reviews (SRs) by analyzing demographical and time-course collaborations between international institutions using protocols registered in the International Prospective Register of Systematic Reviews (PROSPERO) or published in scientific journals. We have published an a priori protocol to develop this study. Protocols published in scientific journals were searched using the MEDLINE and Embase databases; the query terms "Systematic review" [Title] AND "protocol" [Title] were searched from February 2011 to December 2017. Protocols registered at PROSPERO during the same period were obtained by web scraping all non-Cochrane records with a Python script. After excluding protocols that had a fulfillment or duplication rate of less than 90%, they were classified as published "only in PROSPERO", "only in journals", or in "journals and PROSPERO". Results of data and metadata extraction using text mining processes were curated by two reviewers. These Datasets and R scripts are freely available to facilitate reproducibility. We obtained 20,814 protocols of non-Cochrane SRs. While "unique protocols" by reviewers' institutions from 60 countries were the most frequent, a median of 6 (2-150) institutions from 130 different countries were involved in the preparation of "collaborative protocols". The highest Ranked countries involved in overall protocol production were the UK, the U.S., Australia, Brazil, China, Canada, the Netherlands, Germany, and Italy. Most protocols were registered only in PROSPERO. However, the number of protocols published in scientific journals (924) or in both PROSPERO and journals (807) has increased over the last three years. Syst Rev and BMJ Open published more than half of the total protocols. While the more productive countries were involved in "unique" and "collaborative protocols", less productive countries only participated in "collaborative protocols" that were mainly published in PROSPERO. Our results suggest that, although most countries were involved in solitary production of protocols for non-Cochrane SRs during the study period, it would be useful to develop new strategies to promote international collaborations, especially with less productive countries.

Fig 1. PRISMA workflow for searching for PROSPERO records and protocols published in scientific journals with regard to non-Cochrane systematic reviews.

Fig 2. This panel represents the main features of included protocols.

(a) The frequency of protocols published from 2011 to 2017 compared to those published “only in journal”, “only in PROSPERO”, and “journal and PROSPERO”. (b) Venn diagram of the number of protocols published “only in journal” (coral), “only in PROSPERO” (green), and “journal and PROSPERO” (blue)—their intersection. (c) Map representation of the number of protocols produced categorized by country (as a proxy for the reviewer’s affiliation country). Colors represent the levels of productivity, as defined by the quartiles of a newly recoded variable [abs(log2(country.count / all.countries.count))] (red, very high; yellow, high; green, medium; blue, low). (d), (e), and (f) represent the word clouds of “unique countries” (d), “collaborative countries” (e), and journals (f). Text size and centering is proportional to the associated number of protocols. Colors have been randomly assigned. (g), (h), and (i) represent column plots of the “unique countries” (g), “collaborative countries” (h), and journals (i), which are ranked based on the total number of protocols.

Fig 3. This panel represents the frequency and time-course changes of SR protocol publication by journals.

(a) Frequency of protocols published from 2011 to 2017 by journals compared the protocols published “only in journal” with protocols published in “journal and PROSPERO”. (b) Magnified version of the plot (a) centered on the top 10 most published journals. (c) Evolution of “only in journal” vs. in “journal and PROSPERO” protocol publications from 2011 to 2017 compared to “BMJ Open” and “Systematic Reviews” journals. SR: Systematic Review. SR: Systematic Review.

Fig 4. This panel represents the frequency and time-course changes of SR protocol publications categorized by countries.

(a) The frequency of protocols published from 2011 to 2017 categorized by country comparing protocols published “only in a journal” with protocols published in “journal and PROSPERO”. (b) Magnified version of the plot (a) centered on the top 10 most productive countries comparing protocols published “only in a journal”, “only at PROSPERO”, and in both “journal and PROSPERO”. (c) Evolution of “only journal” vs. “journal and PROSPERO” protocols publications from 2011 to 2017 comparing the top 10 most productive countries.

Fig 5. Rank discrepancies between two ordered lists of reviewers’ affiliation countries.

The “Unique protocols” column displays a descending list of reviewers’ affiliation countries, which produced protocols in which all reviewers’ institutions belonged to a unique country. The “Collaborative protocols” column displays a ranked list of reviewers’ affiliation countries that collaborated with other reviewers’ affiliation countries to produce protocols for SRs. Arrows connect the same country from the first list to the second list. Countries represented only in one of the lists are not connected to/by any arrow. The countries were sub-grouped (Q1:Q4) by cutting through 25%, 50%, and 75% of the total number of countries in each list. When comparing the “Unique protocols” and “Collaborative protocols” lists, the country position was considered as being modified if the edge connected two different subgroups (i.e., Q1→Q3). Direction of the change defines ′upgrading′ (Q2 → Q1, Q3 → Q1, Q4 → Q1, Q3 → Q2, Q4 → Q3) or ′downgrading′ the rank position of any country (Q1 → Q2, Q1 → Q3, Q1 → Q4, Q2 → Q3, Q2 → Q4, Q3 → Q4).

Fig 6. Analysis of protocol publication patterns categorized based on the most productive countries.

Countries are listed in a descendent order based on their “Unique protocol” productivity. The points represent a hallmark in every country’s history of protocol publication: first publication of a protocol in “only in journal” (red dot), “only in PROSPERO” (green triangle), and in “journal and PROSPERO” (blue square). Arrows connect two (by a dotted line) or more (by a full line) hallmarks to emphasize how much time a country requires to adopt a new publication process.