Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2023 Apr 10;1(2):e12009.
doi: 10.1002/cesm.12009. eCollection 2023 Apr.

Systematic reviewers' perspectives on replication of systematic reviews: A survey

Phi-Yen Nguyen  1 , Joanne E McKenzie  1 , Daniel G Hamilton  2   3 , David Moher  4   5 , Peter Tugwell  5   6   7 , Fiona M Fidler  2   8 , Neal R Haddaway  9   10 , Julian P T Higgins  11 , Raju Kanukula  1 , Sathya Karunananthan  6   12 , Lara J Maxwell  13 , Steve McDonald  1 , Shinichi Nakagawa  14 , David Nunan  15 , Vivian A Welch  5   6 , Matthew J Page  1
Affiliations

Systematic reviewers' perspectives on replication of systematic reviews: A survey

Phi-Yen Nguyen et al. Cochrane Evid Synth Methods. .

Abstract

Background: Replication is essential to the scientific method. It is unclear what systematic reviewers think about the replication of systematic reviews (SRs). Therefore, we aimed to explore systematic reviewers' perspectives on (a) the definition and importance of SR replication; (b) incentives and barriers to conducting SR replication; and (c) a checklist to guide when to replicate an SR.

Methods: We searched PubMed for SRs published from January to April 2021, from which we randomly allocated 50% to this survey and 50% to another survey on data sharing in SRs. We sent an electronic survey to authors of these SRs (n = 4669) using Qualtrics. Quantitative responses were summarized using frequency analysis. Free-text answers were coded using an inductive approach.

Results: The response rate was 9% (n = 409). Most participants considered "replication of SRs" as redoing an SR (68%) or reanalyzing originally collected data (61%), using the same or similar methods. Participants also considered updating an SR, either one's own (42%) or others (43%), equivalent to replication. Most participants agreed that replication of SRs is important (89%). Although 54% of participants reported having conducted a replication of a SR, only 22% have published a replication within 5 years. Those who published a replication (n = 89) often found their replication supported (47%) or expanded the generalizability of the original review (51%). The most common perceived barriers to replicating SRs were difficulty publishing (75%), less prestige (65%), fewer citations (56%), and less impact on career advancement (55%) compared to conducting an original SR. A checklist to assess the need for replication was deemed useful (79%) and easy to apply in practice (69%) by participants.

Conclusion: Reviewers have various perceptions of what constitutes a replication of SRs. Reviewers see replication as important and valuable but perceive several barriers to conducting replications. Institutional support should be better communicated to reviewers to address these perceptions.

PubMed Disclaimer

Conflict of interest statement

The authors declare no conflict of interest.

Figures

Figure 1
Figure 1
Flowchart of recruitment and data collection.
Figure 2
Figure 2
Participants' perspectives on the definition of replication.
Figure 3
Figure 3
Factors that encourage or discourage replication of systematic reviews.
Figure 4
Figure 4
Evaluation of a checklist to decide when to replicate a systematic review by Tugwell et al. [17].
See this image and copyright information in PMC

References

    1. Bouter LM, Riet G. Replication research series‐paper 2: empirical research must be replicated before its findings can be trusted. J Clin Epidemiol. 2021;129:188‐190. - PubMed
    1. Haddaway NR, Pullin AS. The policy role of systematic reviews: past, present and future. Springer Sci Rev. 2014;2:179‐183.
    1. David Tovey D, Karla S‐W. Cochrane's Editor in Chief Responds to BMJ EBM Article Criticizing HPV Review. Cochrane Collab; 2018.
    1. Low J, Ross JS, Ritchie JD, et al. Comparison of two independent systematic reviews of trials of recombinant human bone morphogenetic protein‐2 (rhBMP‐2): the Yale Open Data Access Medtronic Project. Syst Rev. 2017;6:28. - PMC - PubMed
    1. Nosek BA, Hardwicke TE, Moshontz H, et al. Replicability, robustness, and reproducibility in psychological science. Annu Rev Psychol. 2022;73:719‐748. - PubMed

LinkOut - more resources