. 2022 May 10:377:e069155.

doi: 10.1136/bmj-2021-069155.

Validity of data extraction in evidence synthesis practice of adverse events: reproducibility study

Chang Xu^{1

2

3}, Tianqi Yu⁴, Luis Furuya-Kanamori⁵, Lifeng Lin⁶, Liliane Zorzela⁷, Xiaoqin Zhou⁸, Hanming Dai⁸, Yoon Loke⁹, Sunita Vohra^{10

11}

Affiliations

¹ Key Laboratory of Population Health Across-life Cycle, Ministry of Education of the People's Republic of China, Anhui Medical University, Anhui, China.
² Anhui Provincial Key Laboratory of Population Health and Aristogenics, Anhui Medical University, Anhui, China.
³ School of Public Health, Anhui Medical University, Anhui, China.
⁴ Chinese Evidence-based Medicine Centre, West China Hospital, Sichuan University, Chengdu, China.
⁵ UQ Centre for Clinical Research, Faculty of Medicine, University of Queensland, Brisbane, QLD, Australia.
⁶ Department of Statistics, Florida State University, Tallahassee, FL, USA.
⁷ Department of Pediatrics, Faculty of Medicine and Dentistry, University of Alberta, Edmonton, Alberta, AB, Canada.
⁸ Mental Health Centre, West China Hospital of Sichuan University, Chengdu, China.
⁹ Norwich Medical School, University of East Anglia, Norwich, UK.
¹⁰ Department of Pediatrics, Faculty of Medicine and Dentistry, University of Alberta, Edmonton, Alberta, AB, Canada svohra@ualberta.ca.
¹¹ Departments of Psychiatry, Faculty of Medicine and Dentistry, University of Alberta, Edmonton, Alberta, AB, Canada.

PMID: 35537752
PMCID: PMC9086856
DOI: 10.1136/bmj-2021-069155

Validity of data extraction in evidence synthesis practice of adverse events: reproducibility study

Chang Xu et al. BMJ. 2022.

. 2022 May 10:377:e069155.

doi: 10.1136/bmj-2021-069155.

Authors

Chang Xu^{1

2

3}, Tianqi Yu⁴, Luis Furuya-Kanamori⁵, Lifeng Lin⁶, Liliane Zorzela⁷, Xiaoqin Zhou⁸, Hanming Dai⁸, Yoon Loke⁹, Sunita Vohra^{10

11}

Affiliations

¹ Key Laboratory of Population Health Across-life Cycle, Ministry of Education of the People's Republic of China, Anhui Medical University, Anhui, China.
² Anhui Provincial Key Laboratory of Population Health and Aristogenics, Anhui Medical University, Anhui, China.
³ School of Public Health, Anhui Medical University, Anhui, China.
⁴ Chinese Evidence-based Medicine Centre, West China Hospital, Sichuan University, Chengdu, China.
⁵ UQ Centre for Clinical Research, Faculty of Medicine, University of Queensland, Brisbane, QLD, Australia.
⁶ Department of Statistics, Florida State University, Tallahassee, FL, USA.
⁷ Department of Pediatrics, Faculty of Medicine and Dentistry, University of Alberta, Edmonton, Alberta, AB, Canada.
⁸ Mental Health Centre, West China Hospital of Sichuan University, Chengdu, China.
⁹ Norwich Medical School, University of East Anglia, Norwich, UK.
¹⁰ Department of Pediatrics, Faculty of Medicine and Dentistry, University of Alberta, Edmonton, Alberta, AB, Canada svohra@ualberta.ca.
¹¹ Departments of Psychiatry, Faculty of Medicine and Dentistry, University of Alberta, Edmonton, Alberta, AB, Canada.

PMID: 35537752
PMCID: PMC9086856
DOI: 10.1136/bmj-2021-069155

Abstract

Objectives: To investigate the validity of data extraction in systematic reviews of adverse events, the effect of data extraction errors on the results, and to develop a classification framework for data extraction errors to support further methodological research.

Design: Reproducibility study.

Data sources: PubMed was searched for eligible systematic reviews published between 1 January 2015 and 1 January 2020. Metadata from the randomised controlled trials were extracted from the systematic reviews by four authors. The original data sources (eg, full text and ClinicalTrials.gov) were then referred to by the same authors to reproduce the data used in these meta-analyses.

Eligibility criteria for selecting studies: Systematic reviews were included when based on randomised controlled trials for healthcare interventions that reported safety as the exclusive outcome, with at least one pair meta-analysis that included five or more randomised controlled trials and with a 2×2 table of data for event counts and sample sizes in intervention and control arms available for each trial in the meta-analysis.

Main outcome measures: The primary outcome was data extraction errors summarised at three levels: study level, meta-analysis level, and systematic review level. The potential effect of such errors on the results was further investigated.

Results: 201 systematic reviews and 829 pairwise meta-analyses involving 10 386 randomised controlled trials were included. Data extraction could not be reproduced in 1762 (17.0%) of 10 386 trials. In 554 (66.8%) of 829 meta-analyses, at least one randomised controlled trial had data extraction errors; 171 (85.1%) of 201 systematic reviews had at least one meta-analysis with data extraction errors. The most common types of data extraction errors were numerical errors (49.2%, 867/1762) and ambiguous errors (29.9%, 526/1762), mainly caused by ambiguous definitions of the outcomes. These categories were followed by three others: zero assumption errors, misidentification, and mismatching errors. The impact of these errors were analysed on 288 meta-analyses. Data extraction errors led to 10 (3.5%) of 288 meta-analyses changing the direction of the effect and 19 (6.6%) of 288 meta-analyses changing the significance of the P value. Meta-analyses that had two or more different types of errors were more susceptible to these changes than those with only one type of error (for moderate changes, 11 (28.2%) of 39 v 26 (10.4%) 249, P=0.002; for large changes, 5 (12.8%) of 39 v 8 (3.2%) of 249, P=0.01).

Conclusion: Systematic reviews of adverse events potentially have serious issues in terms of the reproducibility of the data extraction, and these errors can mislead the conclusions. Implementation guidelines are urgently required to help authors of future systematic reviews improve the validity of data extraction.

PubMed Disclaimer

Conflict of interest statement

Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: support from the Australian National Health and Medical Research Council Fellowship, US National Institutes of Health, National Library of Medicine, and National Institute of Mental Health for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.

Figures

**Fig 1**
Flowchart for selection of articles. RCT=randomised controlled trial

**Fig 2**
Data extraction errors at the meta-analysis level. Bar plot is based on studies with data extraction errors (n=554). Error rate within a meta-analysis is calculated by the number of studies with data extraction errors against the total number of studies within a meta-analysis

**Fig 3**
Data extraction errors at the systematic review level. Bar plot is based on studies with data extraction errors (n=171). Error rate within a systematic review is calculated by the number of meta-analyses with data extraction errors against the total number of meta-analyses within a systematic review

**Fig 4**
Proportion of 1762 studies classified by five types of data extraction error

**Fig 5**
Impact of data extraction errors on results

See this image and copyright information in PMC

References

1. Baker M. 1,500 scientists lift the lid on reproducibility. Nature 2016;533:452-4. 10.1038/533452a - DOI - PubMed
1. Chandler J, Cumpston M, Thomas J, et al. . Introduction. In: Higgins JPT, Thomas J, Chandler J, et al., eds. Cochrane Handbook for Systematic Reviews of Interventions version 6.2 (updated February 2021). Cochrane, 2021, www.training.cochrane.org/handbook.
1. Mathes T, Klaßen P, Pieper D. Frequency of data extraction errors and methods to increase data extraction quality: a methodological review. BMC Med Res Methodol 2017;17:152. 10.1186/s12874-017-0431-4 - DOI - PMC - PubMed
1. Jones AP, Remmington T, Williamson PR, Ashby D, Smyth RL. High prevalence but low impact of data extraction and reporting errors were found in Cochrane systematic reviews. J Clin Epidemiol 2005;58:741-2. 10.1016/j.jclinepi.2004.11.024 - DOI - PubMed
1. Gøtzsche PC, Hróbjartsson A, Maric K, Tendal B. Data extraction errors in meta-analyses that use standardized mean differences. JAMA 2007;298:430-7. 10.1001/jama.298.4.430 - DOI - PubMed

MeSH terms

Actions
Actions
Actions
Actions
Actions
Actions

Grants and funding

LinkOut - more resources

Full Text Sources

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

Validity of data extraction in evidence synthesis practice of adverse events: reproducibility study

Affiliations

Validity of data extraction in evidence synthesis practice of adverse events: reproducibility study

Authors

Affiliations

Abstract

Conflict of interest statement

Figures

References

MeSH terms

Grants and funding

LinkOut - more resources

Full Text Sources