Review

. 2020 Aug 26:370:m2898.

doi: 10.1136/bmj.m2898.

Potential impact of missing outcome data on treatment effects in systematic reviews: imputation study

Lara A Kahale^{1

2}, Assem M Khamis³, Batoul Diab¹, Yaping Chang⁴, Luciane Cruz Lopes⁵, Arnav Agarwal^{4

6}, Ling Li⁷, Reem A Mustafa^{4

8}, Serge Koujanian⁹, Reem Waziry¹⁰, Jason W Busse^{4

11

12

13}, Abeer Dakik¹, Holger J Schünemann⁴, Lotty Hooft², Rob Jpm Scholten², Gordon H Guyatt^{4

14}, Elie A Akl^{15

4}

Affiliations

¹ Department of Internal Medicine, American University of Beirut Medical Center, PO Box 11-0236, Riad-El-Solh Beirut 1107 2020, Beirut, Lebanon.
² Cochrane Netherlands and Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, Netherlands.
³ Wolfson Palliative Care Research Centre, Hull York Medical School, University of Hull, Hull, UK.
⁴ Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, ON, Canada.
⁵ Pharmaceutical Sciences Post Graduate Course, University of Sorocaba, UNISO, Sorocaba, Sao Paulo, Brazil.
⁶ Department of Medicine, University of Toronto, Toronto, ON, Canada.
⁷ Chinese Evidence-based Medicine Center, West China Hospital, Sichuan University, Chengdu, China.
⁸ Departments of Medicine and Biomedical and Health Informatics, University of Missouri-Kansas City, Kansas City, MO, USA.
⁹ Department of Evaluative Clinical Sciences, Sunnybrook Health Sciences Centre, Toronto, ON, Canada.
¹⁰ Department of Epidemiology, Harvard TH Chan School of Public Health, Boston, MA, USA.
¹¹ Department of Anaesthesia, McMaster University, Hamilton, ON, Canada.
¹² The Michael G DeGroote Institute for Pain Research and Care, McMaster University, Hamilton, ON, Canada.
¹³ Canadian Veterans Chronic Pain Centre of Excellence, Hamilton, ON, Canada.
¹⁴ Department of Medicine, McMaster University, Hamilton, ON, Canada.
¹⁵ Department of Internal Medicine, American University of Beirut Medical Center, PO Box 11-0236, Riad-El-Solh Beirut 1107 2020, Beirut, Lebanon ea32@aub.edu.lb.

PMID: 32847800
PMCID: PMC7448113
DOI: 10.1136/bmj.m2898

Review

Potential impact of missing outcome data on treatment effects in systematic reviews: imputation study

Lara A Kahale et al. BMJ. 2020.

. 2020 Aug 26:370:m2898.

doi: 10.1136/bmj.m2898.

Authors

Affiliations

¹ Department of Internal Medicine, American University of Beirut Medical Center, PO Box 11-0236, Riad-El-Solh Beirut 1107 2020, Beirut, Lebanon.
² Cochrane Netherlands and Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht University, Utrecht, Netherlands.
³ Wolfson Palliative Care Research Centre, Hull York Medical School, University of Hull, Hull, UK.
⁴ Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, ON, Canada.
⁵ Pharmaceutical Sciences Post Graduate Course, University of Sorocaba, UNISO, Sorocaba, Sao Paulo, Brazil.
⁶ Department of Medicine, University of Toronto, Toronto, ON, Canada.
⁷ Chinese Evidence-based Medicine Center, West China Hospital, Sichuan University, Chengdu, China.
⁸ Departments of Medicine and Biomedical and Health Informatics, University of Missouri-Kansas City, Kansas City, MO, USA.
⁹ Department of Evaluative Clinical Sciences, Sunnybrook Health Sciences Centre, Toronto, ON, Canada.
¹⁰ Department of Epidemiology, Harvard TH Chan School of Public Health, Boston, MA, USA.
¹¹ Department of Anaesthesia, McMaster University, Hamilton, ON, Canada.
¹² The Michael G DeGroote Institute for Pain Research and Care, McMaster University, Hamilton, ON, Canada.
¹³ Canadian Veterans Chronic Pain Centre of Excellence, Hamilton, ON, Canada.
¹⁴ Department of Medicine, McMaster University, Hamilton, ON, Canada.
¹⁵ Department of Internal Medicine, American University of Beirut Medical Center, PO Box 11-0236, Riad-El-Solh Beirut 1107 2020, Beirut, Lebanon ea32@aub.edu.lb.

PMID: 32847800
PMCID: PMC7448113
DOI: 10.1136/bmj.m2898

Erratum in

Potential impact of missing outcome data on treatment effects in systematic reviews: imputation study.
[No authors listed] [No authors listed] BMJ. 2020 Dec 14;371:m4783. doi: 10.1136/bmj.m4783. BMJ. 2020. PMID: 33318027 No abstract available.

Abstract

Objective: To assess the risk of bias associated with missing outcome data in systematic reviews.

Design: Imputation study.

Setting: Systematic reviews.

Population: 100 systematic reviews that included a group level meta-analysis with a statistically significant effect on a patient important dichotomous efficacy outcome.

Main outcome measures: Median percentage change in the relative effect estimate when applying each of the following assumption (four commonly discussed but implausible assumptions (best case scenario, none had the event, all had the event, and worst case scenario) and four plausible assumptions for missing data based on the informative missingness odds ratio (IMOR) approach (IMOR 1.5 (least stringent), IMOR 2, IMOR 3, IMOR 5 (most stringent)); percentage of meta-analyses that crossed the threshold of the null effect for each method; and percentage of meta-analyses that qualitatively changed direction of effect for each method. Sensitivity analyses based on the eight different methods of handling missing data were conducted.

Results: 100 systematic reviews with 653 randomised controlled trials were included. When applying the implausible but commonly discussed assumptions, the median change in the relative effect estimate varied from 0% to 30.4%. The percentage of meta-analyses crossing the threshold of the null effect varied from 1% (best case scenario) to 60% (worst case scenario), and 26% changed direction with the worst case scenario. When applying the plausible assumptions, the median percentage change in relative effect estimate varied from 1.4% to 7.0%. The percentage of meta-analyses crossing the threshold of the null effect varied from 6% (IMOR 1.5) to 22% (IMOR 5) of meta-analyses, and 2% changed direction with the most stringent (IMOR 5).

Conclusion: Even when applying plausible assumptions to the outcomes of participants with definite missing data, the average change in pooled relative effect estimate is substantive, and almost a quarter (22%) of meta-analyses crossed the threshold of the null effect. Systematic review authors should present the potential impact of missing outcome data on their effect estimates and use this to inform their overall GRADE (grading of recommendations assessment, development, and evaluation) ratings of risk of bias and their interpretation of the results.

PubMed Disclaimer

Conflict of interest statement

Competing interest: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: support from the Cochrane Methods Innovation Fund for the submitted work, no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.

Figures

**Fig 1**
Formula to quantify percentage change in relative effect. CCA=complete case analysis

**Fig 2**
Change in relative effect estimate (by direction) between the sensitivity analysis pooled effect estimate (assumption) and the sensitivity analysis pooled percentage of meta-analyses effect estimate (complete case analysis) when considering participants with definite missing data. Coloured bars represent the percentage of meta-analyses with change in relative effect estimate (by direction). Numerical values represent the median (interquartile range) for increase and decrease in relative effect estimate (n=100, respectively. IMOR=informative missing odds ratio

**Fig 3**
Results of meta-analyses that crossed the threshold of null effect when considering participants with definite missing data and comparing the sensitivity analysis pooled relative effect (assumption) with the sensitivity analysis pooled relative effect (complete case analysis) (n=87 systematic reviews that did not cross the threshold of null effect under the complete case analysis method). IMOR=informative missing odds ratio

**Fig 4**
Change in heterogeneity (I²) across different methods of handling missing data. IMOR=informative missing odds ratio

See this image and copyright information in PMC

References

1. National Research Council The Prevention and Treatment of Missing Data in Clinical Trials. The National Academic Press, 2010. - PubMed
1. Akl EA, Briel M, You JJ, et al. Potential impact on estimated treatment effects of information lost to follow-up in randomised controlled trials (LOST-IT): systematic review. BMJ 2012;344:e2809. 10.1136/bmj.e2809 - DOI - PubMed
1. Bell ML, Fiero M, Horton NJ, Hsu CH. Handling missing data in RCTs; a review of the top medical journals. BMC Med Res Methodol 2014;14:118. 10.1186/1471-2288-14-118 - DOI - PMC - PubMed
1. Fielding S, Ogbuagu A, Sivasubramaniam S, MacLennan G, Ramsay CR. Reporting and dealing with missing quality of life data in RCTs: has the picture changed in the last decade? Qual Life Res 2016;25:2977-83. 10.1007/s11136-016-1411-6 - DOI - PMC - PubMed
1. Hussain JA, Bland M, Langan D, Johnson MJ, Currow DC, White IR. Quality of missing data reporting and handling in palliative care trials demonstrates that further development of the CONSORT statement is required: a systematic review. J Clin Epidemiol 2017;88:81-91. 10.1016/j.jclinepi.2017.05.009 - DOI - PMC - PubMed

Publication types

Actions
Actions

MeSH terms

Actions
Actions
Actions
Actions
Actions
Actions
Actions

LinkOut - more resources

Full Text Sources

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

Potential impact of missing outcome data on treatment effects in systematic reviews: imputation study

Affiliations

Potential impact of missing outcome data on treatment effects in systematic reviews: imputation study

Authors

Affiliations

Erratum in

Abstract

Conflict of interest statement

Figures

References

Publication types

MeSH terms

LinkOut - more resources

Full Text Sources