Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2022 Apr;37(5):1247-1253.
doi: 10.1007/s11606-021-07135-3. Epub 2021 Oct 19.

Approaches to Assessing and Adjusting for Selective Outcome Reporting in Meta-analysis

Jeffrey L Jackson  1 , Ethan M Balk  2 , Noorie Hyun  3 , Akira Kuriyama  4
Affiliations

Approaches to Assessing and Adjusting for Selective Outcome Reporting in Meta-analysis

Jeffrey L Jackson et al. J Gen Intern Med. 2022 Apr.

Abstract

Background: Selective or non-reporting of study outcomes results in outcome reporting bias.

Objective: We sought to develop and assess tools for detecting and adjusting for outcome reporting bias.

Design: Using data from a previously published systematic review, we abstracted whether outcomes were reported as collected, whether outcomes were statistically significant, and whether statistically significant outcomes were more likely to be reported. We proposed and tested a model to adjust for unreported outcomes and compared our model to three other methods (Copas, Frosi, trim and fill). Our approach assumes that unreported outcomes had a null intervention effect with variance imputed based on the published outcomes. We further compared our approach to these models using simulation, and by varying levels of missing data and study sizes.

Results: There were 286 outcomes reported as collected from 47 included trials: 142 (48%) had the data provided and 144 (52%) did not. Reported outcomes were more likely to be statistically significant than those collected but for which data were unreported and for which non-significance was reported (RR, 2.4; 95% CI, 1.9 to 3.0). Our model and the Copas model provided similar decreases in the pooled effect sizes in both the meta-analytic data and simulation studies. The Frosi and trim and fill methods performed poorly.

Limitations: Single intervention of a single disease with only randomized controlled trials; approach may overestimate outcome reporting bias impact.

Conclusion: There was evidence of selective outcome reporting. Statistically significant outcomes were more likely to be published than non-significant ones. Our simple approach provided a quick estimate of the impact of unreported outcomes on the estimated effect. This approach could be used as a quick assessment of the potential impact of unreported outcomes.

Keywords: Meta-analysis; Outcome reporting bias; Statistical adjustment.

PubMed Disclaimer

Conflict of interest statement

The authors declare that they do not have a conflict of interest.

Similar articles

See all similar articles

References

    1. Jackson JL, Kuriyama A. From the Editors’ Desk: Bias in Systematic Reviews-Let the Reader Beware. J Gen Intern Med. 2018;33(2):133–135. doi: 10.1007/s11606-017-4236-2. - DOI - PMC - PubMed
    1. Dwan K, Gamble C, Williamson PR, Kirkham JJ. Systematic review of the empirical evidence of study publication bias and outcome reporting bias - an updated review. PLoS One. 2013;8(7):e66844. doi: 10.1371/journal.pone.0066844. - DOI - PMC - PubMed
    1. Harbord RM, Egger M, Sterne JA. A modified test for small-study effects in meta-analyses of controlled trials with binary endpoints. Stat Med. 2006;25(20):3443–3457. doi: 10.1002/sim.2380. - DOI - PubMed
    1. Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L. Comparison of two methods to detect publication bias in meta-analysis. JAMA. 2006;295(6):676–680. doi: 10.1001/jama.295.6.676. - DOI - PubMed
    1. Rothstein HR, Sutton AJ, Borenstein M. Publication bias in meta-analysis. Publication bias in meta-analysis: Prevention, assessment and adjustments. John Wiley & Sons; 2005.

LinkOut - more resources