Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research

Abstract

Background: The reporting of outcomes within published randomized trials has previously been shown to be incomplete, biased and inconsistent with study protocols. We sought to determine whether outcome reporting bias would be present in a cohort of government-funded trials subjected to rigorous peer review.

Methods: We compared protocols for randomized trials approved for funding by the Canadian Institutes of Health Research (formerly the Medical Research Council of Canada) from 1990 to 1998 with subsequent reports of the trials identified in journal publications. Characteristics of reported and unreported outcomes were recorded from the protocols and publications. Incompletely reported outcomes were defined as those with insufficient data provided in publications for inclusion in meta-analyses. An overall odds ratio measuring the association between completeness of reporting and statistical significance was calculated stratified by trial. Finally, primary outcomes specified in trial protocols were compared with those reported in publications.

Results: We identified 48 trials with 68 publications and 1402 outcomes. The median number of participants per trial was 299, and 44% of the trials were published in general medical journals. A median of 31% (10th-90th percentile range 5%-67%) of outcomes measured to assess the efficacy of an intervention (efficacy outcomes) and 59% (0%-100%) of those measured to assess the harm of an intervention (harm outcomes) per trial were incompletely reported. Statistically significant efficacy outcomes had a higher odds than nonsignificant efficacy outcomes of being fully reported (odds ratio 2.7; 95% confidence interval 1.5-5.0). Primary outcomes differed between protocols and publications for 40% of the trials.

Interpretation: Selective reporting of outcomes frequently occurs in publications of high-quality government-funded trials.

Fig. 1: Hierarchy of the levels of outcome reporting. *See the online appendix (www.cmaj.ca/cgi/content/full/171/7/735/DC1) for descriptions of the amount of data required for meta-analyses of fully reported outcomes

Fig. 2: Identification of published randomized trials approved for funding by the Canadian Institutes of Health Research (formerly the Medical Research Council of Canada) from 1990 to 1998.

Fig. 3: Number of incompletely reported outcomes per trial, displayed according to total number of outcomes per trial, for outcomes measured to assess the efficacy of an intervention (top panel) and those measured to assess the harm of an intervention (bottom panel).

Fig. 4: Odds ratios (squares) and 95% confidence intervals for outcome reporting bias involving efficacy outcomes (30 trials, top panel) and harm outcomes (4 trials, bottom panel). The size of the square reflects the statistical weight of the trial in calculating the pooled odds ratio (represented by the diamond and dotted line).