Reporting of Adverse Events in Published and Unpublished Studies of Health Care Interventions: A Systematic Review

Abstract

Background: We performed a systematic review to assess whether we can quantify the underreporting of adverse events (AEs) in the published medical literature documenting the results of clinical trials as compared with other nonpublished sources, and whether we can measure the impact this underreporting has on systematic reviews of adverse events.

Methods and findings: Studies were identified from 15 databases (including MEDLINE and Embase) and by handsearching, reference checking, internet searches, and contacting experts. The last database searches were conducted in July 2016. There were 28 methodological evaluations that met the inclusion criteria. Of these, 9 studies compared the proportion of trials reporting adverse events by publication status. The median percentage of published documents with adverse events information was 46% compared to 95% in the corresponding unpublished documents. There was a similar pattern with unmatched studies, for which 43% of published studies contained adverse events information compared to 83% of unpublished studies. A total of 11 studies compared the numbers of adverse events in matched published and unpublished documents. The percentage of adverse events that would have been missed had each analysis relied only on the published versions varied between 43% and 100%, with a median of 64%. Within these 11 studies, 24 comparisons of named adverse events such as death, suicide, or respiratory adverse events were undertaken. In 18 of the 24 comparisons, the number of named adverse events was higher in unpublished than published documents. Additionally, 2 other studies demonstrated that there are substantially more types of adverse events reported in matched unpublished than published documents. There were 20 meta-analyses that reported the odds ratios (ORs) and/or risk ratios (RRs) for adverse events with and without unpublished data. Inclusion of unpublished data increased the precision of the pooled estimates (narrower 95% confidence intervals) in 15 of the 20 pooled analyses, but did not markedly change the direction or statistical significance of the risk in most cases. The main limitations of this review are that the included case examples represent only a small number amongst thousands of meta-analyses of harms and that the included studies may suffer from publication bias, whereby substantial differences between published and unpublished data are more likely to be published.

Conclusions: There is strong evidence that much of the information on adverse events remains unpublished and that the number and range of adverse events is higher in unpublished than in published versions of the same study. The inclusion of unpublished data can also reduce the imprecision of pooled effect estimates during meta-analysis of adverse events.

Fig 1. Flow diagram for included studies.

Ninety-three articles were excluded based on full-text screening. The excluded studies tended not to include adverse events data or did not present relevant data for both published and unpublished articles (S2 Table).

Fig 2. Percentage of matched published and unpublished studies with adverse event information.

*Classified adverse events information as either “completely reported” versus “incompletely reported.” Incompletely reported adverse events could lack numerical data or include only selected adverse events, for example. Maund 2014a [16] and Maund 2014b [16] compare published trials to registry reports and clinical study reports (CSRs), respectively. Riveros 2013 [35] compares trials with number of adverse events reported and classifies adverse events information as either “completely reported” or “incompletely reported.”

Fig 3. Percentage of unmatched published and unpublished sources with adverse event information.

*Classified adverse events information as either “completely reported” versus “incompletely reported.” Incompletely reported adverse events could lack numerical data or include only selected adverse events, for example. Hemminki 1980a [23], 1980b [23], and 1980c [23] compare different drugs in different countries. Wieseler 2012 [44] and 2013a [43] and Wieseler 2012 [44] and 2013b [43] compare published sources with CSRs and registry reports, respectively.

Fig 4. Adverse events in matched published and unpublished sources.

In circumstances where multiple AEs were presented, the largest category is included (for example, all AEs or all SAEs). Pranić 2015 [34] and Tang 2015 [40] are excluded from the figure because, although the authors state that they identified a higher number of adverse events in unpublished sources than published, the actual figures were not presented. Le Noury 2015a [28] and 2015b [28] compare AEs for different drugs. Maund 2014a [16] and Maund 2014b [16] compare published trials to registry reports and CSRs, respectively. In Jefferson 2011 [26], the number of adverse events is small (10 events) and therefore does not show up in Fig 4 given the scale of the y-axis.

Fig 5. Percentage of adverse events missed without matched unpublished data.

In circumstances where multiple AEs were presented, the largest category is included in Fig 5 (for example, all AEs or all SAEs). Pranić 2015 [34] and Tang 2015 [40] are excluded because, although they state that they identified more adverse events in unpublished sources than published, the actual figures were not presented. Le Noury 2015a [28] and 2015b [28] compare AEs for different drugs. Maund 2014a [16] and Maund 2014b [16] compare published trials to registry reports and CSRs, respectively.

Fig 6. Number of published and unpublished sources in the meta-analyses.

Eyding 2010a [20] examines studies of reboxetine versus placebo and Eyding 2010b [20] examines studies of reboxetine versus selective serotonin reuptake inhibitors (SSRIs).

Fig 7. Percentage of participants in published and unpublished studies in the meta-analyses.

Eyding 2010a [20] examines studies of reboxetine versus placebo and Eyding 2010b [20] examines studies of reboxetine versus selective serotonin reuptake inhibitors (SSRIs).

Fig 8. Forest plots comparing pooled estimates for adverse events from all studies combined (published and unpublished) against published and unpublished studies alone.

There were also two systematic reviews identified in Potthast 2014 [33] in which unpublished additional trial data led to a new comparison not reported in the systematic reviews, which could not be represented here.