Opportunities for selective reporting of harms in randomized clinical trials: Selection criteria for non-systematic adverse events

Abstract

Background: Adverse events (AEs) in clinical trials may be reported in multiple sources. Different methods for reporting adverse events across trials or across sources for a single trial may produce inconsistent information about the adverse events associated with interventions.

Methods: We compared the methods authors use to decide which AEs to include in a particular source (i.e., "selection criteria"), including the number of different types of AEs reported (i.e., rather than the number of events). We compared sources (e.g., journal articles, clinical study reports (CSRs)) of trials for two drug-indications-gabapentin for neuropathic pain and quetiapine for bipolar depression. Electronic searches were completed in 2015. We identified selection criteria and assessed how criteria affected AE reporting.

Results: We identified 21 gabapentin and 7 quetiapine trials. We found 6 gabapentin CSRs and 2 quetiapine CSRs, all written by drug manufacturers. All CSRs reported all AEs without applying selection criteria; by comparison, no other source reported all AEs, and 15/68 (22%) gabapentin sources and 19/48 (40%) quetiapine sources reported using selection criteria. Selection criteria greatly affected the number of AEs reported. For example, 67/316 (21%) AEs in one quetiapine trial met the criterion "occurring in ≥2% of participants in any treatment group," while only 5/316 (2%) AEs met the criterion "occurring in ≥10% of quetiapine-treated patients and twice as frequent in the quetiapine group as the placebo group."

Conclusions: Selection criteria for reporting AEs vary across trials and across sources for individual trials. If investigators do not pre-specify selection criteria, they might "cherry-pick" AEs based on results. Even if investigators pre-specify selection criteria, selective reporting will produce biased meta-analyses and clinical practice guidelines. Data about all AEs identified in clinical trials should be publicly available; however, sharing data will not solve all the problems identified in this study.

Keywords: Adverse events; Clinical trials; Data sharing; Harms; Reporting bias; Selective outcome reporting; Trial registration.

Fig. 1

Observed components of selection criteria. Selection criteria that we applied in sources about gabapentin (a) and quetiapine (b). Shown are all possible combinations of selection criteria. Each of the three rings of the circle represents a different component of the selection criteria: numerical threshold, participant group, and difference in frequency threshold. Green text indicates selection criteria reported in public sources. All = all participants combined across groups; Any group = participants in a particular intervention group; Active = participants in the active intervention group (i.e., gabapentin or quetiapine).

Fig. 2

Reported adverse events (AEs) and adverse event selection criteria. a Gabapentin. Six clinical study reports (CSRs) we identified appeared to report all AEs. All other sources in this figure are public sources. b Quetiapine. Two CSRs we identified appeared to report all AEs. Two CSR-synopses reported AE results and AE selection criteria. All other sources in this figure are public sources

Fig. 3

Percentage of adverse events (AEs) that would be reported using different selection criteria. We applied 45 different selection criteria to AEs in each of the eight trials for which we identified a clinical study report (CSR). To illustrate the potential variation in reported non-systematic AEs for each trial, we calculated the percentage of AEs that would be reported using each selection criterion, and we colored the figure using a heat map, with green representing the most AEs and red representing the fewest AEs. Squares outlined in black represent the selection criteria used in at least one source about that trial (e.g., a source describing Serpell 2002 reported AEs that occurred in ≥ 5% of participants in the gabapentin trial). Some trials did not have any sources that described the selection criteria (e.g., 945–224). A = no difference in frequency threshold; B = higher frequency in gabapentin/quetiapine than in placebo; C = frequency in gabapentin/quetiapine at least twice as high as placebo. ¹Gabapentin trial; ²Quetiapine trial