The statistical significance of meta-analyses is frequently fragile: definition of a fragility index for meta-analyses

Abstract

Objectives: Meta-analyses inform clinical practice by summarizing treatment effect estimates based on results from several trials. However, the statistical significance of a meta-analysis (i.e., whether the pooled treatment effect is statistically significant or not) may rely on the outcome of only a few patients from specific trials in the meta-analysis. We aimed to evaluate the extent to which the statistical significance of meta-analyses can be changed (from statistically significant to nonsignificant, or vice versa) after modifying the event status of patients in specific arms of specific trials.

Methods: We conducted a cross-sectional analysis of meta-analyses of trials with a binary outcome from Cochrane Systematic Reviews. We defined the fragility index of meta-analyses as the minimum number of patients from one or more trials included in the meta-analysis for whom an event-status modification (i.e., changing an event to nonevent or a nonevent to event) would change the statistical significance of the pooled treatment effect. For statistically significant and nonsignificant meta-analyses, we evaluated the fragility index, the ratio between the fragility index and the total number of participants included in the trials, and the ratio between the fragility index and the total number of events.

Results: Our sample comprised 906 meta-analyses: 400 and 506 had statistically significant and nonsignificant pooled treatment effects, respectively. For statistically significant meta-analyses, the median fragility index was 12 (Q1-Q3: 4-33); for 29% the fragility index was 5 or less. Overall, 43% and 9% meta-analyses would have become nonsignificant if the event status was modified for less than 1% of the total participants in one or several specific trials, and for less than 1% of the total number of events, respectively. These proportions were similar for statistically nonsignificant meta-analyses. Overall, the statistical significance of 33% of all meta-analyses depended on the event status of five or fewer participants from one or more specific trials.

Conclusion: The statistical significance of meta-analyses often depends on the outcome of a few patients. The fragility index of meta-analyses may help in interpreting the conclusions of meta-analyses.

Keywords: Fragility index; Meta-analyses; P-value; Research methods; Statistical significance; Systematic reviews.