Identifying null meta-analyses that are ripe for updating

Abstract

Background: As an increasingly large number of meta-analyses are published, quantitative methods are needed to help clinicians and systematic review teams determine when meta-analyses are not up to date.

Methods: We propose new methods for determining when non-significant meta-analytic results might be overturned, based on a prediction of the number of participants required in new studies. To guide decision making, we introduce the "new participant ratio", the ratio of the actual number of participants in new studies to the predicted number required to obtain statistical significance. A simulation study was conducted to study the performance of our methods and a real meta-analysis provides further evidence.

Results: In our three simulation configurations, our diagnostic test for determining whether a meta-analysis is out of date had sensitivity of 55%, 62%, and 49% with corresponding specificity of 85%, 80%, and 90% respectively.

Conclusions: Simulations suggest that our methods are able to detect out-of-date meta-analyses. These quick and approximate methods show promise for use by systematic review teams to help decide whether to commit the considerable resources required to update a meta-analysis. Further investigation and evaluation of the methods is required before they can be recommended for general use.

Figure 1

Cumulative meta-analysis of 9 trials of intravenous streptokinase for acute myocardial infarction [8]showing the application of the prediction (equation 2) using the results of the first 5 trials. The cumulative z-statistic is plotted versus the cumulative number of participants, shown on a square root scale (see Appendix). The horizontal dashed line indicates the two-sided critical value of the z-statistic at α = 0.01. The diagonal dotted line emanates from the origin and passes through the point representing the meta-analysis of the first 5 trials (double circle). The location where this line intersects the horizontal dashed line is marked by a vertical dotted line projecting down to the horizontal axis and indicating the predicted number of participants required to obtain statistical significance.

Figure 2

Boxplots showing distribution of z-statistics for each simulation configuration before (A,B,C) and after (A*,B*,C*) updating. The midline of each box marks the median. The bottom and top of the box represent the 25th and 75th percentiles, respectively; thus the height of the box is the interquartile range (IQR). Values that lie more that 1.5 IQRs beyond the top or bottom of the box (outliers) are shown individually. Lines project from the top and bottom of the box to the most extreme values not more than 1.5 IQRs beyond the box.

Figure 3

Boxplots comparing the new participant ratio for meta-analyses found to be out of date versus those not out of date in simulation configuration C. See caption of Figure 3 for interpretation of boxplots. Not shown are cases where the new participant ratio is infinite (7 cases out of 10000, 5 [71%] of which were in the out-of-date group) or where the new participant ratio < 1/100 (1361 cases out of 10000, 6 [0.4%] of which were in the out-of-date group).

Figure 4

ROC curves for the diagnostic test for each configuration. The dot on each curve indicates the sensitivity and 1-specificity when the threshold for the new participant ratio is q = 1. For configuration A, when q = 1, the sensitivity is 55% and the specificity is 85%. For configuration B these values are 62% and 80%, and for configuration C they are 49% and 90%. Increasing q corresponds to moving counter-clockwise along the curve. Note that a Bernoulli random decision with probability p that a review is out of date has sensitivity p and specificity 1 - p, corresponding to an ROC curve consisting of a 45-degree line. ROC curves above the 45-degree line indicate performance superior to chance.

Figure 5

Geometrical illustration of the prediction. The number of additional participants, n, is determined by extrapolating a line segment starting at the origin (0,0) and passing through the point (, |Z|) to the point where it intersects with a horizontal line at Z_c.