The number of patients and events required to limit the risk of overestimation of intervention effects in meta-analysis--a simulation study

Abstract

Background: Meta-analyses including a limited number of patients and events are prone to yield overestimated intervention effect estimates. While many assume bias is the cause of overestimation, theoretical considerations suggest that random error may be an equal or more frequent cause. The independent impact of random error on meta-analyzed intervention effects has not previously been explored. It has been suggested that surpassing the optimal information size (i.e., the required meta-analysis sample size) provides sufficient protection against overestimation due to random error, but this claim has not yet been validated.

Methods: We simulated a comprehensive array of meta-analysis scenarios where no intervention effect existed (i.e., relative risk reduction (RRR) = 0%) or where a small but possibly unimportant effect existed (RRR = 10%). We constructed different scenarios by varying the control group risk, the degree of heterogeneity, and the distribution of trial sample sizes. For each scenario, we calculated the probability of observing overestimates of RRR>20% and RRR>30% for each cumulative 500 patients and 50 events. We calculated the cumulative number of patients and events required to reduce the probability of overestimation of intervention effect to 10%, 5%, and 1%. We calculated the optimal information size for each of the simulated scenarios and explored whether meta-analyses that surpassed their optimal information size had sufficient protection against overestimation of intervention effects due to random error.

Results: The risk of overestimation of intervention effects was usually high when the number of patients and events was small and this risk decreased exponentially over time as the number of patients and events increased. The number of patients and events required to limit the risk of overestimation depended considerably on the underlying simulation settings. Surpassing the optimal information size generally provided sufficient protection against overestimation.

Conclusions: Random errors are a frequent cause of overestimation of intervention effects in meta-analyses. Surpassing the optimal information size will provide sufficient protection against overestimation.

Figure 1. Flowchart of simulations and analyses.

Simulation scenarios that included combinations of Cochrane Heart Group survey based trial sample size distribution and either ‘moderate’ or ‘high’ control group risks were not performed.

Figure 2. Presents the proportions of pooled intervention effects overestimating the relative risk reduction with 30% (– ▪ – ▪) and 20% (•••••••••) in the scenario with no underlying intervention effect (i.e., RRR = 0%), the trial sample size distribution is based on the Cochrane Heart Group survey, the control group risk is moderate (i.e., drawn from a uniform distribution between 5% and 15%) and the heterogeneity is moderate (i.e., τ² = 0.15).

The proportion of pooled intervention effect estimates (the risk of overestimation) are plotted in relation to the cumulative number of patients (upper plot) and events (lower plot).