Predictive ability of meta-analyses of randomised controlled trials

Abstract

Although meta-analysis of randomised clinical trials is increasingly used, the evaluation of its ability to predict the results of large trials is not available. We have calculated the relative risks (and 95% confidence intervals) for thirty meta-analyses of different interventions in perinatal medicine, covering 185 randomised controlled trials, but with the largest trial removed. We then compared those results with the result of the largest trial (total sample size more than 1000) done on that intervention and outcome. Twenty-four meta-analyses correctly predicted the direction of the treatment effect, but only eighteen of the thirty were the same both in direction of treatment effect and in statistical significance as the largest trial. There was moderate agreement, beyond chance, between meta-analysis and largest trial results (kappa statistic 0.46-0.53). A meta-analysis demonstrating a protective effect from an intervention of more than 40% had a 60% probability of correctly predicting results of the same magnitude of the largest trial. Researchers and funding agencies may use meta-analysis before recommending a clinical practice or to summarise results of three controlled trials before deciding on additional studies of promising interventions. However, further evaluation of the meta-analytical method is needed if the qualitative and quantitative results it yields are to be better understood.