Effect sizes in cumulative meta-analyses of mental health randomized trials evolved over time

Abstract

Background and objective: Meta-analyses of randomized trials may incorporate new evidence, and estimated treatment effects changeover time. We evaluated whether the certainty and estimates of efficacy and tolerability of mental health interventions change over time, as more trials appear on the same topics.

Methods: One hundred meta-analyses (1,024 trial entries; 99,303 participants) with an outcome of death, relapse, failure or dropout and with five or more trials published in three or more different years were examined with cumulative meta-analysis and recursive cumulative meta-analysis.

Results: Eight meta-analyses reached formal statistical significance (P < .05) at some point, but lost this significance eventually when more trials were published; typically large effect sizes in early trials were dissipated with further evidence. With 500 randomized subjects,95% of the time, subsequent changes in odds ratio might be up to 1.5-fold. For death, relapse, and failure outcomes, a decrease in effect size was somewhat more common than an increase, when more data became available (157 vs. 125, P = .06). This was most clear for comparisons of pharmacotherapies versus placebo (79 vs. 51, P = .009).

Conclusions: Evidence based on a small number of randomized subjects should be interpreted cautiously. Early treatment efficacy of pharmacotherapies is occasionally overestimated.