The fading of reported effectiveness. A meta-analysis of randomised controlled trials

Abstract

Background: The "real" effect size of a medical therapy is constant over time. In contrast, the effect size reported in randomised controlled trials (RCTs) may change over time because the sum of all kinds of bias influencing the reported effectiveness is not necessarily constant. As this would affect the validity of meta-analyses, we tested the hypothesis that the reported effect size decreases over time. Furthermore, we tested three hypotheses that would explain a possible change.

Methods: Because of well established outcome measures, the lipid-lowering drugs Pravastatin and Atorvastatin (serum low-density lipoprotein cholesterol, LDL-C) and the anti-glaucoma drugs Timolol and Latanoprost (intraocular pressure, IOP) were chosen for this investigation. Studies were identified by a standardized MEDLINE search. RCTs investigating the above identified medications administered as monotherapy, and in defined dosages, were included. Publication year, baseline (= pre-treatment value in the treatment group of interest) and post intervention means, number of patients and the assignment to experimental or control group were extracted for each study.

Results: A total of 625 citations were screened; 206 met the inclusion criteria. The reported effect size of Pravastatin (change of reported effect size in five years: -3.22% LDL-C, P < .0001), Timolol (-0.56 mmHg, P < .0001) and Latanoprost (-1.78 mmHg, P = .0074) decreased over time, while there was no significant change for Atorvastatin (+0.31% LDL-C, P = .8618). Multiple regression analysis showed that baseline values were the most important influencing factor; study size or treatment group did not play a significant role.

Conclusion: The effectiveness of medical therapies reported in RCTs decreases over time in three of the four investigated pharmaceuticals, caused mainly by baseline differences. We call this phenomenon "fading of reported effectiveness". Under this condition the validity of a meta-analysis may be impaired. Therefore we propose to observe this phenomenon in future meta-analyses in order to guarantee a maximum of transparency.

Figure 1

Flow diagram. Abbreviations: RCT, randomized controlled trial; n, number of trials. * Parameter of interest: Reported effectiveness of the pharmaceutical, measured as change of intraocular pressure (Timolol, Latanoprost) or change in low-density lipoprotein cholesterol (Pravastatin, Atorvastatin). ** Dose of interest: In terms of comparability studies that did not use the pharmaceutical in the most common dosage were excluded, as well as studies that increased the individual dosage until a certain outcome was reached.

Figure 2

Regression analysis of the relation between year of publication and reported effectiveness of Pravastatin (y = 90.818 – 0.643x; P < 0.0001), Atorvastatin (y = 29.900 + 0.062x; P = 0.8618, not significant), Timolol (y = 16.983 – 0.113x; P < 0.0001), and Latanoprost (y = 42.069 – 0.356x; P = 0.0074). Abbreviations: IOP, intraocular pressure; LDL-C, low-density lipoprotein cholesterol; x, reported effectiveness (for Timolol and Latanoprost change of IOP measured in mmHg, for Pravastatin and Atorvastatin change of LDL-C measured in %); y, year of publication minus 1900.

Figure 3

Funnel plots; shown is the relation between study size and reported effectiveness. None of the relations are statistically significant (Pravastatin P = 0.1327; Atorvastatin P = 0.8281; Timolol P = 0.2432; Latanoprost: P = 0.2251). Abbreviations: n, number of trials; IOP, intraocular pressure; LDL-C, low-density lipoprotein cholesterol.