Larger effect sizes were associated with higher quality ratings in complementary and alternative medicine randomized controlled trials

Abstract

Objective: To determine if the quality of complementary and alternative medicine (CAM) placebo controlled randomized controlled trials (RCTs) is related to the effect sizes they produce when the type of interventions and outcome variables are controlled.

Study design and setting: A sample of 25 CAM meta-analyses was identified from MEDLINE and hand searches containing published effect sizes for at least three efficacy trials employing placebo control groups. From these 25 reviews, 26 pairs of trials were selected: the one reporting the largest effect size and the matching trial reporting the smallest effect size. Quality and publication characteristics were then abstracted from each trial.

Results: Unlike the preponderance of past evidence examining the relationship between quality and effect size, the present study found that trials possessing the largest effect sizes within a meta-analysis were also associated with higher quality ratings than their counterparts possessing the lowest effect sizes (P=.019).

Conclusions: Possible theoretical reasons for this unexpected positive relationship include (a) sampling error, (b) reduced within group variation, (c) fraudulent reporting, and (d) the restriction of the analyses to placebo controlled trial.