Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Review
. 2020 May 15:29:e128.
doi: 10.1017/S2045796020000402.

Comparative clinical trials in psychotherapy: Have large effects been replicated?

Nickolas D Frost  1 , Thomas W Baskin  2 , Bruce E Wampold  1   3
Affiliations
Review

Comparative clinical trials in psychotherapy: Have large effects been replicated?

Nickolas D Frost et al. Epidemiol Psychiatr Sci. .

Abstract

Aims: The purpose of this review is to examine the replication attempts of psychotherapy clinical trials for depression and anxiety. We focus specifically on replications of trials that exhibit large differences between psychotherapies. The replicability of these trials is especially important for meta-analysis, where the inclusion of false-positive trials can lead to erroneous conclusions about treatment efficacy.

Methods: Standard replication criteria were developed to distinguish direct from conceptual replication methodologies. Next, an exhaustive literature search was conducted for published meta-analyses of psychotherapy comparisons. Trials that exhibited large effects (d > 0.8) were culled from these meta-analyses. For each trial, a cited replication was conducted to determine if the trial had been subsequently replicated by either 'direct' or 'conceptual' methods. Finally, a broader search was conducted to examine the extent of replication efforts in the psychotherapy literature overall.

Results: In the meta-analytic search, a total of N = 10 meta-analyses met the inclusion criteria. From these meta-analyses, N = 12 distinct trials exhibited large effect sizes. The meta-analyses containing more than two large effect trials reported evidence for treatment superiority. A cited replication search yielded no direct replication attempts (N = 0) for the trials with large effects, and N = 4 conceptual replication attempts of average or above average quality. However, of these four attempts, only two partially corroborated the results from their original trial.

Conclusion: Meta-analytic reviews are influenced by trials with large effects, and it is not uncommon for these reviews to contain several such trials. Since we find no evidence that trials with such large effects are directly replicable, treatment superiority conclusions from these reviews are highly questionable. To enhance the quality of clinical science, the development of authoritative replication criteria for clinical trials is needed. Moreover, quality benchmarks should be considered before trials are included in a meta-analysis, or replications are attempted.

Keywords: Depression; psychotherapy; randomised controlled trials; research design and methods.

PubMed Disclaimer

Conflict of interest statement

The authors have no conflicts of interest to declare.

Figures

Fig. 1.
Fig. 1.
Flow diagram of the meta-analysis selection process.
See this image and copyright information in PMC

References

    1. Arntz A (2003) Cognitive therapy versus applied relaxation as treatment of generalized anxiety disorder. Behavior Research and Therapy 41, 633–646. - PubMed
    1. Ayen I and Hautzinger M (2004) Kognitive Verhaltenstherapie bei Depressionen im Klimakterium: Eine kontrollierte, randomisierte Interventionsstudie. [Cognitive behavior therapy for depression in menopausal women. A controlled, randomized treatment study.]. Zeitschrift für Klinische Psychologie und Psychotherapie: Forschung und Praxis. 33, 290–299.
    1. Baardseth TP, Goldberg SB, Pace BT, Wislocki AP, Frost ND, Siddiqui JR and Wampold BE (2013) Cognitive-behavioral therapy versus other therapies: Redux. Clinical Psychology Review 33, 395–405. - PubMed
    1. Baldwin SA and Imel ZE (2013) Therapist effects: finding and methods In Lambert MJ (ed.), Bergin and Garfield's Handbook of Psychotherapy and Behavior Change. New York: Wiley, pp. 258–297.
    1. Barkham M, Rees A, Shapiro DA, Stiles WB, Agnew RM, Halstead J, Culverwell A and Harrington VM (1996) Outcomes of time-limited psychotherapy in applied settings: replicating the Second Sheffield Psychotherapy Project. Journal of Consulting and Clinical Psychology 64, 1079–1085. - PubMed

LinkOut - more resources