Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 Sep 9;24(1):196.
doi: 10.1186/s12874-024-02318-y.

Major mistakes or errors in the use of trial sequential analysis in systematic reviews or meta-analyses - the METSA systematic review

Christian Gunge Riberholt  1   2   3 , Markus Harboe Olsen  4   5 , Joachim Birch Milan  4 , Sigurlaug Hanna Hafliðadóttir  6 , Jeppe Houmann Svanholm  7 , Elisabeth Buck Pedersen  8 , Charles Chin Han Lew  9   10 , Mark Aninakwah Asante  4 , Johanne Pereira Ribeiro  11   12 , Vibeke Wagner  8   13 , Buddheera W M B Kumburegama  4 , Zheng-Yii Lee  14   15 , Julie Perrine Schaug  11 , Christina Madsen  16 , Christian Gluud  4   17
Affiliations

Major mistakes or errors in the use of trial sequential analysis in systematic reviews or meta-analyses - the METSA systematic review

Christian Gunge Riberholt et al. BMC Med Res Methodol. .

Abstract

Background: Systematic reviews and data synthesis of randomised clinical trials play a crucial role in clinical practice, research, and health policy. Trial sequential analysis can be used in systematic reviews to control type I and type II errors, but methodological errors including lack of protocols and transparency are cause for concern. We assessed the reporting of trial sequential analysis.

Methods: We searched Medline and the Cochrane Database of Systematic Reviews from 1 January 2018 to 31 December 2021 for systematic reviews and meta-analysis reports that include a trial sequential analysis. Only studies with at least two randomised clinical trials analysed in a forest plot and a trial sequential analysis were included. Two independent investigators assessed the studies. We evaluated protocolisation, reporting, and interpretation of the analyses, including their effect on any GRADE evaluation of imprecision.

Results: We included 270 systematic reviews and 274 meta-analysis reports and extracted data from 624 trial sequential analyses. Only 134/270 (50%) systematic reviews planned the trial sequential analysis in the protocol. For analyses on dichotomous outcomes, the proportion of events in the control group was missing in 181/439 (41%), relative risk reduction in 105/439 (24%), alpha in 30/439 (7%), beta in 128/439 (29%), and heterogeneity in 232/439 (53%). For analyses on continuous outcomes, the minimally relevant difference was missing in 125/185 (68%), variance (or standard deviation) in 144/185 (78%), alpha in 23/185 (12%), beta in 63/185 (34%), and heterogeneity in 105/185 (57%). Graphical illustration of the trial sequential analysis was present in 93% of the analyses, however, the Z-curve was wrongly displayed in 135/624 (22%) and 227/624 (36%) did not include futility boundaries. The overall transparency of all 624 analyses was very poor in 236 (38%) and poor in 173 (28%).

Conclusions: The majority of trial sequential analyses are not transparent when preparing or presenting the required parameters, partly due to missing or poorly conducted protocols. This hampers interpretation, reproducibility, and validity.

Study registration: PROSPERO CRD42021273811.

Keywords: Meta-analysis; Methodology; Research-on-research; Systematic review; Trial sequential analysis.

PubMed Disclaimer

Conflict of interest statement

All authors have completed the ICMJE uniform disclosure form at http://www.icmje.org/disclosure-of-interest/ and declare: no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work. CG initiated and participated in the development of the Trial Sequential Analysis programme.

Figures

Fig. 1
Fig. 1
Flowchart. MEDLINE: Medical Literature Analysis and Retrieval System online; CDSR: Cochrane Database of Systematic Reviews; RCT: randomised clinical trial; TSA: Trial Sequential Analysis
Fig. 2
Fig. 2
Comparison between the number of outcomes and alpha level. Figures are showing the number of outcomes in studies with dichotomous outcomes (A) and continuous outcomes (B). Studies not reporting the alpha level are depicted in grey. A: 176 of 409 (43%) dichotomous outcomes had a reported alpha level of 5% or higher and more than one primary outcome. B: 91 of 162 (56%) continuous outcomes had a reported alpha level of 5% or higher and more than one primary outcome
See this image and copyright information in PMC

References

    1. Higgins J, Thomas J, Chandler J et al. Cochrane Handbook for Systematic Reviews of Interventions version 6.3 (updated February 2022). Cochrane. 2022. www.training.cochrane.org/handbook
    1. Ioannidis JPA. The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. Milbank Q. 2016;94:485–514. 10.1111/1468-0009.12210 - DOI - PMC - PubMed
    1. Gluud C. Testosterone and alcoholic cirrhosis. Epidemiologic, pathophysiologic and therapeutic studies in men. Dan Med Bull. 1988;35:564–75. - PubMed
    1. Fontelo P, Liu F. A review of recent publication trends from top publishing countries. Syst Rev. 2018;7:1–9. 10.1186/s13643-018-0819-1 - DOI - PMC - PubMed
    1. Zhao X, Jiang H, Yin J, et al. Changing trends in clinical research literature on PubMed database from 1991 to 2020. Eur J Med Res. 2022;27:1–11. 10.1186/s40001-022-00717-9 - DOI - PMC - PubMed

Publication types

MeSH terms

LinkOut - more resources