The conclusiveness of trial sequential analysis varies with estimation of between-study variance: a case study

Abstract

Background: Trial sequential methods have been introduced to address issues related to increased likelihood of incorrectly rejecting the null hypothesis in meta-analyses due to repeated significance testing. Between-study variance (τ²) and its estimate ( $\hat{\tau }$ ²) play a crucial role in both meta-analysis and trial sequential analysis with the random-effects model. Therefore, we investigated how different $\hat{\tau }$ ² impact the results of and quantities used in trial sequential analysis.

Methods: This case study was grounded in a Cochrane review that provides data for smaller (< 10 randomized clinical trials, RCTs) and larger (> 20 RCTs) meta-analyses. The review compared various outcomes between video-laryngoscopy and direct laryngoscopy for tracheal intubation, and we used outcomes including hypoxemia and failed intubation, stratified by difficulty, expertise, and obesity. We calculated odds ratios using inverse variance method with six estimators for τ², including DerSimonian-Laird, restricted maximum-likelihood, Paule-Mandel, maximum-likelihood, Sidik-Jonkman, and Hunter-Schmidt. Then we depicted the relationships between $\hat{\tau }$ ² and quantities in trial sequential analysis including diversity, adjustment factor, required information size (RIS), and α-spending boundaries.

Results: We found that diversity increases logarithmically with $\hat{\tau }$ ², and that the adjustment factor, RIS, and α-spending boundaries increase linearly with $\hat{\tau }$ ². Also, the conclusions of trial sequential analysis can differ depending on the estimator used for between-study variance.

Conclusion: This study highlights the importance of $\hat{\tau }$ ² in trial sequential analysis and underscores the need to align the meta-analysis and the trial sequential analysis by choosing estimators to avoid introducing biases and discrepancies in effect size estimates and uncertainty assessments.

Keywords: Heterogeneity; Meta-analysis; Optimal information size; Required information size; Sequential method; Tau-square.

Fig. 1

Scatter plots of (A) diversity ($\widehat{D}$ ²) versus estimated between-study variance ($\widehat{\tau }$ ²) and (B) required information size ($\widehat{\mathit{RIS}}$) versus estimated between-study variance ($\widehat{\tau }$ ²), for smaller meta-analysis without significant heterogeneity, smaller meta-analysis with significant heterogeneity, larger meta-analysis without significant heterogeneity, and larger meta-analysis with significant heterogeneity. DL, DerSimonian-Laird estimator; HS, Hunter-Schmidt estimator; ML, Maximum-likelihood estimator; PM, Paule-Mandel estimator; REML, Restricted maximum-likelihood estimator; SJ, Sidik-Jonkman estimator

Fig. 2

Scatter plots of (A) α-spending monitoring boundary versus estimated between-study variance ($\widehat{\tau }$ ²) and (B) observed cumulative z-score versus estimated between-study variance ($\widehat{\tau }$ ²), for smaller meta-analysis without significant heterogeneity, smaller meta-analysis with significant heterogeneity, larger meta-analysis without significant heterogeneity, and larger meta-analysis with significant heterogeneity. DL, DerSimonian-Laird estimator; HS, Hunter-Schmidt estimator; ML, Maximum-likelihood estimator; PM, Paule-Mandel estimator; REML, Restricted maximum-likelihood estimator; SJ, Sidik-Jonkman estimator

Fig. 3

Trial sequential analysis plots using different between-study variance estimators ($\widehat{\tau }$ ²) for failed intubations comparing hyper-angulated video laryngoscopy and direct laryngoscopy. DL, DerSimonian-Laird estimator; HS, Hunter-Schmidt estimator; ML, Maximum-likelihood estimator; PM, Paule-Mandel estimator; REML, Restricted maximum-likelihood estimator; SJ, Sidik-Jonkman estimator