Time to stop randomized and large pragmatic trials for intensive care medicine syndromes: the case of sepsis and acute respiratory distress syndrome

Abstract

In this paper we discuss the limitations of large randomized controlled trials with mortality endpoints in patients with critical illness associated diagnoses such as sepsis. When patients with the same syndrome diagnosis do not share the pathways that lead to death (the attributable risk), any therapy can only lead to small effects in these populations. Using Monte Carlo simulations, we show how the syndrome-attributable risks of critical illness-associated diagnoses are likely overestimated using common statistical methods. This overestimation of syndrome-attributable risks leads to a corresponding overestimation of attainable treatment effects and an underestimation of required sample sizes. We demonstrate that larger and more 'pragmatic' randomized trials are not the solution because they decrease therapeutic and diagnostic precision, the therapeutic effect size and the probability of finding a beneficial effect. Finally, we argue that the most logical solution is a renewed focus on mechanistic research into the complexities of critical illness syndromes.

Keywords: Acute respiratory distress syndrome (ARDS); Monte Carlo simulation; clinical trials; research methods.

Figure 1

Schematic representation of syndrome-attributable risk. Critical illness syndromes are associated with high mortality risks, but the underlying conditions that cause these syndromes are themselves important root causes of mortality. ARDS, acute respiratory distress syndrome.

Figure 2

Trial sample size requirements as a function of the syndrome-attributable fraction of mortality for a two-group trial at a 35% control-group mortality rate, 5% type-I error rate and 20% type-II error rate. The relationship between required sample sizes and the syndrome-attributable fraction of mortality is displayed for three different hypothesized effect sizes. Vertical axis on the log scale.

Figure 3

Results of a simulation study demonstrating how syndrome-attributable risk are overestimated by matching for severity of illness. (A) In a simulated cohort of 4,000 patients, the risk of developing a critical illness syndrome is correlated to the mortality risk, but the presence of the syndrome doesn’t cause an increase in mortality risk (no horizontal shift between those with the syndrome compared to those without). (B) To adjust for confounding by illness severity, a researcher wants to match patients with the syndrome to similar patients without the syndrome on the basis of disease characterization (summarized by severity scores). (C) The unadjusted comparison shows patients with the syndrome are increased risk of dying. After adjustment by a good but imperfect severity score, the syndrome-attributable mortality appears to be 11%, while full characterization of the underlying (syndrome-independent) mortality risk reveals that the syndrome-attributable risk is negligible.

Figure 4

Simulation of hypothetical large ‘pragmatic’ trials investigating low tidal volume ventilation. The effects of low tidal volume ventilation in patients with ARDS and without ARDS were based on two large RCTs (A). We simulated the results of large ‘pragmatic’ trials in populations with different incidences of ARDS (B,C) and with changing diagnostic precision (D). ARDS, acute respiratory distress syndrome. ARDS, acute respiratory distress syndrome.