Outcomes and statistical power in adult critical care randomized trials

Abstract

Rationale: Intensive care unit (ICU)-based randomized clinical trials (RCTs) among adult critically ill patients commonly fail to detect treatment benefits.

Objectives: Appraise the rates of success, outcomes used, statistical power, and design characteristics of published trials.

Methods: One hundred forty-six ICU-based RCTs of diagnostic, therapeutic, or process/systems interventions published from January 2007 to May 2013 in 16 high-impact general or critical care journals were studied.

Measurement and main results: Of 146 RCTs, 54 (37%) were positive (i.e., the a priori hypothesis was found to be statistically significant). The most common primary outcomes were mortality (n = 40 trials), infection-related outcomes (n = 33), and ventilation-related outcomes (n = 30), with positive results found in 10, 58, and 43%, respectively. Statistical power was discussed in 135 RCTs (92%); 92 cited a rationale for their power parameters. Twenty trials failed to achieve at least 95% of their reported target sample size, including 11 that were stopped early due to insufficient accrual/logistical issues. Of 34 superiority RCTs comparing mortality between treatment arms, 13 (38%) accrued a sample size large enough to find an absolute mortality reduction of 10% or less. In 22 of these trials the observed control-arm mortality rate differed from the predicted rate by at least 7.5%.

Conclusions: ICU-based RCTs are commonly negative and powered to identify what appear to be unrealistic treatment effects, particularly when using mortality as the primary outcome. Additional concerns include a lack of standardized methods for assessing common outcomes, unclear justifications for statistical power calculations, insufficient patient accrual, and incorrect predictions of baseline event rates.

Keywords: critical care; intensive care; intensive care unit; randomized clinical trial; randomized controlled trial.

Figure 1.

Analytic sample of published randomized clinical trials (RCTs) of critical care interventions. *See Figure 5 legend. ICU = intensive care unit.

Figure 2.

Adjusted associations of selected randomized clinical trial (RCT) characteristics with positive primary outcomes. RCTs that included measures of morbidity or other clinical measures in the primary outcome were not categorized as mortality trials. ICU = intensive care unit.

Figure 3.

Expected versus actual treatment effect on mortality in 34 superiority trials in which the primary outcome was mortality. See Table E3 in the online supplement for the numbers shown in this figure and for further information regarding all trials in which mortality was a primary outcome.

Figure 4.

Expected versus actual treatment effect in 47 superiority trials with a binary nonmortal primary outcome. See Table E4 in the online supplement for the numbers shown in this figure and for further information regarding each trial and the primary outcomes.

Figure 5.

Simulation results of superiority trials where the primary outcome was mortality assuming 80% power to find a treatment-associated mortality reduction of 3 to 15%. n = 33 randomized clinical trials as noted in Figure 1 with the exception of the Prospective Recombinant Human Activated Protein C Worldwide Evaluation in Severe Sepsis and Septic Shock (PROWESS-SHOCK) trial due to the use of an adaptive control arm.