Effect sizes in ongoing randomized controlled critical care trials

Abstract

Background: An important limitation of many critical care trial designs is that they hypothesize large, and potentially implausible, reductions in mortality. Interpretation of trial results could be improved by systematic assessment of the plausibility of trial hypotheses; however, such assessment has not been attempted in the field of critical care medicine. The purpose of this study was to determine clinicians' views about prior probabilities and plausible effect sizes for ongoing critical care trials where the primary endpoint is landmark mortality.

Methods: We conducted a systematic review of clinical trial registries in September 2015 to identify ongoing critical care medicine trials where landmark mortality was the primary outcome, followed by a clinician survey to obtain opinions about ten large trials. Clinicians were asked to estimate the probability that each trial would demonstrate a mortality effect equal to or larger than that used in its sample size calculations.

Results: Estimates provided by individual clinicians varied from 0% to 100% for most trials, with a median estimate of 15% (IQR 10-20%). The median largest absolute mortality reduction considered plausible was 4.5% (IQR 3.5-5%), compared with a median absolute mortality reduction used in sample size calculations of 5% (IQR 3.6-10%) (P = 0.27).

Conclusions: For some of the largest ongoing critical care trials, many clinicians regard prior probabilities as low and consider that plausible effects on absolute mortality are less than 5%. Further work is needed to determine whether pooled estimates obtained by surveying clinicians are replicable and accurate or whether other methods of estimating prior probability are preferred.

Keywords: Clinical trial design; Critical care; Intensive care; Intensive care unit; Randomized clinical trial.

Fig. 1

Graphical representation of the method of estimation of the chances that a statistically significant result represents a “true–positive” based on 100 hypothetical trials where there is a 10% chance the hypothesis is correct and experiments are conducted with 90% power at an α of 0.05. In this example, where there is a 10% prior probability that the hypothesis is correct, each box represents a hypothetical trial. The top row of boxes (surrounded by a green line) represent the 10 occasions where the hypothesis is correct; the remaining 90 boxes represent the occasions where the null hypothesis is correct. In an experiment with 90% power, one would expect to correctly identify nine of ten correct hypotheses (the area shaded red). Because the α value is defined as the probability of rejecting the null hypothesis when the null hypothesis is correct, one would also expect to incorrectly reject the null hypothesis on 4.5 of 90 occasions (the area shaded blue). As a result, with a 10% prior probability in an experiment with 90% power, a true-positive result is expected 67% of the time when the P value is 0.05

Fig. 2

Trials included in the clinician survey. RCT Randomized controlled trial