Individualized Treatment Effects of Bougie versus Stylet for Tracheal Intubation in Critical Illness

Abstract

Rationale: A recent randomized trial found that using a bougie did not increase the incidence of successful intubation on first attempt in critically ill adults. The average effect of treatment in a trial population, however, may differ from effects for individuals. Objective: We hypothesized that application of a machine learning model to data from a clinical trial could estimate the effect of treatment (bougie vs. stylet) for individual patients based on their baseline characteristics ("individualized treatment effects"). Methods: This was a secondary analysis of the BOUGIE (Bougie or Stylet in Patients Undergoing Intubation Emergently) trial. A causal forest algorithm was used to model differences in outcome probabilities by randomized group assignment (bougie vs. stylet) for each patient in the first half of the trial (training cohort). This model was used to predict individualized treatment effects for each patient in the second half (validation cohort). Measurements and Main Results: Of 1,102 patients in the BOUGIE trial, 558 (50.6%) were the training cohort, and 544 (49.4%) were the validation cohort. In the validation cohort, individualized treatment effects predicted by the model significantly modified the effect of trial group assignment on the primary outcome (P value for interaction = 0.02; adjusted qini coefficient, 2.46). The most important model variables were difficult airway characteristics, body mass index, and Acute Physiology and Chronic Health Evaluation II score. Conclusions: In this hypothesis-generating secondary analysis of a randomized trial with no average treatment effect and no treatment effect in any prespecified subgroups, a causal forest machine learning algorithm identified patients who appeared to benefit from the use of a bougie over a stylet and from the use of a stylet over a bougie using complex interactions between baseline patient and operator characteristics.

Keywords: critical illness; intubation; machine learning; prediction models.

Figure 1.

Variable importance plot. This figure displays the 10 most important causal forest model variables, as determined by the number of times a candidate partitioning variable was chosen to be in the first splits of a tree in the causal forest model. x-axis scale of importance was normalized to 100% for the most important variable. Difficult airway characteristics included any of the following: vomiting; witnessed aspiration; upper GI bleeding; epistaxis or oral bleeding; upper airway mass, infection, or trauma; head and neck radiation; limited neck mobility; limited mouth opening; history of obstructive sleep apnea; or other. APACHE = Acute Physiology and Chronic Health Evaluation II Score; BMI = body mass index; SBP = systolic blood pressure; Sp_O2 = saturation of peripheral oxygen.

Figure 2.

Observed treatment effect in the validation cohort by predicted treatment effect quartile from a causal forest model. Patients in the validation cohort are grouped into quartiles by their individual predicted treatment effect from the causal forest model, ranging from the quartile predicted to most benefit from use of a bougie (Q1) to the quartile predicted to most benefit from use of a stylet (Q4). The observed average treatment effect, overall and in each quartile, is the difference in the incidence of the primary outcome (successful intubation on the first attempt) between the bougie group and the stylet group. Bars indicate 95% confidence intervals. The interaction between predicted treatment effect quartile and the effect of trial group assignment on the primary outcome was significant (P = 0.02).

Figure 3.

Qini plot. This figure depicts the discrimination of the causal forest model in the validation cohort. The difference between the solid line (bougie vs. stylet selected for patients based on predicted individualized treatment effect from the model) versus the dotted line (bougie vs. stylet selected randomly) demonstrates the uplift gain, defined as the difference between the areas under the curve plotted by the model-based targeting and random targeting. Consistent with the high discrimination of the model, the qini curve first increases (showing that the patients for whom the model predicted the largest treatment effect with a use of the bougie experienced the largest benefit from use of a bougie) then plateaus (as the population begin to include patients with similar outcomes with either bougie or stylet), and finally decreases (showing that the patients for whom the model predicted the largest treatment effect with use of a stylet experienced the largest benefit from use of a stylet). Adj = adjusted.

Figure 4.

Partial dependence plot. This figure depicts the change in predicted benefit over ranges for the most important continuous predictors. The x-axis shows the change in the variable of interest, and the y-axis shows the direction of benefit. Bootstrapped 95% confidence intervals are shown in gray. APACHE = Acute Physiology and Chronic Health Evaluation II score; BMI = body mass index; ITE = individualized treatment effect; SBP = systolic blood pressure.

Figure 5.

Individual patient examples. This figure depicts the influence of individual variables on the predicted treatment effect for bougie versus stylet for two individual patients in the validation cohort. For each variable on the y-axis, the value for that patient is presented compared with the median value for the cohort in parentheses. The x-axis shows the model’s predicted treatment effect with bougie versus stylet for the patient, with 0.0 representing no difference in successful intubation on the first attempt between use of a bougie and use of a stylet. Blue arrows signify variables that make benefit from use of a bougie more likely, and red arrows signify variables that make benefit from use of a stylet more likely. (A) Information for a patient whose predicted individualized treatment effect (ITE) of 0.124 was consistent with a 12.4% absolute increase in the incidence of successful intubation on the first attempt in the bougie group compared with the stylet group. (B) Information for a patient whose ITE of −0.084 was consistent with an 8.4% absolute decrease in the incidence of successful intubation on the first attempt in the bougie group compared with the stylet group. APACHE II = Acute Physiology and Chronic Health Evaluation II.