Short-term mortality prediction for acute lung injury patients: external validation of the Acute Respiratory Distress Syndrome Network prediction model

Abstract

Objective: An independent cohort of patients with acute lung injury was used to evaluate the external validity of a simple prediction model for short-term mortality previously developed using data from Acute Respiratory Distress Syndrome Network (ARDSNet) trials.

Design: Data for external validation were obtained from a prospective cohort study of patients with acute lung injury.

Setting: Thirteen intensive care units at four teaching hospitals in Baltimore, MD.

Patients: Five hundred and eight nontrauma patients with acute lung injury.

Interventions: None.

Measurements and main results: Of the 508 patients eligible for this analysis, 234 (46%) died inhospital. Discrimination of the ARDSNet prediction model for inhospital mortality, evaluated by the area under the receiver operator characteristic curves, was 0.67 for our external validation data set vs. 0.70 and 0.68 using Acute Physiology and Chronic Health Evaluation II and the ARDSNet validation data set, respectively. In evaluating calibration of the model, predicted vs. observed inhospital mortality for the external validation data set was similar for both low-risk (ARDSNet model score = 0) and high-risk (score = 3 or 4+) patient strata. However, for intermediate-risk (score = 1 or 2) patients, observed inhospital mortality was substantially higher than predicted mortality (25.3% vs. 16.5% and 40.6% vs. 31.0% for score = 1 and 2, respectively). Sensitivity analyses limiting our external validation data set to only those patients meeting the ARDSNet trial eligibility criteria and to those who received mechanical ventilation in compliance with the ARDSNet ventilation protocol did not substantially change the model's discrimination or improve its calibration.

Conclusions: Evaluation of the ARDSNet prediction model using an external acute lung injury cohort demonstrated similar discrimination of the model as was observed with the ARDSNet validation data set. However, there were substantial differences in observed vs. predicted mortality among intermediate-risk patients with acute lung injury. The ARDSNet model provided reasonable, but imprecise, estimates of predicted mortality when applied to our external validation cohort of patients with acute lung injury.

Figure. Receiver operating characteristics (ROC) curves

Using data from the external validation model, this Figure compares the ROC curves for the ARDSNet prediction model versus the Acute Physiology Assessment and Chronic Health Evaluation II score, respectively, for in-hospital mortality (area under curve = 0.67 vs. 0.70, p=0.207) in Panel A, and for 28-day mortality (0.64 vs. 0.70, p=0.039) in Panel B.