When do confounding by indication and inadequate risk adjustment bias critical care studies? A simulation study

Abstract

Introduction: In critical care observational studies, when clinicians administer different treatments to sicker patients, any treatment comparisons will be confounded by differences in severity of illness between patients. We sought to investigate the extent that observational studies assessing treatments are at risk of incorrectly concluding such treatments are ineffective or even harmful due to inadequate risk adjustment.

Methods: We performed Monte Carlo simulations of observational studies evaluating the effect of a hypothetical treatment on mortality in critically ill patients. We set the treatment to have either no association with mortality or to have a truly beneficial effect, but more often administered to sicker patients. We varied the strength of the treatment's true effect, strength of confounding, study size, patient population, and accuracy of the severity of illness risk-adjustment (area under the receiver operator characteristics curve, AUROC). We measured rates in which studies made inaccurate conclusions about the treatment's true effect due to confounding, and the measured odds ratios for mortality for such false associations.

Results: Simulated observational studies employing adequate risk-adjustment were generally able to measure a treatment's true effect. As risk-adjustment worsened, rates of studies incorrectly concluding the treatment provided no benefit or harm increased, especially when sample size was large (n = 10,000). Even in scenarios of only low confounding, studies using the lower accuracy risk-adjustors (AUROC < 0.66) falsely concluded that a beneficial treatment was harmful. Measured odds ratios for mortality of 1.4 or higher were possible when the treatment's true beneficial effect was an odds ratio for mortality of 0.6 or 0.8.

Conclusions: Large observational studies confounded by severity of illness have a high likelihood of obtaining incorrect results even after employing conventionally "acceptable" levels of risk-adjustment, with large effect sizes that may be construed as true associations. Reporting the AUROC of the risk-adjustment used in the analysis may facilitate an evaluation of a study's risk for confounding.

Figure 1

Distributions of risk of death among the patients used in the Monte Carlo simulations. The gray distribution represents the baseline risk of death while the blue distribution represents the risk after administration of a treatment with an odds ratio of 0.6 for mortality.

Figure 2

Rates of falsely concluding a safe treatment (odds ratio = 1.0) caused statistically significant harm among simulated cohort studies. (A) Rates in studies of n = 1,000 and (B) rates in studies of n = 10,000. AUROC, area under the receiver operator characteristic curve.

Figure 3

Rates of falsely concluding a beneficial treatment (odds ratio = 0.8) caused no benefit (false negative) or statistically significant harm (false harm) among simulated cohort studies. (A) Rates in low confounding scenarios of n = 1,000. (B) Rates in high confounding scenarios of n = 1,000. (C) Rates in low confounding scenarios of n = 10,000. (D) Rates in high confounding scenarios of n = 10,000. AUROC, area under the receiver operator characteristic curve.

Figure 4

Relationship between the true treatment effect, risk-adjuster accuracy, and measured treatment effect in low and high confounding scenarios. AUROC, area under the receiver operator characteristic curve.

Figure 5

Distribution of measured effect sizes in simulated studies when sample size is 1,000. The center line of the box represents the 50^th percentile odds ratio, the box extends from the 25^th percentile to the 75^th percentile, and whiskers extend from the 2.5^th and 97.5^th percentile (representing 95% confidence intervals). (A) Low confounding scenarios. (B) High confounding scenarios. AUROC, area under the receiver operator characteristic curve.