Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2020 Dec;48(12):1720-1728.
doi: 10.1097/CCM.0000000000004612.

Observational Research for Therapies Titrated to Effect and Associated With Severity of Illness: Misleading Results From Commonly Used Statistical Methods

Harm-Jan de Grooth  1   2 , Armand R J Girbes  1 , Fleur van der Ven  3 , Heleen M Oudemans-van Straaten  1 , Pieter R Tuinman  1 , Angélique M E de Man  1
Affiliations

Observational Research for Therapies Titrated to Effect and Associated With Severity of Illness: Misleading Results From Commonly Used Statistical Methods

Harm-Jan de Grooth et al. Crit Care Med. 2020 Dec.

Abstract

Objectives: In critically ill patients, treatment dose or intensity is often related to severity of illness and mortality risk, whereas overtreatment or undertreatment (relative to the individual need) may further increase the odds of death. We aimed to investigate how these relationships affect the results of common statistical methods used in observational studies.

Design: Using Monte Carlo simulation, we generated data for 5,000 patients with a treatment dose related to the pretreatment mortality risk but with randomly distributed overtreatment or undertreatment. Significant overtreatment or undertreatment (relative to the optimal dose) further increased the mortality risk. A prognostic score that reflects the mortality risk and an outcome of death or survival was then generated. The study was analyzed: 1) using logistic regression to estimate the effect of treatment dose on outcome while controlling for prognostic score and 2) using propensity score matching and inverse probability weighting of the effect of high treatment dose on outcome. The data generation and analyses were repeated 1,500 times over sample sizes between 200 and 30,000 patients, with an increasing accuracy of the prognostic score and with different underlying assumptions.

Setting: Computer-simulated studies.

Measurements and main results: In the simulated 5,000-patient observational study, higher treatment dose was found to be associated with increased odds of death (p = 0.00001) while controlling for the prognostic score with logistic regression. Propensity-matched analysis led to similar results. Larger sample sizes led to equally biased estimates with narrower CIs. A perfect risk predictor negated the bias only under artificially perfect assumptions.

Conclusions: When a treatment dose is associated with severity of illness and should be dosed "enough," logistic regression, propensity score matching, and inverse probability weighting to adjust for confounding by severity of illness lead to biased results. Larger sample sizes lead to more precisely wrong estimates.

PubMed Disclaimer

Comment in

References

    1. Zhang Z, Hong Y, Liu N: Scientific evidence underlying the recommendations of critical care clinical practice guidelines: A lack of high level evidence. Intensive Care Med. 2018; 44:10–12.
    1. Sjoding MW, Luo K, Miller MA, et al.: When do confounding by indication and inadequate risk adjustment bias critical care studies? A simulation study. Crit Care. 2015; 19:195.
    1. Salluh JI, Soares M: ICU severity of illness scores: APACHE, SAPS and MPM. Curr Opin Crit Care. 2014; 20:557–565.
    1. Kuzniewicz MW, Vasilevskis EE, Lane R, et al.: Variation in ICU risk-adjusted mortality: Impact of methods of assessment and potential confounders. Chest. 2008; 133:1319–1327.
    1. Cook NR: Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation. 2007; 115:928–935.

Publication types

LinkOut - more resources