Development of a Machine Learning Model Using Electronic Health Record Data to Identify Antibiotic Use Among Hospitalized Patients

Abstract

Importance: Comparisons of antimicrobial use among hospitals are difficult to interpret owing to variations in patient case mix. Risk-adjustment strategies incorporating larger numbers of variables haves been proposed as a method to improve comparisons for antimicrobial stewardship assessments.

Objective: To evaluate whether variables of varying complexity and feasibility of measurement, derived retrospectively from the electronic health records, accurately identify inpatient antimicrobial use.

Design, setting, and participants: Retrospective cohort study, using a 2-stage random forests machine learning modeling analysis of electronic health record data. Data were split into training and testing sets to measure model performance using area under the curve and absolute error. All adult and pediatric inpatient encounters from October 1, 2015, to September 30, 2017, at 2 community hospitals and 1 academic medical center in the Duke University Health System were analyzed. A total of 204 candidate variables were categorized into 4 tiers based on feasibility of measurement from the electronic health records.

Main outcomes and measures: Antimicrobial exposure was measured at the encounter level in 2 ways: binary (ever or never) and number of days of therapy. Analyses were stratified by age (pediatric or adult), unit type, and antibiotic group.

Results: The data set included 170 294 encounters and 204 candidate variables from 3 hospitals during the 3-year study period. Antimicrobial exposure occurred in 80 190 encounters (47%); 64 998 (38%) received 1 to 6 days of therapy, and 15 192 (9%) received 7 or more days of therapy. Two-stage models identified antimicrobial use with high fidelity (mean area under the curve, 0.85; mean absolute error, 1.0 days of therapy). Addition of more complex variables increased accuracy, with largest improvements occurring with inclusion of diagnosis information. Accuracy varied based on location and antibiotic group. Models underestimated the number of days of therapy of encounters with long lengths of stay.

Conclusions and relevance: Models using variables derived from electronic health records identified antimicrobial exposure accurately. Future risk-adjustment strategies incorporating encounter-level information may make comparisons of antimicrobial use more meaningful for hospital antimicrobial stewardship assessments.

Figure 1.. Model Performance When Identifying Ever Receiving an Antimicrobial During the Encounter by Age, Location, Antimicrobial Group, and Input Variable Feasibility Tier

A, Model output for adult encounters. B, Model output for pediatric and neonatal encounters. Each data point represents a unique model built based on location, feasibility tier of variables used, antimicrobial group, and adult or pediatric populations. The closer the AUC value is to 1, the better the model was at classifying whether antimicrobials were administered. Some location strata in the analysis of pediatric encounters were too small to fit a model. In these scenarios, only estimates for the “all locations” category were reported. Antimicrobial groups and agents are listed in eTable 1 in the Supplement. AUC indicates area under the curve; BL, beta-lactam; CDI, Clostridioides difficile infection risk agents; CO, community onset; and ICU, intensive care unit.

Figure 2.. Model Performance When Identifying Days of Therapy of Antimicrobials During the Encounter by Age, Location, Antimicrobial Group, and Input Variable Feasibility Tier

A, Model output for adult encounters. B, Model output for pediatric and neonatal encounters. Each data point represents a unique model built based on location, feasibility tier of variables used, antimicrobial group, and adult or pediatric populations. The closer the mean absolute error is to 0, the better the model was at estimating the number of days of antimicrobial therapy. Some location strata in the analysis of pediatric encounters were too small to fit a model. In these scenarios, only estimates for the “all locations” category were reported. Antimicrobial groups and agents are listed in eTable 1 in the Supplement. BL indicates beta-lactam; CDI, Clostridioides difficile infection risk agents; CO, community onset; and ICU, intensive care unit.