Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2019 Mar 1;2(3):e190606.
doi: 10.1001/jamanetworkopen.2019.0606.

Assessment of a Deep Learning Model Based on Electronic Health Record Data to Forecast Clinical Outcomes in Patients With Rheumatoid Arthritis

Beau Norgeot  1 , Benjamin S Glicksberg  1 , Laura Trupin  2 , Dmytro Lituiev  1 , Milena Gianfrancesco  2 , Boris Oskotsky  1 , Gabriela Schmajuk  2   3 , Jinoos Yazdany  2 , Atul J Butte  1   4
Affiliations

Assessment of a Deep Learning Model Based on Electronic Health Record Data to Forecast Clinical Outcomes in Patients With Rheumatoid Arthritis

Beau Norgeot et al. JAMA Netw Open. .

Abstract

Importance: Knowing the future condition of a patient would enable a physician to customize current therapeutic options to prevent disease worsening, but predicting that future condition requires sophisticated modeling and information. If artificial intelligence models were capable of forecasting future patient outcomes, they could be used to aid practitioners and patients in prognosticating outcomes or simulating potential outcomes under different treatment scenarios.

Objective: To assess the ability of an artificial intelligence system to prognosticate the state of disease activity of patients with rheumatoid arthritis (RA) at their next clinical visit.

Design, setting, and participants: This prognostic study included 820 patients with RA from rheumatology clinics at 2 distinct health care systems with different electronic health record platforms: a university hospital (UH) and a public safety-net hospital (SNH). The UH and SNH had substantially different patient populations and treatment patterns. The UH has records on approximately 1 million total patients starting in January 2012. The UH data for this study were accessed on July 1, 2017. The SNH has records on 65 000 unique individuals starting in January 2013. The SNH data for the study were collected on February 27, 2018.

Exposures: Structured data were extracted from the electronic health record, including exposures (medications), patient demographics, laboratories, and prior measures of disease activity. A longitudinal deep learning model was used to predict disease activity for patients with RA at their next rheumatology clinic visit and to evaluate interhospital performance and model interoperability strategies.

Main outcomes and measures: Model performance was quantified using the area under the receiver operating characteristic curve (AUROC). Disease activity in RA was measured using a composite index score.

Results: A total of 578 UH patients (mean [SD] age, 57 [15] years; 477 [82.5%] female; 296 [51.2%] white) and 242 SNH patients (mean [SD] age, 60 [15] years; 195 [80.6%] female; 30 [12.4%] white) were included in the study. Patients at the UH compared with those at the SNH were seen more frequently (median time between visits, 100 vs 180 days) and were more frequently prescribed higher-class medications (biologics) (364 [63.0%] vs 70 [28.9%]). At the UH, the model reached an AUROC of 0.91 (95% CI, 0.86-0.96) in a test cohort of 116 patients. The UH-trained model had an AUROC of 0.74 (95% CI, 0.65-0.83) in the SNH test cohort (n = 117) despite marked differences in the patient populations. In both settings, baseline prediction using each patients' most recent disease activity score had statistically random performance.

Conclusions and relevance: The findings suggest that building accurate models to forecast complex disease outcomes using electronic health record data is possible and these models can be shared across hospitals with diverse patient populations.

PubMed Disclaimer

Conflict of interest statement

Conflict of Interest Disclosures: Ms Trupin reported receiving grants from the National Institute of Arthritis and Musculoskeletal and Skin Diseases (NIAMS), National Institutes of Health (NIH) outside the submitted work. Dr Lituiev reported receiving grants from the NIH during the conduct of the study and personal fees and nonfinancial support from Nvidia outside the submitted work. Dr Gianfrancesco reported receiving grants from NIAMS of the NIH outside the submitted work. Dr Schmajuk reported receiving grants from the Agency for Healthcare Research and Quality and Russell/Engleman Medical Research Center for Arthritis during the conduct of the study, grants from Pfizer (investigator initiated), and other support from the American College of Rheumatology outside the submitted work. Dr Yazdany reported receiving grants from Pfizer outside the submitted work. Dr Butte reported receiving grants and personal fees from the NIH and Genentech; grants from the Bakar Family and Priscilla Chan and Mark Zuckerberg during the conduct of the study; personal fees from Merck & Co., Eli Lilly and Company, Roche, Pfizer, Bayer, the American Association of Allergy, Asthma, and Immunology, and the Federation of Clinical Immunology Societies; and other support from Google, Microsoft, Apple, Amazon, and CVS outside the submitted work. No other disclosures were reported.

Figures

Figure 1.
Figure 1.. Forecasting Performance of the Deep Learning Models in the University Hospital (UH) Cohort
The distribution of outcomes from the training cohort at UH was 60% controlled and 40% uncontrolled according to the clinical disease activity index. This was previously used to train the outcome posterior classifier at UH (area under the receiver operating characteristic curve [AUROC], 0.535). The likelihood of switching outcomes between visits within the training cohort was 25%. This was used previously to train the change posterior classifier at UH (AUROC, 0.554). Deep Learning produced the best results (AUROC, 0.912).
Figure 2.
Figure 2.. Confusion Plot
Confusion plot consisting of the final embedding of the model, the learned patient trajectory vectors, visualized using t-distributed stochastic neighbor embedding, with colors according to the ground truth of the patients outcome at their next visit. The model places observations onto a 1-dimension manifold with controlled and uncontrolled outcomes clustering along different ends of the manifold.
Figure 3.
Figure 3.. Forecasting Performance of the Deep Learning Models in the Safety-Net Hospital Cohort
UCSF indicates University of California, San Francisco; ZSFG, Zuckerberg San Francisco General Hospital.
See this image and copyright information in PMC

References

    1. Spector TD. Rheumatoid arthritis. Rheum Dis Clin North Am. 1990;16(3):-. - PubMed
    1. Singh JA, Saag KG, Bridges SL Jr, et al. ; American College of Rheumatology . 2015 American College of Rheumatology guideline for the treatment of rheumatoid arthritis. Arthritis Care Res (Hoboken). 2016;68(1):1-25. doi:10.1002/acr.22783 - DOI - PubMed
    1. Yazdany J, Bansback N, Clowse M, et al. . Rheumatology Informatics System for Effectiveness: a national informatics-enabled registry for quality improvement. Arthritis Care Res (Hoboken). 2016;68(12):1866-1873. doi:10.1002/acr.23089 - DOI - PMC - PubMed
    1. Harrell FE Jr, Lee KL, Pollock BG. Regression models in clinical studies: determining relationships between predictors and response. J Natl Cancer Inst. 1988;80(15):1198-1202. doi:10.1093/jnci/80.15.1198 - DOI - PubMed
    1. Strobl C, Boulesteix AL, Zeileis A, Hothorn T. Bias in random forest variable importance measures: illustrations, sources and a solution. BMC Bioinformatics. 2007;8:25. doi:10.1186/1471-2105-8-25 - DOI - PMC - PubMed

Publication types