Automated and flexible identification of complex disease: building a model for systemic lupus erythematosus using noisy labeling

Abstract

Accurate and efficient identification of complex chronic conditions in the electronic health record (EHR) is an important but challenging task that has historically relied on tedious clinician review and oversimplification of the disease. Here we adapt methods that allow for automated "noisy labeling" of positive and negative controls to create a "silver standard" for machine learning to automate identification of systemic lupus erythematosus (SLE). Our final model, which includes both structured data as well as text processing of clinical notes, outperformed all existing algorithms for SLE (AUC 0.97). In addition, we demonstrate how the probabilistic outputs of this model can be adapted to various clinical needs, selecting high thresholds when specificity is the priority and lower thresholds when a more inclusive patient population is desired. Deploying a similar methodology to other complex diseases has the potential to dramatically simplify the landscape of population identification in the EHR.

Mesh terms: Electronic Health Records, Machine Learning, Lupus Erythematosus, Phenotype, Algorithms.

Figure 1.

Overview of methodology for training and test set development.

Figure 2.

Algorithmic probabilities assigned to test set data. Data are shown in 0.05 sized probability bins and labeled according to how each case was classified by clinician expert review. Patients were classified as “not SLE,” “possible SLE” (diagnostic uncertainty or lack of documentation were present), “probable SLE” (features of the disease were present but not meeting full criteria), or “definite SLE” meeting American College of Rheumatology (ACR) criteria.

Figure 3.

ROC curves for “strict” and “inclusive” definitions of SLE. A) ROC curve for the strict definition of SLE shows an AUC 0.97. B) ROC curve for the inclusive definition of SLE shows an AUC of 0.94.