Identifying antinuclear antibody positive individuals at risk for developing systemic autoimmune disease: development and validation of a real-time risk model

Abstract

Objective: Positive antinuclear antibodies (ANAs) cause diagnostic dilemmas for clinicians. Currently, no tools exist to help clinicians interpret the significance of a positive ANA in individuals without diagnosed autoimmune diseases. We developed and validated a risk model to predict risk of developing autoimmune disease in positive ANA individuals.

Methods: Using a de-identified electronic health record (EHR), we randomly chart reviewed 2,000 positive ANA individuals to determine if a systemic autoimmune disease was diagnosed by a rheumatologist. A priori, we considered demographics, billing codes for autoimmune disease-related symptoms, and laboratory values as variables for the risk model. We performed logistic regression and machine learning models using training and validation samples.

Results: We assembled training (n = 1030) and validation (n = 449) sets. Positive ANA individuals who were younger, female, had a higher titer ANA, higher platelet count, disease-specific autoantibodies, and more billing codes related to symptoms of autoimmune diseases were all more likely to develop autoimmune diseases. The most important variables included having a disease-specific autoantibody, number of billing codes for autoimmune disease-related symptoms, and platelet count. In the logistic regression model, AUC was 0.83 (95% CI 0.79-0.86) in the training set and 0.75 (95% CI 0.68-0.81) in the validation set.

Conclusion: We developed and validated a risk model that predicts risk for developing systemic autoimmune diseases and can be deployed easily within the EHR. The model can risk stratify positive ANA individuals to ensure high-risk individuals receive urgent rheumatology referrals while reassuring low-risk individuals and reducing unnecessary referrals.

Keywords: antinuclear antibodies; autoimmune disease; electronic health record; rheumatology; risk model.

Figure 1

Timeline of model covariates. We assessed billing codes up to 5 years prior to the first positive antinuclear antibody (ANA) test. Laboratory values were assessed up to 1 year and 1 month after the ANA test. We conducted chart review for the model outcome of developing a systemic autoimmune disease diagnosed by a rheumatologist up to 10 years after the first positive ANA test.

Figure 2

Importance of Variables in ANA Risk Model. The list of variables in the final ANA risk model are shown to the left with p values to the right. The x axis shows variable importance using a Wald statistic. Ever-present antibody refers to having a disease-specific autoantibody such as a rheumatoid factor or dsDNA. ICD count refers to billing code category count that ranges from 0 to 9.

Figure 3

Model performance for training and validation sets. (A) shows ROC for the training set with an AUC 0.83 (95% CI 0.79-0.86). (B) shows calibration curve with a slope of 1 and intercept of 0 for the training set. Slopes that approach 1, as shown by the shaded grey line, demonstrate ideal calibration, agreement between predicted risk for systemic autoimmune disease and observed rate. (C) shows ROC for the validation set with an AUC 0.75 (95% CI 0.68-0.81). (D) shows calibration curve for the validation set. Calibration slope was equal to 0.71 and intercept was equal to 0.08.

Figure 4

Screenshot of Shiny app for risk model for systemic autoimmune disease. The screenshot shows the risk model covariates used to estimate risk for systemic autoimmune disease. This app demonstrates how the risk score is calculated and is not intended for clinical practice. The Shiny app can be accessed at the following link: https://cqs.app.vumc.org/shiny/AutoimmuneDiseasePrediction/.