Validation of a machine learning approach to estimate expanded disability status scale scores for multiple sclerosis

Abstract

Background: Disability assessment using the Expanded Disability Status Scale (EDSS) is important to inform treatment decisions and monitor the progression of multiple sclerosis. Yet, EDSS scores are documented infrequently in electronic medical records.

Objective: To validate a machine learning model to estimate EDSS scores for multiple sclerosis patients using clinical notes from neurologists.

Methods: A machine learning model was developed to estimate EDSS scores on specific encounter dates using clinical notes from neurologist visits. The OM1 MS Registry data were used to create a training cohort of 2632 encounters and a separate validation cohort of 857 encounters, all with clinician-recorded EDSS scores. Model performance was assessed using the area under the receiver-operating-characteristic curve (AUC), positive predictive value (PPV), and negative predictive value (NPV), calculated using a binarized version of the outcome. The Spearman R and Pearson R values were calculated. The model was then applied to encounters without clinician-recorded EDSS scores in the MS Registry.

Results: The model had a PPV of 0.85, NPV of 0.85, and AUC of 0.91. The model had a Spearman R value of 0.75 and Pearson R value of 0.74 when evaluating performance using the continuous estimated EDSS and clinician-recorded EDSS scores. Application of the model to eligible encounters resulted in the generation of eEDSS scores for an additional 190,282 encounters from 13,249 patients.

Conclusion: EDSS scores can be estimated with very good performance using a machine learning model applied to clinical notes, thus increasing the utility of real-world data sources for research purposes.

Keywords: Multiple sclerosis; disability evaluation; health services research; machine learning; outcome assessment; registries.

Figure 1.

The area under the receiver-operating-characteristic curve (AUC). The AUC was calculated using a binarized version of the outcome in which the positive class is defined as those notes with scores greater or equal to 6 (the threshold at which EDSS scores reflects the requirement for ambulatory aid), and the negative class is defined as those records with scores less than or equal to 5.5.

Figure 2.

Distribution of estimated and clinician-recorded EDSS scores in the validation cohort. The distribution of eEDSS scores was compared to the distribution of clinician-recorded EDSS scores.

Figure 3.

Confusion matrix showing agreement between estimated and clinician-recorded EDSS scores in the validation cohort. A confusion matrix was generated to further assess the agreement between the eEDSS scores and clinician-recorded EDSS scores.

Figure 4.

Distribution of estimated and clinician-recorded EDSS scores in the validation cohort. The distribution of eEDSS scores for eligible encounters in the MS Registry was compared to the distribution of clinician-recorded EDSS scores in the validation cohort, with the scores on the x-axis and the percentage of total encounters on the y-axis.