Performance of machine learning models for predicting high-severity symptoms in multiple sclerosis

Abstract

Current care in multiple sclerosis (MS) primarily relies on infrequently obtained data such as magnetic resonance imaging, clinical laboratory tests or clinical history, resulting in subtle changes that may occur between visits being missed. Mobile technology enables continual collection of data and can pave the path for predicting complex aspects of MS such as symptoms and disease courses. To this end, we conducted a first-of-its-kind observational study called MS Mosaic. First, we developed and publicly launched a mobile app for collecting longitudinal data from MS subjects in the United States. Second, we ran the study across 3 years in order to capture complex patterns for this slow progressing disease. Finally, we retrospectively developed three classical ML methods and two deep learning models to accurately and continually predict the incidence of five high-severity symptoms (fatigue, sensory disturbance, walking instability, depression or anxiety and cramps/spasms) three months in advance.

Figure 1

Receiver operating characteristic curves (left column) and calibration curves (right column) for machine learning models (gradient boosting classifier, logistic regression, multi-layer perceptron, recurrent neural network, and temporal convolutional network) for predicting whether the median value of a user-reported symptom will be of high-severity ($>2$) in the next three months. The symptoms considered are (A) cramps, (B) depression or anxiety, (C) fatigue, (D) sensory disturbance, and (E) walking instability or coordination problems respectively.

Figure 2

Area under the receiver operating characteristic curve (AUROC) obtained by Gradient boosting classifier (GBC) for predicting occurrence of high-severity symptoms on different subgroups of the data: (A) four subtypes of multiple sclerosis and (B) three age groups. We include the 95% confidence intervals as error-bars. The bar representing “age $<30$” for Cramps is missing from the figure due to the absence of participants in that intersection.

Figure 3

Area under the receiver operating characteristic curve (AUROC) achieved by Gradient Boosting Classifier (GBC) while predicting the incidence of high-severity symptoms on different groups of features: symptoms, demographics (only included age in our case), functional tests, passive signals, active features (combination of symptoms and functional tests), and lastly a set with all features. 95% confidence intervals have been included as error-bars.

Figure 4

Study diagram showcasing all the steps from inception to research output.