Transparent Quality Optimization for Machine Learning-Based Regression in Neurology

Abstract

The clinical monitoring of walking generates enormous amounts of data that contain extremely valuable information. Therefore, machine learning (ML) has rapidly entered the research arena to analyze and make predictions from large heterogeneous datasets. Such data-driven ML-based applications for various domains become increasingly applicable, and thus their software qualities are taken into focus. This work provides a proof of concept for applying state-of-the-art ML technology to predict the distance travelled of the 2-min walk test, an important neurological measurement which is an indicator of walking endurance. A transparent lean approach was emphasized to optimize the results in an explainable way and simultaneously meet the specified software requirements for a generic approach. It is a general-purpose strategy as a fractional−factorial design benchmark combined with standardized quality metrics based on a minimal technology build and a resulting optimized software prototype. Based on 400 training and 100 validation data, the achieved prediction yielded a relative error of 6.1% distributed over multiple experiments with an optimized configuration. The Adadelta algorithm (LR=0.000814, fModelSpread=5, nModelDepth=6, nepoch=1000) performed as the best model, with 90% of the predictions with an absolute error of <15 m. Factors such as gender, age, disease duration, or use of walking aids showed no effect on the relative error. For multiple sclerosis patients with high walking impairment (EDSS Ambulation Score ≥6), the relative difference was significant (n=30; 24.0%; p<0.050). The results show that it is possible to create a transparently working ML prototype for a given medical use case while meeting certain software qualities.

Keywords: deep learning; fractional factorial design benchmark; inertial measurement units; machine learning; multiple sclerosis; software quality.

Figure 1

Minimal viable ML-based approach for 2MWT prediction. After aggregating MLS data and manually measured walking distances from PwMS, the data are transposed to a table-based (columns $P_{1..\max }$: patients; rows: speed, …, 2MW: features and learning objective), thus ML-compatible representation, prepared, split and fed into a DFFNN based on TensorFlow. The model training bases on incrementally improved configurations (FFDB) to optimize the prediction quality, expressed by predefined metrics. [Abbreviations: ML = Machine Learning; 2MWT = 2 minute Walk Test; MLS = Mobility Lab System; PwMS = People with Multiple Sclerosis; DFFNN = Deep Feed Forward Neural Network; FFDB = Fractional Factorial Design Benchmark].

Figure 2

$LR$ Optimization. $MS{E}_{\mathrm{arr}}$ as prediction quality ((a) normal, (b) moving average and marked minima) in dependence of $LR$s for 7 ML training algorithms, optimizing DFFNNs of fixed shape [Abbreviations: LR = Learning Rate, ML = Machine Learning, DFFNN = Deep Feed Forward Neural Network].

Figure 3

Model Shape Optimization. (a) Optimizer = Adadelta; Average, Relative RMSE for Model Spread and Depth (%); (b) Optimizer = Adadelta; Average, Relative SD for Model Spread and Depth (%); (c) Optimizer = RMSProp; Average, Relative RMSE for Model Spread and Depth (%); (d) Optimizer = RMSProp; Average, Relative SD for Model Spread and Depth (%) $MS{E}_{\mathrm{arr}}$ as prediction quality (c) and its SD (d) in dependence of $f_{\mathrm{ModelSpread}}$ and $n_{\mathrm{ModelDepth}}$ for the Adadelta (best case) and the RMSProp algorithm (worst case); values are scaled for better readability [Abbreviations: RMSE = Rooted Mean Square Error; SD = Standard Deviation].

Figure 4

(a) Optimization algorithm comparison. $MS{E}_{\mathrm{arr}}$ as prediction quality and its SD for each algorithm; (b,c) Prediction distribution; individual results of the best regressor model after completing the FFDB as total error [Abbreviations: SD = Standard Deviation; FFDB = Fractional Factorial Design Benchmark; LR = Learning Rate].