Accurate prediction of clinical stroke scales and improved biomarkers of motor impairment from robotic measurements

Abstract

Objective: One of the greatest challenges in clinical trial design is dealing with the subjectivity and variability introduced by human raters when measuring clinical end-points. We hypothesized that robotic measures that capture the kinematics of human movements collected longitudinally in patients after stroke would bear a significant relationship to the ordinal clinical scales and potentially lead to the development of more sensitive motor biomarkers that could improve the efficiency and cost of clinical trials.

Materials and methods: We used clinical scales and a robotic assay to measure arm movement in 208 patients 7, 14, 21, 30 and 90 days after acute ischemic stroke at two separate clinical sites. The robots are low impedance and low friction interactive devices that precisely measure speed, position and force, so that even a hemiparetic patient can generate a complete measurement profile. These profiles were used to develop predictive models of the clinical assessments employing a combination of artificial ant colonies and neural network ensembles.

Results: The resulting models replicated commonly used clinical scales to a cross-validated R2 of 0.73, 0.75, 0.63 and 0.60 for the Fugl-Meyer, Motor Power, NIH stroke and modified Rankin scales, respectively. Moreover, when suitably scaled and combined, the robotic measures demonstrated a significant increase in effect size from day 7 to 90 over historical data (1.47 versus 0.67).

Discussion and conclusion: These results suggest that it is possible to derive surrogate biomarkers that can significantly reduce the sample size required to power future stroke clinical trials.

Fig 1. Dependence of the length of the ant’s path L (log scale) on the value of R².

As can be seen from this plot, L increases 10-fold as R² decreases by 0.2 units up to a R² value of ~0.2, and at a much greater rate for R² values lower than 0.2.

Fig 2. Schematic illustration of the ensemble neural network models used to predict the clinical scales from the RMK variables.

Each subset of features identified by the artificial ant algorithm was used to construct 10 independent neural network models using exactly the same network topology and training parameters but a different random seed number (and thus different initial synaptic parameters and presentation sequence of the training samples). The predictions of these 10 models were averaged to produce the aggregate prediction of the ensemble.

Fig 3. SPE map of the correlation distances of the clinical and RMK parameters for the completers cohort.

The map was derived by computing the pairwise Pearson correlation coefficients (R) for all pairs of features, converting them to correlation distances (1-abs(R)), and embedding the resulting matrix into 2 dimensions in such a way that the distances of the points on the map approximate as closely as possible the correlation distances of the respective features. The clinical parameters are highlighted in red, the RMK parameters on the affected side in blue, and the RMK parameters on the unaffected side in green. The map also shows distinct clusters of correlated variables which are preserved on both the affected and unaffected sides (outlined by green and blue ellipses, respectively).

Fig 4. Cross-validated R² of the best models derived from the completers (solid lines) and validated with the non-completers (dashed lines) for each of the four clinical scales, using 2, 4, 6, 8, 10, 12 and 14 robot-derived RMK features.

The figure shows the ability of the robot-derived RMK models to predict the clinical scales with an increasing number of features. The model performance exhibits an asymptotic behavior with respect to the number of RMK features, reaching the point of diminishing returns at approximately 8 features for all four clinical scales. Note the small variance in the prediction of the trained data as shown by the small “whiskers,” which for the most part, are not visible in the figure.

Fig 5. Cross-validated R² of the best 8-feature models derived using three different approaches for feature selection and model building: 1) features selected by artificial ants with neural networks, model derived with neural networks (dark blue); 2) features selected by artificial ants with neural networks, model derived with multi-linear regression (light blue); and 3) features selected by PCA, model derived with multi-linear regression (red).

Every model was cross-validated using the same 10-fold cross-validation procedure described in the Methods session.

Fig 6. Importance of individual RMK features in predicting the four clinical scales, obtained by systematically removing each feature from the training sample and repeating the feature selection, aggregation and cross-validation procedure for each derived data set.

Only models with 8 input and 2 hidden units are shown. The left-most data point and the horizontal solid line on the top part of the plot represent the cross-validated R² of the best model with all features included, averaged over all 10 cross-validation runs (standard deviations shown as error bars). Each subsequent point shows the R² of the corresponding apo model, i.e., the model derived by omitting the feature shown on the x axis (the model still includes 8 features, just not the one shown on the x axis). The individual markers at the bottom part of the plot indicate whether there is a statistically significant difference between the R² distributions of the all-feature and the respective apo models (the presence of a marker indicates that the difference between the two distributions is statistically significant, and the absence that it is not).

Fig 7. Optimization of effect size for robot-derived RMK metrics.

The horizontal lines show the day 7 to day 90 effect size for comparable patients of the historical VISTA data for the NIH, as well as the effect sizes for the NIH, FM and MP assessment scales for our completers cohort. The figure also shows the performance of the robot-derived RMK composites optimized for effect size for the trained (solid lines) and cross-validated sets (dashed lines). Note the increase of over 20% in cross-validated effect size for the RMK composites over the clinical scales with 4-features for this study (and over 70% over the historical data).

Fig 8. Correlation (expressed as R²) of the FM, MP and NIH original scales with the corresponding optimized 8-feature composites obtained with the greedy algorithm.