Concurrent validity of machine learning-classified functional upper extremity use from accelerometry in chronic stroke

Abstract

Objective: This study aims to investigate the validity of machine learning-derived amount of real-world functional upper extremity (UE) use in individuals with stroke. We hypothesized that machine learning classification of wrist-worn accelerometry will be as accurate as frame-by-frame video labeling (ground truth). A second objective was to validate the machine learning classification against measures of impairment, function, dexterity, and self-reported UE use. Design: Cross-sectional and convenience sampling. Setting: Outpatient rehabilitation. Participants: Individuals (>18 years) with neuroimaging-confirmed ischemic or hemorrhagic stroke >6-months prior (n = 31) with persistent impairment of the hemiparetic arm and upper extremity Fugl-Meyer (UEFM) score = 12-57. Methods: Participants wore an accelerometer on each arm and were video recorded while completing an "activity script" comprising activities and instrumental activities of daily living in a simulated apartment in outpatient rehabilitation. The video was annotated to determine the ground-truth amount of functional UE use. Main outcome measures: The amount of real-world UE use was estimated using a random forest classifier trained on the accelerometry data. UE motor function was measured with the Action Research Arm Test (ARAT), UEFM, and nine-hole peg test (9HPT). The amount of real-world UE use was measured using the Motor Activity Log (MAL). Results: The machine learning estimated use ratio was significantly correlated with the use ratio derived from video annotation, ARAT, UEFM, 9HPT, and to a lesser extent, MAL. Bland-Altman plots showed excellent agreement between use ratios calculated from video-annotated and machine-learning classification. Factor analysis showed that machine learning use ratios capture the same construct as ARAT, UEFM, 9HPT, and MAL and explain 83% of the variance in UE motor performance. Conclusion: Our machine learning approach provides a valid measure of functional UE use. The accuracy, validity, and small footprint of this machine learning approach makes it feasible for measurement of UE recovery in stroke rehabilitation trials.

Keywords: ADLs; accelerometry; disability evaluation; machine learning; paresis/rehabilitation; psychometrics; sensors; stroke rehabilitation.

FIGURE 1

Simulated apartment in outpatient rehabilitation. (A) Laundry task. (B) Shopping task. (C) Kitchen task. (D) Bed-making and laundry-folding tasks. Activity scripts are completed using the facilities at MedStar National Rehabilitation Hospital, Washington DC.

FIGURE 2

Scatter plots of video-labeled or estimated use ratios with (A) ARAT and (B) MAL.

FIGURE 3

Bland–Altman plots. Difference in use ratios calculated using machine learning versus video-labeled accelerometry data are shown on the y-axis. The mean of use ratios calculated using machine learning and video-labeled accelerometry are shown on the x-axis. Solid gray line shows mean difference (mean ± SD = 0.004 ± 0.0.04) between use ratios calculated by the two methods; solid red lines show the 95% CI for the mean difference.

FIGURE 4

Component plot in rotated factor space. A single component accounts for 83% of the variance in UE behavioral measures of UEFM, ARAT, nine-hole peg test, self-reported MAL, and the use ratios. This is evidenced on the component plot. Forcing a 2-component solution results in MAL and nine-hole peg test scores splitting from the UEFM, ARAT, and use ratios, evidenced by MAL falling closer to component 1 axis, as shown here.