Closed-loop Neuromotor Training System Pairing Transcutaneous Vagus Nerve Stimulation with Video-based Real-time Movement Classification

Abstract

As an emerging neurostimulation for improving motor rehabilitation, applying vagus nerve stimulation (VNS) after successful movement during training facilitates motor recovery in animals with neuromotor impairment. To translate this procedure to human rehabilitation in a non-invasive, objective, and automated manner, real-time classification of movement quality on a trial-by-trial basis in a minimally constrained state is required. In this work, we developed an integrated closed-loop system using video-based real-time movement classification that can automatically trigger transcutaneous VNS (tVNS) wirelessly as soon as successful movement is detected. We also created a film-like conformable tVNS electrode to be attached over the outer ear. For movement training, we focused on the use case of dance therapy (backward walking), which is widely used for people with Parkinson's disease and older adults. Our markerless video analysis model could detect steps with 0.91 precision and 0.72 recall and classify successful backward steps with a 0.93 F1 score. The classification triggers tVNS through Bluetooth Low Energy communications with a trigger relay device we created. The integrated system enabled real-time automated classification and stimulation, triggering tVNS with 71.3% of the successful movements and taking 2.24 s from video capture to tVNS. We consider our work to be an important step toward patient-driven rehabilitation at home showcasing non-invasive, low-cost, and automated closed-loop neurostimulation technologies.

Figure 1:

Overall pipeline combining video-based backward walking analysis and movement-associated transcutanous vagus nerve stimulation (tVNS). The machine learning (ML) pipeline detects backward steps from the frontal view videos and classifies the successful and unsuccessful backward steps. When successful steps are detected, tVNS is triggered for stimulation. Here, we show the view from the side view to better illustrate the backward step in our study design.

Figure 2:

Camera and studio setup in the three locations of Ingleside Life-Plan Community

Figure 3:

An example showing the normalization of pose keypoints of participants with different heights during backward steps. The panels illustrate the normalization process for a tall person before (a) and after (c) normalization, and a short person before (b) and after (d) normalization. This helps our downstream ML models learn the movement characteristics invariant to subject’s heights and distances from the camera.

Figure 4:

An example showing the comparison of traditional peak detection and actual step occurrences. The red dots represent detected peaks corresponding to heel strikes (top panel). The detected backward steps (yellow windows in the middle panel) do not align accurately with the true backward steps (yellow regions in the bottom panel).

Figure 5:

Assigning weight transfer quality labels for the detected steps based on the overlaps with ground truth step windows with the corresponding labels.

Figure 6:

Top 10 most important features across all folds.

Figure 7:

Feature importance heat map by keypoint.

Figure 8:

81 features including all the angles and distances from the MediaPipe key-body points.

Figure 9:

An example showing the ground truth steps (top), detected steps before (middle) and after majority voting (bottom)

Figure 10:

The commercially-available gel-type electrode (left) and the developed film-type gold electrode (right) for tVNS.

Figure 11:

The developed trigger relay device (left: outside, right: inside)

Figure 12:

Configuration of the closed-loop neuromotor training system pairing tVNS (left) and movement classification (right). The arrows with a dotted line indicate wireless communication. Note that the trigger relay device (black box, left) is worn next to the stimulator in the photo for demonstration purposes, but it is intended to be worn on the other side for load balance.