Feeling the beat: a smart hand exoskeleton for learning to play musical instruments

Abstract

Individuals who have suffered neurotrauma like a stroke or brachial plexus injury often experience reduced limb functionality. Soft robotic exoskeletons have been successful in assisting rehabilitative treatment and improving activities of daily life but restoring dexterity for tasks such as playing musical instruments has proven challenging. This research presents a soft robotic hand exoskeleton coupled with machine learning algorithms to aid in relearning how to play the piano by 'feeling' the difference between correct and incorrect versions of the same song. The exoskeleton features piezoresistive sensor arrays with 16 taxels integrated into each fingertip. The hand exoskeleton was created as a single unit, with polyvinyl acid (PVA) used as a stent and later dissolved to construct the internal pressure chambers for the five individually actuated digits. Ten variations of a song were produced, one that was correct and nine containing rhythmic errors. To classify these song variations, Random Forest (RF), K-Nearest Neighbor (KNN), and Artificial Neural Network (ANN) algorithms were trained with data from the 80 taxels combined from the tactile sensors in the fingertips. Feeling the differences between correct and incorrect versions of the song was done with the exoskeleton independently and while the exoskeleton was worn by a person. Results demonstrated that the ANN algorithm had the highest classification accuracy of 97.13% ± 2.00% with the human subject and 94.60% ± 1.26% without. These findings highlight the potential of the smart exoskeleton to aid disabled individuals in relearning dexterous tasks like playing musical instruments.

Keywords: 3D print; artificial intelligence; exoskeleton; hand; sensor array; soft robot.

FIGURE 1

(A) Soft actuator with sensor arrays; (B) CAD model for the new sensorized soft hand exoskeleton (i) top view, (ii) bottom view; (C) The new soft hand exoskeleton (i) top view, (ii) bottom view.

FIGURE 2

Manufacturing Process: (A) (i) All printed components in CAD assembly, cast made from mold 1, PVA stents, and tubing, (ii) Complete stage 1 cast, shown after filling with Dragon Skin and sealing shut; (B) (i) Result of stage 1 cast, (ii) Cast is equipped with strain-limiting layers and pressure sensor arrays; (C) (i) Fully equipped cast is lain in the stage 2 mold to encase the strain limiting layers and sensors as part of the exoskeleton, (ii) Complete stage 2 cast, shown after filling with Dragon Skin and sealing shut; (D) (i) Result of the stage 2 cast, (ii) PVA stents are dissolved, and stage 3 casting is done to seal the pressure chambers.

FIGURE 3

Control system. (A) The control scheme for the exoskeleton and sensors; (B) The valve control signals of each finger for playing “Mary Had A Little Lamb.” Illustrative examples are shown of the correct song and the song variations that had errors introduced.

FIGURE 4

(A) The hand exoskeleton was equipped with an internal pressure sensor and applied forces to the load cell; (B) Color map shows the spatial location of the 16 taxels on the sensor of each finger; (C) Force measured by the load cell for the little finger at an internal pressure of 0.14 MPa; (D) Corresponding taxel signals at the pressure of 0.14 MPa; (E) Force-pressure relationship using 16 actuation cycles of the little finger for three different internal pressures; (F) the maximal generated fingertip forces correlated almost linearly to increasing pressure over the tested range.

FIGURE 5

Exoskeleton playing the piano independently and while being worn. (A) (i) Actuation of each finger while playing a song. (ii) Color map of the tactile sensor showing the locations of taxels on each finger; (B) (i) The exoskeleton as used independently, (ii) Two illustrative taxels for each finger are shown while playing the song. (iii) The response of all the taxels for the little finger during a single keystroke; (C) (i) The exoskeleton was inserted into a glove and worn by the human subject. (ii) Two illustrative taxels for each finger. (iii) The response of all the taxels for the little finger from a single keystroke.

FIGURE 6

(A) Illustrative confusion matrices for the ANN showed the accuracy for classifying the 10 different song alterations during independent use and (B) while user-worn; (C) Comparison of 3 classification algorithms during independent use and with a human subject wearing the soft robotic exoskeleton. The ANN had significantly higher accuracy than the KNN and RF algorithms.