Learning-Based Motion-Intention Prediction for End-Point Control of Upper-Limb-Assistive Robots

Abstract

The lack of intuitive and active human-robot interaction makes it difficult to use upper-limb-assistive devices. In this paper, we propose a novel learning-based controller that intuitively uses onset motion to predict the desired end-point position for an assistive robot. A multi-modal sensing system comprising inertial measurement units (IMUs), electromyographic (EMG) sensors, and mechanomyography (MMG) sensors was implemented. This system was used to acquire kinematic and physiological signals during reaching and placing tasks performed by five healthy subjects. The onset motion data of each motion trial were extracted to input into traditional regression models and deep learning models for training and testing. The models can predict the position of the hand in planar space, which is the reference position for low-level position controllers. The results show that using IMU sensor with the proposed prediction model is sufficient for motion intention detection, which can provide almost the same prediction performance compared with adding EMG or MMG. Additionally, recurrent neural network (RNN)-based models can predict target positions over a short onset time window for reaching motions and are suitable for predicting targets over a longer horizon for placing tasks. This study's detailed analysis can improve the usability of the assistive/rehabilitation robots.

Keywords: human–robot interaction; machine learning; motion intention detection; sensory fusion; upper limb assistive robots; wearable sensors.

Figure 1

Overview of the proposed learning-based controller. The onset motion data are acquired from different sensory interfaces. The input features are then computed for four onset time windows of motion. Then, the processed motion samples are given as input to multiple learning-based intention prediction models for end-point prediction. Finally, the target end-point positions will be given as a reference point to the low-level controller for the robot to move.

Figure 2

Experimental setup: The subject sat in front of the experimental table. Then, he moved the cup according to the number of the positions indicated on the screen of a laptop.

Figure 3

Schematic of the experimental table: Each red position is made up of a 4 cm radius circular. Twelve magnetic stickers are placed in different locations.

Figure 4

Data acquisition flow: The IMUs and the EMG and MMG sensors were placed on the upper limb of the participant. The EMG and IMU data were transmitted wirelessly, and the MMG data were captured via a wired connection. A DAQ acquires the data (Quanser PIDe) connected to a PC.

Figure 5

The input features are calculated within the whole time window of onset movement. The motion data of each subject can be reformed as a single vector.

Figure 6

The settled margin of radius error for object’s location. The largest range was chosen to be coincident with the palm-to-wrist length of the $99\%$ of adults, corresponding to 8 cm [42].

Figure 7

Prediction accuracy of the “reaching” task for different combinations of input features considering all subjects. The performance of the analysis mainly targets the LSTM model. The blue bars represent the results that only use kinematic input features labeled as IMU. The orange bars refer to the physiological features input only, labeled as EMG + MMG. Results with the combinations of input features are shown in green (labeled as all).

Figure 8

Prediction accuracy of the “placing task” for different combinations of input features for all participants subjects. The LSTM model is the main model to be analyzed in this section. The blue bars represent the results that only use kinematic input features labeled as IMU. The orange bars refer to the physiological features input only, labeled as EMG + MMG. Results with the combinations of input features are shown in green (labeled as all).