Neuromimetic Event-Based Detection for Closed-Loop Tactile Feedback Control of Upper Limb Prostheses

Abstract

Upper limb amputees lack the valuable tactile sensing that helps provide context about the surrounding environment. Here we utilize tactile information to provide active touch feedback to a prosthetic hand. First, we developed fingertip tactile sensors for producing biomimetic spiking responses for monitoring contact, release, and slip of an object grasped by a prosthetic hand. We convert the sensor output into pulses, mimicking the rapid and slowly adapting spiking responses of receptor afferents found in the human body. Second, we designed and implemented two neuromimetic event-based algorithms, Compliant Grasping and Slip Prevention, on a prosthesis to create a local closed-loop tactile feedback control system (i.e. tactile information is sent to the prosthesis). Grasping experiments were designed to assess the benefit of this biologically inspired neuromimetic tactile feedback to a prosthesis. Results from able-bodied and amputee subjects show the average number of objects that broke or slipped during grasping decreased by over 50% and the average time to complete a grasping task decreased by at least 10% for most trials when comparing neuromimetic tactile feedback with no feedback on a prosthesis. Our neuromimetic method of closed-loop tactile sensing is a novel approach to improving the function of upper limb prostheses.

Keywords: Prosthetics; force feedback; neuromimetic; real-time control.

Fig. 1

Adaptation of results from [25], this schematic shows the amount of skin indentation (top) and typical RA (middle) and SA1 (bottom) responses. RA receptors respond during the transient periods of indentation to help indicate contact and release while SA1 receptors exhibit a response during sustained indentation.

Fig. 2

System diagram showing the closed-loop nature of the tactile feedback system. The prosthesis control unit receives both amputee EMG signals and tactile information before sending out a command to the terminal device.

Fig. 3

(a) Textile sensor cuff design, which includes flexible and stretchable materials that allow the sensor to be placed on a prosthesis phalanx. Conductive traces act as the sensing elements and are protected by an outer fabric layer along with a rubber coating. (b) Sensor cuffs are placed on the tips of the thumb, index, and middle fingers of the prosthesis.

Fig. 4

A grasp-hold-release event with tactile feedback. The top plot shows the onset, hold, and release of an object grasped by a prosthetic hand. The RA-like tactile response (middle) produces a small cluster of positive spikes during the onset of object contact and negative spikes during object release. The SA1-like response (bottom) simultaneously measures sustained grip force.

Fig. 5

(a) The neuromimetic touch feedback algorithm uses the RA-like sensor response, R(t), which is found by passing the force signal (S(t)) through a high pass filter and comparing it to the threshold β, to determine the onset of object contact, release, and slip. (b) The Compliant Grasping strategy uses object contact to dynamically modulating the user’s EMG gain, α, to help prevent grasping objects with excessive force, and (c) uses the same neuromimetic RA-like response to monitor and correct for object slip.

Fig. 6

The true EMG gain measured from the prosthesis controller during a prosthesis grasping task with increasing grip force and Compliant Grasping. To prevent the EMG signal from shrinking to zero, a lower threshold of 20% is placed on the gain.

Fig. 7

The Slip Prevention control strategy uses the biomimetic RA-like sensor response, R(t), spikes to monitor for object slip. Instances of slip are identified using this neuromimetic approach by measuring the rate of change of the grip force. An instance of slip triggers the prosthesis to close to prevent an object from slipping from its grasp.

Fig. 8

(a) A custom brace is used for operation of a prosthetic hand by able-bodied subjects. A pair of Otto Bock electrodes (MYOBOCK, Otto Bock healthcare, Minneapolis, USA) are placed on the forearm of the subject to collect the EMG signals. (b) The amputee participants used their personal prosthetic socket with embedded Otto Bock EMG electrodes.

Fig. 9

A tripod grip is used by the prosthesis for all grasping tasks. For this grip, the thumb as well as the index and middle fingers are used to grasp an object.

Fig. 10

The items used for the Compliant Grasping task. From left to right: packing foam, cracker, hollow egg, and a polystyrene cup. These common objects, most of which have been used in previous grasping studies, were chosen due to their delicate nature [14], [40], [41].

Fig. 11

The average number of broken objects during the Compliant Grasping tests for the (a) able-bodied and (b) amputee subjects.

Fig. 12

The normalized time to complete a Compliant Grasping tests for the (a) able-bodied and (b) amputee subjects. Trial completion times are normalized using the average time to complete a task for a particular item using the unmodified prosthesis. Both plots show a decrease in the time required to complete item movements while using tactile feedback as an input for the control algorithm, with the exception of the eggs for the amputee subjects.

Fig. 13

The average distance the grasped cylinder slipped during the Slip Prevention tests for the (a) able-bodied and (b) amputee subjects

Fig. 14

The number of times, as a percentage of the total number of trials, the grasped cylinder fell from the prosthesis during the Slip Prevention tests for the able-bodied subjects. There were no failed trials during experiments with the amputee subject.