Recognition of handwriting from electromyography

Abstract

Handwriting--one of the most important developments in human culture--is also a methodological tool in several scientific disciplines, most importantly handwriting recognition methods, graphology and medical diagnostics. Previous studies have relied largely on the analyses of handwritten traces or kinematic analysis of handwriting; whereas electromyographic (EMG) signals associated with handwriting have received little attention. Here we show for the first time, a method in which EMG signals generated by hand and forearm muscles during handwriting activity are reliably translated into both algorithm-generated handwriting traces and font characters using decoding algorithms. Our results demonstrate the feasibility of recreating handwriting solely from EMG signals - the finding that can be utilized in computer peripherals and myoelectric prosthetic devices. Moreover, this approach may provide a rapid and sensitive method for diagnosing a variety of neurogenerative diseases before other symptoms become clear.

Figure 1. Data acquisition.

A: A photograph of a recording session. B: Electrode placement over the hand (top) and forearm muscles (bottom).

Figure 2. Reconstruction of handwriting traces using the Wiener filter.

A: Schematics of the Wiener filter. EMG signals (rectified EMGs) from multiple models were fed into two independent Wiener filters which reconstructed X and Y coordinates of the pen, respectively. Each filter represented reconstructed coordinate as a weighted sum of EMGs. B: Examples of reconstructed traces from one recording session. Actual traces are shown in blue; reconstructed traces are shown in red. The first two columns show X(t) and Y(t), respectively. The third column shows X-Y plots.

Figure 3. Transformation of EMG records into font characters.

A: Schematics of the algorithm. Compound EMG (the sum of all rectified EMGs) was first used to detect the periods during which handwriting occurred. Compound EMG was first segmented into epochs corresponding to individual characters using a threshold that detected EMG bursts. Then, a generic compound EMG template was calculated by averaging these epochs. Template matching was used to refine the EMG segments, which were then classified using linear discriminant analysis. B: Example of discrimination for a representative recording session. From top to bottom: Eight EMGs were used for character recognition. 3.5-s segments corresponding to individual characters are highlighted as blue bars which are aligned on peak correlation coefficient, R, for template matching. Posterior probabilities for character recognition which were computed by discriminant analysis are shown as color plots. Recognized font character which corresponds to the highest probability is shown near each plot. Original handwriting is shown at the bottom.

Figure 4. Performance of reconstruction and recognition algorithms as the function of the number of recorded EMGs and the amount of training data.

The analyses were conducted were all muscles and hand and forearm muscles only (see key on top). A: Reconstruction accuracy of the X-coordinate of the pen as the function of number of muscles recorded. Muscles were taken in different combinations, and R² was averaged across these combinations and across subjects. B: Recognition accuracy of the Y-coordinate. C: Recognition accuracy as the function of the number of recorded muscles. D: Improvement in recognition accuracy as the function of training set size.