The Painful Face - Pain Expression Recognition Using Active Appearance Models

Abstract

Pain is typically assessed by patient self-report. Self-reported pain, however, is difficult to interpret and may be impaired or in some circumstances (i.e., young children and the severely ill) not even possible. To circumvent these problems behavioral scientists have identified reliable and valid facial indicators of pain. Hitherto, these methods have required manual measurement by highly skilled human observers. In this paper we explore an approach for automatically recognizing acute pain without the need for human observers. Specifically, our study was restricted to automatically detecting pain in adult patients with rotator cuff injuries. The system employed video input of the patients as they moved their affected and unaffected shoulder. Two types of ground truth were considered. Sequence-level ground truth consisted of Likert-type ratings by skilled observers. Frame-level ground truth was calculated from presence/absence and intensity of facial actions previously associated with pain. Active appearance models (AAM) were used to decouple shape and appearance in the digitized face images. Support vector machines (SVM) were compared for several representations from the AAM and of ground truth of varying granularity. We explored two questions pertinent to the construction, design and development of automatic pain detection systems. First, at what level (i.e., sequence- or frame-level) should datasets be labeled in order to obtain satisfactory automatic pain detection performance? Second, how important is it, at both levels of labeling, that we non-rigidly register the face?

Fig. 1. Examples of temporally subsampled sequences. (a),(c) illustrate Pain and (b),(d) No Pain

Fig. 2. Example of AAM derived representations (a) Top row: input shape(s), Bottom row: input image, (b) Top row: Similarity Normalized Shape(s_n), Bottom Row: Similarity Normalized Appearance(a_n), (c) Top Row: Base Shape(s₀), Bottom Row: Shape Normalized Appearance(a₀)

Fig. 3

Example of video sequence prediction. The x-axis in the above plot represents the frame index in the video, while the y-axis represents the predicted pain score. The dotted arrows show the correspondence between the image frames (top-row) and their predicted pain scores. For intance, Frame 29 in the top row shows an intense pain and corresponds to the peak in the plot.

Fig. 4. Sequence-level pain detection results for experiments performed in section 5.2, showing the ROC for classifiers based on three different representations. The crosses indicate the EER point. The best results (EER: 15.7%) are achieved by using a combination of canonical-appearance and similarity normalized shape (C-APP + S-PTS)

Fig. 5

Frame-level performance based on experiments performed in Section 5.3. (a) Hit rate and false-acceptance rates for SVMs trained using different ground-truth granularity. Training SVMs by using frame-level ground truth improved performance. Frame-level hit rate increased from 77.9% to 82.4%, and frame-level false acceptance rate (FAR) decreased from 44% to 30.1%. (b) Confusion matrix for sequence–trained SVM. (c) Confusion matrix for frame-trained SVM.

Fig. 6

Comparison between the SVM scores for sequence-level ground truth and frame-level ground truth (a) Sample frames from a pain-video sequence with their frame indices, (b) Scores for individual frames for the two SVM training strategies. Points corresponding to the frames shown in (a) are highlighted as crossed. Output of SVM trained on frame-level groundtruth remains lower for frames without pain, and hence leads to a lower false acceptance rate

Fig. 7

Comparison of ROCs for SVMs trained on sequence-and frame-level labels. To demonstrate the efficacy of the sequence-level trained SVM on the frame-level detection task the ROC for a “random-chance” classifier is also included. One can see that although the sequence-level SVM behaves worse than the frame-level SVM it is significantly better than random-chance demonstrating that coarse-level labeling strategies are effective and useful in automatic pain recognition tasks.

Fig. 8

Weighted combination of support vectors to visualize contribution of different face regions for pain recognition. (a) For pain, (b) For no pain. For pain, the brighter regions represent more weightage. For no pain, the darker regions repersent more weightage.