Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2021 Aug;9(16):1307.
doi: 10.21037/atm-21-3457.

Detection of hypomimia in patients with Parkinson's disease via smile videos

Ge Su  1 , Bo Lin  1 , Jianwei Yin  1 , Wei Luo  2 , Renjun Xu  3 , Jie Xu  2 , Kexiong Dong  4
Affiliations

Detection of hypomimia in patients with Parkinson's disease via smile videos

Ge Su et al. Ann Transl Med. 2021 Aug.

Abstract

Background: Parkinson's disease (PD) is a neurodegenerative disease characterized by the impairment of facial expression, known as hypomimia. Hypomimia has serious impacts on patients' ability to communicate, and it is difficult to detect at early stages of the disease. Furthermore, due to bradykinesia or other reasons, it is inconvenient for PD patients to visit the hospital. Therefore, it is appealing to develop an auxiliary diagnostic method that remotely detects hypomimia.

Methods: We proposed an automatic detection system for Parkinson's hypomimia based on facial expressions (DSPH-FE). DSPH-FE provides a convenient remote service for those who potentially suffer from hypomimia and only requires patients to input their facial videos. Specifically, patients can detect hypomimia through two aspects: geometric features and texture features. Geometric features focus on visually representing structures of facial muscles. Facial expression factors (FEFs) are used as the first metric to quantify the current activation state of the facial muscles. Facial expression change factors (FECFs) are subsequently used as the second metric to calculate the moving trajectories of the activation states in the videos. Geometric features primarily concentrate on spatial information, with little involvement of temporal information. Thus, the extended histogram of oriented gradients (HOG) algorithm is introduced. This algorithm can extract texture features within multiple continuous frames and incorporate the temporal information into the features. Finally, these features are applied to four machine learning algorithms to model the relationship between these features and hypomimia.

Results: The DSPH-FE detection system achieved the best performance when concatenating geometric features and texture features, resulting in a F1 score of 0.9997. The best F1 scores achieved with geometric features and texture features were 0.8286 and 0.9446, respectively. This indicated that both geometric features and texture features have an ability to predict hypomimia, and demonstrated that temporal information can boost the model performance. Thus, DSPH-FE is an effective supportive tool in the medical management of PD patients.

Conclusions: Comprehensive experiments demonstrated that proposed features fit well with real-world videos and are beneficial in the clinical diagnosis of hypomimia. In particular, hypomimia had a greater impact on eyes and mouths when patients are smiling.

Keywords: Parkinson’s hypomimia (PD); detection system; facial expressions; geometric features; texture features.

PubMed Disclaimer

Conflict of interest statement

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at https://dx.doi.org/10.21037/atm-21-3457). Dr. GS, Dr. BL, and Dr. JY report that this work was supported by the National Key Research and Development Program of China (No. 2017YFB1400603), the National Natural Science Foundation of China (Grant No. 61825205, No. 61772459), and the National Science and Technology Major Project of China (No.50-D36B02-9002-16/19). Dr. GS, Dr. BL, Dr. JY, and Dr. WL report patents pending of A Construction Method for Detecting Facial Bradykinesia based on Geometric Features and Texture Features, and report provision of study materials from Second Affiliated Hospital, Zhejiang University School of Medicine. Dr. KD is employed by Technical Department in Hangzhou Healink Technology Corporation Limited, which located at 188 Liyi Rd, Hangzhou, China. The other authors have no conflicts of interest to declare.

Figures

Figure 1
Figure 1
Facial keypoints. These 68 facial keypoints conform to the definition of previous face recognition technologies (32-36). This image is published with the patient/participant’s consent.
Figure 2
Figure 2
The extended HOG algorithm. The video information is regarded as a three-dimensional information space, which is represented by a space made up of X, Y, and T axis. The XY plane represents spatial information, and the XT plane or YT plane mainly represents temporal information and linear spatial information.
Figure 3
Figure 3
The overall framework of the DSPH-FE. The geometric features process a single image while the texture features process the image sequence. DSPH-FE, Parkinson’s hypomimia based on facial expressions.
Figure 4
Figure 4
The area of interest for hypomimia. The blue square blocks represent the top 10 significant texture features, the darker blue squares indicate areas with time dimension (XT plane or YT plane). The light blue belongs to XY plane, the blue is XT plane, the darker blue is YT plane. The red arrows represent the top 10 significant facial expression change factors (FECFs) corresponding to the facial expression factors (FEFs). This image is published with the patient/participant’s consent.
Figure 5
Figure 5
Feature adaptation on the model. The first row corresponds to the geometric features (GFs) in Table 4, the second row corresponds to texture features (TFs) in Table 4, and the third row corresponds to fusion features (FFs) in Table 4. Experiments were performed three times for each group of features.
Figure 6
Figure 6
The facial expression change factor (FECF) map of 0838HC and 0508PD based on SEM.
Figure 7
Figure 7
XT plane with the Y-axis coordinate fixed at 40, wherein, the first graph belongs to 0842HC, the second graph belongs to 0451PD.
See this image and copyright information in PMC

References

    1. Poewe W, Seppi K, Tanner CM, et al. Parkinson disease. Nat Rev Dis Primers 2017;3:17013. 10.1038/nrdp.2017.13 - DOI - PubMed
    1. Sveinbjornsdottir S. The clinical symptoms of Parkinson's disease. J Neurochem 2016;139 Suppl 1:318-24. 10.1111/jnc.13691 - DOI - PubMed
    1. Ricciardi L, De Angelis A, Marsili L, et al. Hypomimia in Parkinson's disease: an axial sign responsive to levodopa. Eur J Neurol 2020;27:2422-9. 10.1111/ene.14452 - DOI - PubMed
    1. Ho MW, Chien SH, Lu MK, et al. Impairments in face discrimination and emotion recognition are related to aging and cognitive dysfunctions in Parkinson's disease with dementia. Sci Rep 2020;10:4367. 10.1038/s41598-020-61310-w - DOI - PMC - PubMed
    1. Ricciardi L, Visco-Comandini F, Erro R, et al. Facial Emotion Recognition and Expression in Parkinson's Disease: An Emotional Mirror Mechanism? PLoS One 2017;12:e0169110. 10.1371/journal.pone.0169110 - DOI - PMC - PubMed