Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2022 Jul 1:13:878282.
doi: 10.3389/fneur.2022.878282. eCollection 2022.

Human vs. Machine Learning Based Detection of Facial Weakness Using Video Analysis

Chad M Aldridge  1 , Mark M McDonald  1 , Mattia Wruble  1 , Yan Zhuang  2 , Omar Uribe  1 , Timothy L McMurry  3 , Iris Lin  4 , Haydon Pitchford  1 , Brett J Schneider  1 , William A Dalrymple  1 , Joseph F Carrera  5 , Sherita Chapman  1 , Bradford B Worrall  1   3 , Gustavo K Rohde  2   6 , Andrew M Southerland  1   3
Affiliations

Human vs. Machine Learning Based Detection of Facial Weakness Using Video Analysis

Chad M Aldridge et al. Front Neurol. .

Abstract

Background: Current EMS stroke screening tools facilitate early detection and triage, but the tools' accuracy and reliability are limited and highly variable. An automated stroke screening tool could improve stroke outcomes by facilitating more accurate prehospital diagnosis and delivery. We hypothesize that a machine learning algorithm using video analysis can detect common signs of stroke. As a proof-of-concept study, we trained a computer algorithm to detect presence and laterality of facial weakness in publically available videos with comparable accuracy, sensitivity, and specificity to paramedics.

Methods and results: We curated videos of people with unilateral facial weakness (n = 93) and with a normal smile (n = 96) from publicly available web-based sources. Three board certified vascular neurologists categorized the videos according to the presence or absence of weakness and laterality. Three paramedics independently analyzed each video with a mean accuracy, sensitivity and specificity of 92.6% [95% CI 90.1-94.7%], 87.8% [95% CI 83.9-91.7%] and 99.3% [95% CI 98.2-100%]. Using a 5-fold cross validation scheme, we trained a computer vision algorithm to analyze the same videos producing an accuracy, sensitivity and specificity of 88.9% [95% CI 83.5-93%], 90.3% [95% CI 82.4-95.5%] and 87.5 [95% CI 79.2-93.4%].

Conclusions: These preliminary results suggest that a machine learning algorithm using computer vision analysis can detect unilateral facial weakness in pre-recorded videos with an accuracy and sensitivity comparable to trained paramedics. Further research is warranted to pursue the concept of augmented facial weakness detection and external validation of this algorithm in independent data sets and prospective patient encounters.

Keywords: access to care; cerebrovascular disease; computer vision; diagnostic test; infarction; machine learning; stroke.

PubMed Disclaimer

Conflict of interest statement

MM and OU disclose ownership interest in Neuroview Diagnostics, LLC. Neuroview Diagnostics played no role in study funding, data collection, or data analysis. AS and GR served as unpaid scientific advisors for Neuroview Diagnostics, LLC. AS receives research support for a prehospital stroke trial from Diffusion Pharmaceuticals, Inc. MM, YZ, OU, AS, and GR disclose U.S. Provisional Patent Application No. 62/620,096 and International Patent Application No. PCT/US19/14605. BBW serves as Deputy Editor of the journal, Neurology. AS is past section Editor of the Neurology podcast. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Figures

Figure 1
Figure 1
Facial landmark extraction and classification. For algorithm training and performance, videos were decomposed into individual frames and facial landmarks were extracted. Normalization was performed by aligning the extracted landmarks of current input sequences to a template to handle different head scales, location, and orientation. The same transformation was applied to the pixel intensity information to remove these variations. A predictive model computed the classification result for each individual frame and a voting classifier reduced the output into discrete classifications comprised of no weakness, left facial weakness, and right facial weakness categories.
Figure 2
Figure 2
ROC curves of facial weakness and laterality detection of the algorithm and each paramedic. The overall performance of paramedics vs. the algorithm in the detection of unilateral facial weakness failed to reach significance (p-value 0.074).
See this image and copyright information in PMC

References

    1. Ramanujam P, Guluma KZ, Castillo EM, Chacon M, Jensen MB, Patel E, et al. . Accuracy of stroke recognition by emergency medical dispatchers and paramedics—San Diego experience. 10.1080/10903120802099526. (2009) 12:307–13. Available online at: https://www.tandfonline.com/doi/abs/10.1080/10903120802099526 (accessed March 23, 2022). - DOI - DOI - PubMed
    1. Glober NK, Sporer KA, Guluma KZ, Serra JP, Barger JA, Brown JF, et al. . Acute stroke: current evidence-based recommendations for prehospital care. Western J Emerg Med. (2016) 17:104. Available online at: /pmc/articles/PMC4786229/ (accessed March 23, 2022). - PMC - PubMed
    1. Adams HP, del Zoppo G, Alberts MJ, Bhatt DL, Brass L, Furlan A, et al. . Guidelines for the early management of adults with ischemic stroke. Stroke. (2007) 38:1655–711. Available online at: https://www.ahajournals.org/doi/abs/10.1161/STROKEAHA.107.181486 (accessed March 23, 2022). - DOI - PubMed
    1. Mohd Nor A, McAllister C, Louw SJ, Dyker AG, Davis M, Jenkinson D, et al. . Agreement between ambulance paramedic- and physician-recorded neurological signs with face arm speech test (FAST) in acute stroke patients. Stroke. (2004) 35:1355–9. Available online at: https://www.ahajournals.org/doi/abs/10.1161/01.STR.0000128529.63156.c5 (accessed March 23, 2022). - DOI - PubMed
    1. Lyden P, Raman R, Liu L, Emr M, Warren M, Marler J. National institutes of health stroke scale certification is reliable across multiple venues. Stroke. (2009) 40:2507–11. Available online at: https://www.ahajournals.org/doi/abs/10.1161/STROKEAHA.108.532069 (accessed March 23, 2022). - DOI - PMC - PubMed