DeepSmile: Anomaly Detection Software for Facial Movement Assessment

Abstract

Facial movements are crucial for human interaction because they provide relevant information on verbal and non-verbal communication and social interactions. From a clinical point of view, the analysis of facial movements is important for diagnosis, follow-up, drug therapy, and surgical treatment. Current methods of assessing facial palsy are either (i) objective but inaccurate, (ii) subjective and, thus, depending on the clinician's level of experience, or (iii) based on static data. To address the aforementioned problems, we implemented a deep learning algorithm to assess facial movements during smiling. Such a model was trained on a dataset that contains healthy smiles only following an anomaly detection strategy. Generally speaking, the degree of anomaly is computed by comparing the model's suggested healthy smile with the person's actual smile. The experimentation showed that the model successfully computed a high degree of anomaly when assessing the patients' smiles. Furthermore, a graphical user interface was developed to test its practical usage in a clinical routine. In conclusion, we present a deep learning model, implemented on open-source software, designed to help clinicians to assess facial movements.

Keywords: anomaly detection; deep learning; facial paralysis; long-short term memory.

Figure 1

The pipeline’s workflow.

Figure 2

Set of markers virtually placed on a 3D model of the face. The selected markers appear highlighted in violet and red (commissure markers).

Figure 3

Average of the scaled displacement for the commissure markers.

Figure 4

Representation of an LSTM cell.

Figure 5

Performance of LSTM during training.

Figure 6

LSTM assessment of a relevant pair of distal markers on the test dataset.

Figure 7

Predictions of the left oral commissure marker on a healthy smile.

Figure 8

DeepSmile’s graphical user interface.