Remote Digital Measurement of Facial and Vocal Markers of Major Depressive Disorder Severity and Treatment Response: A Pilot Study

Abstract

Objectives: Multiple machine learning-based visual and auditory digital markers have demonstrated associations between major depressive disorder (MDD) status and severity. The current study examines if such measurements can quantify response to antidepressant treatment (ADT) with selective serotonin reuptake inhibitors (SSRIs) and serotonin-norepinephrine uptake inhibitors (SNRIs). Methods: Visual and auditory markers were acquired through an automated smartphone task that measures facial, vocal, and head movement characteristics across 4 weeks of treatment (with time points at baseline, 2 weeks, and 4 weeks) on ADT (n = 18). MDD diagnosis was confirmed using the Mini-International Neuropsychiatric Interview (MINI), and the Montgomery-Åsberg Depression Rating Scale (MADRS) was collected concordantly to assess changes in MDD severity. Results: Patient responses to ADT demonstrated clinically and statistically significant changes in the MADRS [F _{(2, 34)} = 51.62, p < 0.0001]. Additionally, patients demonstrated significant increases in multiple digital markers including facial expressivity, head movement, and amount of speech. Finally, patients demonstrated significantly decreased frequency of fear and anger facial expressions. Conclusion: Digital markers associated with MDD demonstrate validity as measures of treatment response.

Keywords: Montgomery-Åsberg Depression Rating Scale; antidepressant treatment; computer vision; digital biomarker; digital phenotyping; machine learning; major depressive disorder.

Figure 1

Depiction of the smartphone-based assessment that all individuals completed. Video and audio of participant responses were recorded during the assessment and used to quantify behavioral characteristics and subsequently measure digital markers of major depressive disorder (MDD) severity.

Figure 2

Response to treatment as measured by two independent assessments. Mean for each time point and standard error bars are shown. Results demonstrate treatment's significant effect on digital markers, which are highly concordant with change in depression symptom severity. (A) Montgomery–Åsberg Depression Rating Scale (MADRS) scores acquired at baseline (BL), week 2, and week 4 showed significant decrease in response to antidepressant therapy (ADT) [F_{(2, 34)} = 51.62, p < 0.0001]. (B) Overall facial expressivity measured in response to positive [F_{(2, 28)} = 40.66; p < 0.0001], neutral [F_{(2, 28)} = 32.6; p < 0.0001], and negative [F_{(2, 28)} = 36.95; p < 0.0001] images demonstrated a significant increase in response to ADT as MADRS scores decreased. (C) Percentage of frames with voice measured in response to positive [F_{(2, 26)} = 3.59; p = 0.04], neutral [F_{(2, 26)} = 5.59; p 0.009], and negative [F_{(2, 28)} = 4.65; p = 0.02] images also demonstrated a significant increase in response to ADT as MADRS scores decreased. All values in (B,C) were normalized between 0 and 1.