Feasibility Analysis of Phenotype Quantification from Unstructured Clinical Interactions

Abstract

We conducted a feasibility analysis to determine the quality of data that could be collected ambiently during routine clinical conversations. We used inexpensive, consumer-grade hardware to record unstructured dialogue and open-source software tools to quantify and model face, voice (acoustic and language) and movement features. We used an external validation set to perform proof-of-concept predictive analyses and show that clinically relevant measures can be produced without a restrictive protocol.

Keywords: acoustic; conversation; digital phenotype; facial feature; voice.

Figure 1

Quantitative face and voice features versus clinical progress. (A) Face psychomotor activity (gaze and head pose in radians per second scaled from 0 to 1) sized by BPRS depression score. The non-anxious depressed patients (3,5,6,7) tended to have more movement as their depressive mood scores increased and less overall than non-depressed patients. Patient 8 was anxious and depressed, hence her head movement decreased as she recovered. Individual patients are assigned their own color and are numbered by session (Patient 4’s third session is represented by a red dot with the number 3). The background density plot (blue hues) provides context from a larger (142 sessions), independently collected dataset, illustrating how features derived from unstructured conversation fall within the scope of a structured exam. (B) Vowel space density plots visually reveal the trajectory of acoustic changes in a depressed patient who received ketamine infusion. Reduced vowel space is clearly visible during an early session (representing a more monotonic voice, left) compared with a later session (representing a more varied voice, right) for the same patient. The BPRS depressive mood score for the first session was 6 and 0 for the second. Restricted vowel space is a well-documented acoustic feature which has been shown to correlate with depressive mood (Scherer et al., 2016).

Figure 2

Conversational effort and speech content can be measured from unstructured clinical conversation. A) Conversational effort illustrates words per session for clinician and patient. The Patient 8 (grey circles) displays an anxious depression phenotype producing markedly more words than the clinician. Participant feature data plotted longitudinally exposes subtle changes in objective measures that, we speculate, the human brain would find difficult to identify from memory. The cross-sectional plot facilitates patient comparison. B) Speech content analysis quantifies diminution of perseveration. Here, we used semantic analysis to calculate the cosine distance between the single perseverating patient’s speech vectors to the GloVe vector for the concept “consulting.” As the patient’s perseveration decreased, this topic became less frequent. No other patients displayed this behavior.