Automated Video Analysis of Non-verbal Communication in a Medical Setting

Abstract

Non-verbal communication plays a significant role in establishing good rapport between physicians and patients and may influence aspects of patient health outcomes. It is therefore important to analyze non-verbal communication in medical settings. Current approaches to measure non-verbal interactions in medicine employ coding by human raters. Such tools are labor intensive and hence limit the scale of possible studies. Here, we present an automated video analysis tool for non-verbal interactions in a medical setting. We test the tool using videos of subjects that interact with an actor portraying a doctor. The actor interviews the subjects performing one of two scripted scenarios of interviewing the subjects: in one scenario the actor showed minimal engagement with the subject. The second scenario included active listening by the doctor and attentiveness to the subject. We analyze the cross correlation in total kinetic energy of the two people in the dyad, and also characterize the frequency spectrum of their motion. We find large differences in interpersonal motion synchrony and entrainment between the two performance scenarios. The active listening scenario shows more synchrony and more symmetric followership than the other scenario. Moreover, the active listening scenario shows more high-frequency motion termed jitter that has been recently suggested to be a marker of followership. The present approach may be useful for analyzing physician-patient interactions in terms of synchrony and dominance in a range of medical settings.

Keywords: doctor-patient interactions; entrainment; non-verbal communication; performance; synchronization; video analysis.

Figure 1

Examples of performance A and B in the dyadic actor-subject interaction. In performance A, the actor mainly types, and asks a few closed questions. In performance B, the actor actively listens to the subject using open questions and reflections, and explains the mechanism and effect of the drug.

Figure 2

Cross-correlation of subject and actor motion kinetic energy shows higher synchrony and symmetric followership in performance B (blue) compared with performance A (red). Height at zero delay means synchrony of motion, height at positive delay means subject's entrainment by the actor, and height at negative delay means actor's entrainment by the subject. x-axis: time delay [sec], y-axis: Normalized cross-correlation. Inset, two examples of specific dyadic cross-correlation of performance A (red) and performance B (blue).

Figure 3

Synchrony and entrainment of dyadic interaction differentiates between performance A and B in the video analysis. Performance B (blue circles) has higher synchrony values and more equal entrainment between the actor and subject compared with performance A (red circles). A logistic regression classifier separates the two performances with a 72% accuracy (black dashed line). The 70% probability function lines for performance B (green dashed line) and for performance A (purple dashed line) are shown. The classifier probability function can be described as: P(1) = 1/(1 + ae^bx+cy). The parameters of the logistic regression classifier are: a = 5 ± 1, b = −23 ± 5, c = 0.6 ± 0.2, mean ± std, as determined by bootstrapping with 1000 repeats.

Figure 4

Jitter of subjects and actor differs between performance A and performance B. The jitter motion (Fourier Power at frequencies 1.5–5 Hz) of both subjects (A) and actor (B) is higher at performance B scenario, suggesting more followership motion (Noy et al., 2011) of both interlocutors. ^*p < 0.05, ^***p < 0.001. Subjects: Mann–Whitney test, p < 0.03, rank biserial correlation, r = 0.39. Actor: Mann–Whitney test, p < 0.001, rank biserial correlation, r = 0.8.