Depth from motion parallax: Deictic consistency, eye contact, and a serious problem with Zoom

Abstract

The dynamics of head and eye gaze between two or more individuals displayed during verbal and nonverbal face-to-face communication contains a wealth of information and is used for both volitionary and unconscious signaling. Current video communication systems convey visual signals about gaze behavior and other directional cues, but the information they carry is often spurious and potentially misleading. I discuss the consequences of this situation, identify the source of the problem as a more general lack of deictic consistency, and demonstrate that using display technologies that simulate motion parallax are both necessary and sufficient to alleviate it. I then devise an avatar-based remote communication solution that achieves deictic consistency and provides natural, dynamic eye contact for computer-mediated audiovisual communication.

Figure 1.

Deictic consistency, its violation in video conferencing, and an outline for its rectification. (a) Deictic consistency means that both interlocutors agree on their orientations α and β with respect to the common line of sight. There is no reason to disagree when meeting face-to-face in the real world. (b) However, there are conflicts when using video communication systems such as Zoom or Skype. Deictic consistency is violated. In the depicted example, user Y perceives himself to be oriented an angle β clockwise from the line of sight whereas user X perceives him oriented an angle γ from the line of sight. For another example, user X perceives herself oriented an angle α counterclockwise from the line of sight, but user Y would perceive her oriented parallel to the line of sight, looking directly at him. S, screen; C, camera; blue lines connect cameras with their respective screens and are no longer drawn in the following panels. (c) This sketch shows the same situation as in panel b. However, superimposing the two screens on top of each other better shows the nature of the problem and what needs to be done to solve it. (d) To simulate a situation where the two interlocutors see each other through a window that connects their individual, ego-centric spaces, the camera C_Y needs to be placed at user X’s location and needs to be moved along if that location changes. The camera feed from C_Y then needs to be projected back from X’s location onto the screen S_X. Note, that in the general case, the point of projection may not be on the central normal of the screen. The above also applies to user Y, camera C_X and screen S_Y.

Figure 2.

In virtual reality, observers were standing upright in front of two large openings representing two adjacent walls of a hexagonal gazebo located in a forest. The left opening was a normal open window. The right opening behaved like a window with respect to motion parallax, but like a screen with respect to binocular disparity (parallax-only condition, left and middle box), or it behaved like a screen with respect to motion parallax, but like a window with respect to binocular disparity (stereo-only, right box). Binocular, but not monocular observers perceived veridical parallax to be too large and scaled it down. No such scaling was observed for binocular disparity. Box and Whisker plots, adapted from Wang et al. (2020).