Pupil Responses During Interactive Conversation

Abstract

Pupillometry has been used to assess effort in a variety of listening experiments. However, measuring listening effort during conversational interaction remains difficult as it requires a complex overlap of attention and effort directed to both listening and speech planning. This work introduces a method for measuring how the pupil responds consistently to turn-taking over the course of an entire conversation. Pupillary temporal response functions to the so-called conversational state changes are derived and analyzed for consistent differences that exist across people and acoustic environmental conditions. Additional considerations are made to account for changes in the pupil response that could be attributed to eye-gaze behavior. Our findings, based on data collected from 12 normal-hearing pairs of talkers, reveal that the pupil does respond in a time-synchronous manner to turn-taking. Preliminary interpretation suggests that these variations correspond to our expectations around effort direction in conversation.

Keywords: listening effort; pupillometry; temporal response function; turn-taking.

Figure 1.

The four conversational state change events denoted in a sample turn-taking exchange between two talkers.

Figure 2.

Cumulative distribution functions of the state changes relative to other state changes, over time. Each panel corresponds to a different type of state change. The colored curves indicate the probability that each of the other state changes has occurred, as a function of time, with respect to the state change each panel belongs to. In computing these curves, only positive FTOs are included, such that the probability at time 0 will be 0 for all resulting CDFs.

Figure 3.

State change response curves obtained by averaging the results for all participants in all conversations. The highlighted regions indicate the pointwise confidence intervals. The red bars indicate consecutive pointwise significant intervals over which the summed One-sample t statistic exceeds a significance threshold set based on a P value of .05, denoting that the curves are significantly different from 0.

Figure 4.

State change responses obtained only by averaging conversations that took place in quiet (solid line) and 70 dBA noise (dotted line). The shaded regions indicate the pointwise one-sample confidence intervals of each curve. The red bars indicate intervals over which the summed two-sample t statistic exceeds the significance threshold set based on a P value of .05, denoting the two curves are significantly different from each other.

Figure 5.

Derived state change responses (solid lines) plotted against the grand averages of fixed windows (dotted lines). The solid lines corresponding to the derived pupillary state change responses are identical to those presented in Figure 3.

Figure 6.

Distributions of gaze depth and the duration of fixations to the partner across all conversations, by condition. Color indicates the condition the conversation took place in. Also included are the overall percentages of each conversation that were spent looking at the partner, by condition, determined as the percentage of time points where the gaze depth was greater than the 95 cm threshold defined previously.

Figure 7.

Pupil responses to change of gaze target. The shaded regions indicate the pointwise confidence intervals. The red bars indicate intervals over which the set of TRFs averaged to produce these curves has a mean significantly different from 0.

Figure 8.

Pupil responses to change of gaze by condition, where the solid line corresponds to the quiet condition and the dotted line corresponds to the 70 dBA background noise condition. The shaded regions indicate the pointwise confidence intervals of each curve. No significantly different intervals were identified between conditions for these curves.