Face masks and speaking style affect audio-visual word recognition and memory of native and non-native speech

Abstract

Though necessary, protective mask wearing in response to the COVID-19 pandemic presents communication challenges. The present study examines how signal degradation and loss of visual information due to masks affects intelligibility and memory for native and non-native speech. We also test whether clear speech can alleviate perceptual difficulty for masked speech. One native and one non-native speaker of English recorded video clips in conversational speech without a mask and conversational and clear speech with a mask. Native English listeners watched video clips presented in quiet or mixed with competing speech. The results showed that word recognition and recall of speech produced with a mask can be as accurate as without a mask in optimal listening conditions. Masks affected non-native speech processing at easier noise levels than native speech. Clear speech with a mask significantly improved accuracy in all listening conditions. Speaking clearly, reducing noise, and using surgical masks as well as good signal amplification can help compensate for the loss of intelligibility due to background noise, lack of visual cues, physical distancing, or non-native speech. The findings have implications for communication in classrooms and hospitals where listeners interact with teachers and healthcare providers, oftentimes non-native speakers, through their protective barriers.

FIG. 1.

(Color online) Experiment overview. Each participant heard one of the talkers (native talker or non-native) and one noise condition (quiet, +5 dB SNR, and 0 dB SNR; an additional −5 dB SNR was used for the native talker only) (A). Each listener saw video clips from all three speaking style and mask conditions: conversational speech no mask; conversational speech mask, clear speech mask. Speaking style and mask were counterbalanced across listeners (B). Immediately after each audio-video clip, listeners were instructed to type what they heard to test intelligibility. At the end of each block, they were asked to answer content-related questions to test recall.

FIG. 2.

Intelligibility (top panels) and recall (bottom panels) for the native talker in quiet (n = 32), in 6T +5 dB SNR (n = 40), in 6T 0 dB SNR (n = 38), and in 6T −5 dB SNR (n = 32). ***, p < 0.001; **, p < 0.01; *, p < 0.05; ns, no statistical difference (mixed-effect logistic regression simple effects analysis).

FIG. 3.

Intelligibility (top panels) and recall (bottom panels) for the non-native talker in quiet (n = 45), in 6T +5 dB SNR (n = 32), and in 6T 0 dB SNR (n = 32). ***, p < 0.001; **, p < 0.01; *, p < 0.05; ns, no statistical difference (mixed-effect logistic regression simple effects analysis).