Impact of mask use on face recognition: an eye-tracking study

Abstract

We examined how mask use affects performance and eye movements in face recognition and whether strategy change reflected in eye movements is associated with performance change. Eighty-eight participants performed face recognition with masked faces either during learning only, during recognition only, or during both learning and recognition. As compared with the baseline condition where faces were unmasked during both learning and recognition, participants had impaired performance in all three scenarios, with larger impairment when mask conditions during learning and recognition did not match. When recognizing unmasked faces, whether the faces were learned with or without a mask on did not change eye movement behavior. Nevertheless, when recognizing unmasked faces that were learned with a mask on, participants who adopted more eyes-focused patterns had less performance impairment as compared with the baseline condition. When recognizing masked faces, participants had more eyes-focused patterns and more consistent gaze transition behavior than recognizing unmasked faces regardless of whether the faces were learned with or without a mask on. Nevertheless, when recognizing masked faces that were learned without a mask, participants whose gaze transition behavior was more consistent had less performance impairment as compared with the baseline condition. Thus, although eye movements during recognition were mainly driven by the mask condition during recognition but not that during learning, those who adjusted their strategy according to the mask condition difference between learning and recognition had better performance. This finding has important implications for identifying populations vulnerable to the impact of mask use and potential remedial strategies.

Keywords: EMHMM; Eye movement; Face mask use; Face recognition.

Fig. 1

a Participants’ d′ in different mask conditions (unmasked–unmasked: Faces were unmasked during both learning and recognition. Unmasked–masked: Faces were unmasked during learning and masked during recognition. Masked–unmasked: Faces were masked during learning and unmasked during recognition. Masked–masked: Faces were masked during both learning and recognition). b Participants’ RT in ms in different mask conditions (*p < .05, ** p < .01, ***p < .001, paired t test)

Fig. 2

Eye movement data during face learning. a The two representative eye movement patterns discovered using EMHMM during face learning: In each pattern, ellipses show ROIs as 2-D Gaussian emissions. Priors in the table show the probabilities that a fixation sequence starts from the ellipse. The table also shows transition probabilities among the ROIs. The smaller image on the top-right shows the assignment of actual fixations to different ROIs. The assignment of fixations to the ROIs was based on the ROI sequence with the largest posterior probability given the fixation sequence. The smaller image on the bottom-right shows the corresponding heatmap. b Eye movement pattern measured in A–B scale in the unmasked and masked conditions during face learning

Fig. 3

Eye movement data during the recognition phase. a The two representative eye movement patterns discovered using EMHMM during face recognition: In each pattern, the ellipses show ROIs as 2-D Gaussian emissions. Priors in the table show the probabilities that a fixation sequence starts from the ellipse. The table also shows transition probabilities among the ROIs. The smaller image on the top-right shows the assignment of actual fixations to different ROIs. The assignment of fixations to the ROIs was based on the ROI sequence with the largest posterior probability given the fixation sequence. The smaller image on the bottom-right shows the corresponding heatmap. Note that ROI 5 (Cyan) in both patterns captures outlier fixations that do not belong to other ROIs. b Eye movement pattern measured in A–B scale in different mask conditions. c Eye gaze transition consistency from the first fixation to the second fixation as measured in conditional entropy. d Eye gaze transition consistency from the second fixation to the third fixation as measured in conditional entropy (*p < .05, ** p < .01, ***p < .001, paired t test)

Fig. 4

Correlation between the change in A–B scale due to mask use during learning (unmasked–unmasked condition minus masked–unmasked condition; more positive change indicates larger change toward the more eyes-focused Pattern B with mask use) and the corresponding performance impairment in d′: The more change toward the more eyes-focused Pattern B, the less the performance impairment

Fig. 5

Correlation between the change in conditional entropy of the third fixation given the second fixation due to mask use during recognition (unmasked–unmasked condition minus unmasked–masked condition; more positive change indicates larger change toward lower entropy and thus more consistent transition with mask use) and the corresponding performance impairment in d′: The more change toward low-entropy/more consistent gaze transition behavior, the less the performance impairment