Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2023 Sep 26;9(10):196.
doi: 10.3390/jimaging9100196.

BlinkLinMulT: Transformer-Based Eye Blink Detection

Ádám Fodor  1 , Kristian Fenech  1 , András Lőrincz  1
Affiliations

BlinkLinMulT: Transformer-Based Eye Blink Detection

Ádám Fodor et al. J Imaging. .

Abstract

This work presents BlinkLinMulT, a transformer-based framework for eye blink detection. While most existing approaches rely on frame-wise eye state classification, recent advancements in transformer-based sequence models have not been explored in the blink detection literature. Our approach effectively combines low- and high-level feature sequences with linear complexity cross-modal attention mechanisms and addresses challenges such as lighting changes and a wide range of head poses. Our work is the first to leverage the transformer architecture for blink presence detection and eye state recognition while successfully implementing an efficient fusion of input features. In our experiments, we utilized several publicly available benchmark datasets (CEW, ZJU, MRL Eye, RT-BENE, EyeBlink8, Researcher's Night, and TalkingFace) to extensively show the state-of-the-art performance and generalization capability of our trained model. We hope the proposed method can serve as a new baseline for further research.

Keywords: classification; deep learning; eye blink detection; multimodal fusion; transformers.

PubMed Disclaimer

Conflict of interest statement

The authors declare no conflict of interest.

Figures

Figure 1
Figure 1
BlinkLinMulT: a multimodal transformer architecture for blink presence detection and frame-wise eye state recognition.
Figure 2
Figure 2
Overview of the blink estimation pipeline.
Figure 3
Figure 3
Head pose angle dependence of BlinkLinMulT in the case of blink presence detection task. The head poses are predicted by 3DDFA_V2; the colors represent the F1 score calculated for the blink presence task, which is also written within the boxes together with the number of samples (in parenthesis) considered during the metric evaluations. F1 score cannot be calculated for those extreme cases, where closed-eye samples are not available. Test samples from all 5 sequence datasets are used for the experiment. Blinks can be predicted accurately in the case of frontal faces, and while the participant is looking up. Performance slightly decreases when the monitored person is looking down.
See this image and copyright information in PMC

References

    1. Chen B.C., Wu P.C., Chien S.Y. Real-time eye localization, blink detection, and gaze estimation system without infrared illumination; Proceedings of the 2015 IEEE International Conference on Image Processing (ICIP); Quebec City, QC, Canada. 27–30 September 2015; pp. 715–719. - DOI
    1. Phuong T.T., Hien L.T., Vinh N.D. An Eye Blink detection technique in video surveillance based on Eye Aspect Ratio; Proceedings of the 2022 24th International Conference on Advanced Communication Technology (ICACT); Pyeongchang, Republic of Korea. 2022. pp. 534–538.
    1. Kuwahara A., Nishikawa K., Hirakawa R., Kawano H., Nakatoh Y. Eye fatigue estimation using blink detection based on Eye Aspect Ratio Mapping(EARM) Cogn. Robot. 2022;2:50–59. doi: 10.1016/j.cogr.2022.01.003. - DOI
    1. Kraft D., Hartmann F., Bieber G. Camera-based Blink Detection using 3D-Landmarks; Proceedings of the 7th International Workshop on Sensor-Based Activity Recognition and Artificial Intelligence; Rostock, Germany. 19–20 September 2022; pp. 1–7.
    1. Zhao L., Wang Z., Zhang G., Qi Y., Wang X. Eye state recognition based on deep integrated neural network and transfer learning. Multimed. Tools Appl. 2018;77:1–24. doi: 10.1007/s11042-017-5380-8. - DOI

LinkOut - more resources