Robust Eye Blink Detection Using Video Vision Transformer

Abstract

Eye blinking is a momentary body activity that is commonly used to diagnose mental state, fatigue detection, and monitor health. With the increase in applications, many researchers have investigated how to effectively detect eye blinks. Approaches range from bio-signal analysis to typical image processing, with a current focus on deep learning-based methods. However, most eyeblink datasets have been collected exclusively in frontal or well-controlled environments, which is a limitation in validating previous methods in real-world scenarios. In this study, a video vision transformer (ViViT) is presented based on dual embedding for robust eye blink detection. We modified the existing tubelet embedding for our task and added a new residual embedding. Both were designed to capture the key movements and slight changes in the frame over time. In addition, we created a new dataset and used it with existing public datasets to evaluate the robustness of the proposed method from different aspects. We also performed an analysis of the differences in multiple camera perspectives. The results showed that the dual-embedding-based vision transformer consistently had the best performance, and proved to be robust in different environments. In addition, we found that certain camera positions affected accuracy. Finally, we discuss the use of attention score for blink timing and the potential implications for robust eyeblink detection.