iLid: Low-power Sensing of Fatigue and Drowsiness Measures on a Computational Eyeglass

Abstract

The ability to monitor eye closures and blink patterns has long been known to enable accurate assessment of fatigue and drowsiness in individuals. Many measures of the eye are known to be correlated with fatigue including coarse-grained measures like the rate of blinks as well as fine-grained measures like the duration of blinks and the extent of eye closures. Despite a plethora of research validating these measures, we lack wearable devices that can continually and reliably monitor them in the natural environment. In this work, we present a low-power system, iLid, that can continually sense fine-grained measures such as blink duration and Percentage of Eye Closures (PERCLOS) at high frame rates of 100fps. We present a complete solution including design of the sensing, signal processing, and machine learning pipeline; implementation on a prototype computational eyeglass platform; and extensive evaluation under many conditions including illumination changes, eyeglass shifts, and mobility. Our results are very encouraging, showing that we can detect blinks, blink duration, eyelid location, and fatigue-related metrics such as PERCLOS with less than a few percent error.

Keywords: Applied computing → Consumer health; Blinks; Computing methodologies → Interest point and salient region detections; Drowsiness; Eyeglasses; Eyelid; Fatigue; Human-centered computing → Mobile devices; PERCLOS; User studies.

Fig. 1

The upper eyelid detection pipeline: 1) 4 columns of pixels near the center of the image are sampled, 2) pixel values are horizontally projected, 3) values are median filtered, 4) values are convolved with a box filter for edge detection, 5) specular reflection is detected and removed (top), 6) pupil is detected and removed (bottom), 7) the upper eyelid position is estimated.

Fig. 2

Increasing the number of sampled columns to ameliorate the specular reflection problem: Figure (a) shows the noisy eyelid detector output in the existence of specular reflection when sampling only 4 columns of pixels, while Figure (b) shows the output of the eyelid detector for the same data segment after increasing the number of sampled columns to 11.

Fig. 3

Filtering the eyelid detector output: The high frequency noise in the raw eyelid detector output (a) is removed by applying a low-pass filter (b).

Fig. 4

Blink detection pipeline: 1) A moving window selects samples from the eyelid position sequence, 2) the template matching scores over the window are generated, 3) the scores are used as feature vectors for a logistic regression classifier that detects blinks, 4) detected blinks after eliminating redundant detections.

Fig. 5

The three blink templates representing, from left to right, slow, normal, and fast blinks. The templates are horizontally scaled versions of the same blink profile defined by the five key points A through E which have been interpolated using cubic splines. Points A and E determine the duration of the blink T. Point B sitting at 0.25T on the horizontal axis determines the height of the blink H. Points C and D which give a slide pattern to the blink profile in the eyeopening phase of the blink have relative coordinates of (0.5T, 0.45H) and (0.9T, 0.02H), respectively.

Fig. 6

Blink duration fix: The rough position of the blinks is first detected(a) and then a local search is performed to locate the ending points of the blinks(b).

Fig. 7

Upper eyelid location time series

Fig. 8

Eyeglass platform containing an eye-facing imager, two NIR LEDs, and a PCB board with the micro-controller, Bluetooth, and other modules on the left, as well as the battery board on the right.

Fig. 9

Breakdown of each analytic component’s contribution to overall results. The baseline represents basic eyelid detection (ED) + logistic regression classification, the second bar includes specular reflection and pupil removal (SPF) and the final bar includes templates (TM) (i.e. the entire pipeline).

Fig. 10

Blink detector performance for individual subjects.

Fig. 11

Blink duration measurement error CDF.

Fig. 12

Eyelid detector performance in different eye closure states.

Fig. 13

Eyelid detection performance in different illumination conditions.

Fig. 14

Blink detection performance versus different eyeglass displacement values.

Fig. 15

Eyelid detection performance for different eyeglass displacement values.

Fig. 16

Eyelid detection performance compared in mobile and stationary scenarios.

Fig. 17

PERCLOS results for the driving simulator experiment. The vertical axis represents PERCLOS value and the horizontal axis shows time in seconds. The measured PERCLOS values is represented with the blue solid line and the dashed orange line shows the ground truth.

Fig. 18

Comparing blink detection performance of a preliminary version of iLid with JINS MEME. The performance of both platforms are shown in three different scenarios. Error bars represent standard error.

Fig. 19

JINS MEME’s EOG signal output. The signal consists of 4 channels of vertical (V), horizontal (H), left (L), and right (R). Figure (a) shows the EOG signal corresponding to a blink when the person is completely stationary. Figure (b) shows the EOG signal relating to 4 different states of the eye. Segment 0 shows a period when the eyes are wide open. Segment 1 corresponds to a case when the eyes are slowly closed which is followed by segment 2, in which the eyes are kept closed. In segment 3 the eyes are slowly opened again. As it can be seen the EOG signal is the same for segments 2 and 0 which relate to steady cases when the eyes are kept closed and open respectively. Figure (c) also depicts the cross talk on the EOG signal induced by talking.