Fine-Grained Intoxicated Gait Classification Using a Bilinear CNN

Abstract

Consuming excessive amounts of alcohol causes impaired mobility and judgment and driving accidents, resulting in more than 800 injuries and fatalities each day. Passive methods to detect intoxicated drivers beyond the safe driving limit can facilitate Just-In-Time alerts and reduce Driving Under the Influence (DUI) incidents. Popularly-owned smartphones are not only equipped with motion sensors (accelerometer and gyroscope) that can be employed for passively collecting gait (walk) data but also have the processing power to run computationally expensive machine learning models. In this paper, we advance the state-of-the-art by proposing a novel method that utilizes a Bi-linear Convolution Neural Network (BiCNN) for analyzing smartphone accelerometer and gyroscope data to determine whether a smartphone user is over the legal driving limit (0.08) from their gait. After segmenting the gait data into steps, we converted the smartphone motion sensor data to a Gramian Angular Field (GAF) image and then leveraged the BiCNN architecture for intoxication classification. Distinguishing GAF-encoded images of the gait of intoxicated vs. sober users is challenging as the differences between the classes (intoxicated vs. sober) are subtle, also known as a fine-grained image classification problem. The BiCNN neural network has previously produced state-of-the-art results on fine-grained image classification of natural images. To the best of our knowledge, our work is the first to innovatively utilize the BiCNN to classify GAF encoded images of smartphone gait data in order to detect intoxication. Prior work had explored using the BiCNN to classify natural images or explored other gait-related tasks but not intoxication Our complete intoxication classification pipeline consists of several important pre-processing steps carefully adapted to the BAC classification task, including step detection and segmentation, data normalization to account for inter-subject variability, data fusion, GAF image generation from time-series data, and a BiCNN classification model. In rigorous evaluation, our BiCNN model achieves an accuracy of 83.5%, outperforming the previous state-of-the-art and demonstrating the feasibility of our approach.

Keywords: Blood Alchol Content (BAC); Convolutional Neural Networks (CNNs); Gait Analysis; Neural Networks.

Fig. 1.

Gait cycle, stance and foot placement during one step (top). Corresponding accelerometer magnitude (bottom)

Fig. 2.

Three modalities of gait analysis: Vision, pressure and accelerometers [22]

Fig. 3.

(a) Accelerometer’s SVM signal gait cycle; (b) Recurrence plot image; (c) GAF Imaging

Fig. 4.

Illustration of time series encoding through GAF

Fig. 5.

Bilinear CNN Model Structure

Fig. 6.

A sample of BAC values recorded based on our alcohol administration protocol

Fig. 7.

Summary of alcohol administration protocol

Fig. 8.

Map of walking scenario at Butler Hospital

Fig. 9.

Overview diagram of our intoxication classification pipeline

Fig. 10.

Examples of normalized and fixed-length (a) Sober gait; (b) Intoxicated gait;

Fig. 11.

Implementation of data preprocessing.(SVM denotes Signal Vector Magnitude; NCC denotes Normalized Cross Correlation;)

Fig. 12.

(a) Accelerometer signal magnitude of one walking scenario and three templates generated through normalized cross correlation; (b) Utilizing the rank-1 templates to find all possible gait salient points; (c) The salient points (peak/valley) index recorded during a scenario

Fig. 13.

Example: Step Segmentation and Normalization

Fig. 14.

Converting Time-series Sober Step into a GAF Image (a) Original accelerometor signal of one sober segmented step (b) Representing a step cycle segment in Cartesian coordinates (c) Representing a step cycle in GAF

Fig. 15.

Converting a Intoxicated Time-series Step into a GAF Image (a) Original accelerometer signal of one intoxicated segmented step (b) Representing a step cycle segment on Cartesian coordinates (c) Representing a step cycle in GAF

Fig. 16.

(a) Normalized accelerometer signal with a fixed length of 200 data points; (b) Representing the accelerometer magnitude using a GAF image; In both (a) and ( b), the first row shows the sober step; the corresponding images in the second rows are intoxicated steps of the same subject; Parameters used figures (a) and (b) are selected to generate illustrative representations that explain GAF concepts.

Fig. 17.

Number of data instances collected at various BAC levels

Fig. 18.

Training Test Accuracy

Fig. 19.

(a)GAF images ecoded from three walking gait(b)Visualization of Activation heatmaps of three GAF images.(Notes: the high intensity visuals(black part) reflects model’s interest)

Fig. 20.

Confidence of intoxication classification of subjects with various models

Fig. 21.

Running time of the main stages of our BiCNN model