REAL-Time Smartphone Activity Classification Using Inertial Sensors-Recognition of Scrolling, Typing, and Watching Videos While Sitting or Walking

Abstract

By developing awareness of smartphone activities that the user is performing on their smartphone, such as scrolling feeds, typing and watching videos, we can develop application features that are beneficial to the users, such as personalization. It is currently not possible to access real-time smartphone activities directly, due to standard smartphone privileges and if internal movement sensors can detect them, there may be implications for access policies. Our research seeks to understand whether the sensor data from existing smartphone inertial measurement unit (IMU) sensors (triaxial accelerometers, gyroscopes and magnetometers) can be used to classify typical human smartphone activities. We designed and conducted a study with human participants which uses an Android app to collect motion data during scrolling, typing and watching videos, while walking or seated and the baseline of smartphone non-use, while sitting and walking. We then trained a machine learning (ML) model to perform real-time activity recognition of those eight states. We investigated various algorithms and parameters for the best accuracy. Our optimal solution achieved an accuracy of 78.6% with the Extremely Randomized Trees algorithm, data sampled at 50 Hz and 5-s windows. We conclude by discussing the viability of using IMU sensors to recognize common smartphone activities.

Keywords: machine learning; real-time classification; smartphone IMU sensors; smartphone activity recognition.

Figure 1

Ten seconds of acceleration data for a. typing (left) and b. watching (right), both while seated. The acceleration data for typing shows greater variation while the data for watching is smoother.

Figure 2

Study data collection and processing workflow: a user uses the smartphone for specified time periods and specified activities, while sensor data is recorded. In real time, data is transmitted and classified.

Figure 3

Foyer of the building used for the walking condition, selected to simulate the real walking conditions that smartphone users might experience in daily life.

Figure 4

Screenshots of the four tasks which participants completed as part of the study, while software collects the IMU sensor data. Each of the four tasks is completed while sitting and while walking.

Figure 5

Correlation table before and after feature extraction.

Figure 6

Autocorrelation for the final selected features.

Figure 7

Confusion matrix for the three models (Multi-Layer Perceptron (MLP), Random Forest (RF) and Extremely Randomized Trees (ET)) with different frequencies.

Figure 8

Confusion matrix for the three models (MLP, RF and ET) with different frequencies, after combining labels.

Figure 9

Accuracy vs sample size plot for the three models (MLP, RF and ET).