Human Activity Recognition Algorithm with Physiological and Inertial Signals Fusion: Photoplethysmography, Electrodermal Activity, and Accelerometry

Abstract

Inertial signals are the most widely used signals in human activity recognition (HAR) applications, and extensive research has been performed on developing HAR classifiers using accelerometer and gyroscope data. This study aimed to investigate the potential enhancement of HAR models through the fusion of biological signals with inertial signals. The classification of eight common low-, medium-, and high-intensity activities was assessed using machine learning (ML) algorithms, trained on accelerometer (ACC), blood volume pulse (BVP), and electrodermal activity (EDA) data obtained from a wrist-worn sensor. Two types of ML algorithms were employed: a random forest (RF) trained on features; and a pre-trained deep learning (DL) network (ResNet-18) trained on spectrogram images. Evaluation was conducted on both individual activities and more generalized activity groups, based on similar intensity. Results indicated that RF classifiers outperformed corresponding DL classifiers at both individual and grouped levels. However, the fusion of EDA and BVP signals with ACC data improved DL classifier performance compared to a baseline DL model with ACC-only data. The best performance was achieved by a classifier trained on a combination of ACC, EDA, and BVP images, yielding F1-scores of 69 and 87 for individual and grouped activity classifications, respectively. For DL models trained with additional biological signals, almost all individual activity classifications showed improvement (p-value < 0.05). In grouped activity classifications, DL model performance was enhanced for low- and medium-intensity activities. Exploring the classification of two specific activities, ascending/descending stairs and cycling, revealed significantly improved results using a DL model trained on combined ACC, BVP, and EDA spectrogram images (p-value < 0.05).

Keywords: accelerometer; blood volume pulse (BVP); convolutional neural network (CNN); electrodermal activity (EDA); human activity recognition; machine learning; multi-modal classification.

Figure 1

Flowchart for HAR algorithm development; ACC, BVP, and EDA data were collected from a wristband device. RF algorithms were trained on features extracted from signals, and DL algorithms were trained on spectrogram images.

Figure 2

Paired time and time–frequency plots (STFT) of each activity corresponding to their ACC, BVP, and EDA segments (top to bottom) from a 20-year-old female subject. Activities represented are lying, standing, walking, brisk walking, jogging, running, stairs, and cycling (left to right). The colors in spectrogram represents the amplitude of a specific frequency at a given time, ranging from dark blues for low amplitudes and dark red, indicating stronger amplitudes.