Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2021 Feb 26;21(5):1636.
doi: 10.3390/s21051636.

LSTM Networks Using Smartphone Data for Sensor-Based Human Activity Recognition in Smart Homes

Sakorn Mekruksavanich  1 , Anuchit Jitpattanakul  2
Affiliations

LSTM Networks Using Smartphone Data for Sensor-Based Human Activity Recognition in Smart Homes

Sakorn Mekruksavanich et al. Sensors (Basel). .

Abstract

Human Activity Recognition (HAR) employing inertial motion data has gained considerable momentum in recent years, both in research and industrial applications. From the abstract perspective, this has been driven by an acceleration in the building of intelligent and smart environments and systems that cover all aspects of human life including healthcare, sports, manufacturing, commerce, etc. Such environments and systems necessitate and subsume activity recognition, aimed at recognizing the actions, characteristics, and goals of one or more individuals from a temporal series of observations streamed from one or more sensors. Due to the reliance of conventional Machine Learning (ML) techniques on handcrafted features in the extraction process, current research suggests that deep-learning approaches are more applicable to automated feature extraction from raw sensor data. In this work, the generic HAR framework for smartphone sensor data is proposed, based on Long Short-Term Memory (LSTM) networks for time-series domains. Four baseline LSTM networks are comparatively studied to analyze the impact of using different kinds of smartphone sensor data. In addition, a hybrid LSTM network called 4-layer CNN-LSTM is proposed to improve recognition performance. The HAR method is evaluated on a public smartphone-based dataset of UCI-HAR through various combinations of sample generation processes (OW and NOW) and validation protocols (10-fold and LOSO cross validation). Moreover, Bayesian optimization techniques are used in this study since they are advantageous for tuning the hyperparameters of each LSTM network. The experimental results indicate that the proposed 4-layer CNN-LSTM network performs well in activity recognition, enhancing the average accuracy by up to 2.24% compared to prior state-of-the-art approaches.

Keywords: HAR; LSTM; deep learning; feature extraction; smartphone sensor; time-series data.

PubMed Disclaimer

Conflict of interest statement

The authors declare no conflict of interest.

Figures

Figure 1
Figure 1
Sensor-based HAR approaches using (a) ML technique and (b) the DL technique.
Figure 2
Figure 2
The unfold architecture of one-layer standard LSTM.
Figure 3
Figure 3
The structure of an LSTM neuron.
Figure 4
Figure 4
The proposed framework of LSTM-based HAR.
Figure 5
Figure 5
Accelerometer data from UCI-HAR dataset.
Figure 6
Figure 6
Gyroscope data from UCI-HAR dataset.
Figure 7
Figure 7
Data segmentation process by a sliding window.
Figure 8
Figure 8
Activity label distribution of UCI-HAR dataset.
Figure 9
Figure 9
Histograms visualization of data from (a) accelerometer (b) gyroscope.
Figure 10
Figure 10
Vanilla LSTM network architecture.
Figure 11
Figure 11
2-Stacked LSTM network architecture.
Figure 12
Figure 12
3-Stacked LSTM network architecture.
Figure 13
Figure 13
CNN-LSTM network architecture.
Figure 14
Figure 14
The proposed architecture of 4-layer CNN-LSTM network.
Figure 15
Figure 15
The accuracy of each model after optimization process.
Figure 16
Figure 16
Receiver operating characteristic curves of five LSTM models.
Figure 17
Figure 17
Bar chart showing F1-score of the different LSTM networks on the UCI-HAR dataset.
Figure 18
Figure 18
Accuracy and loss examples of training process of Vanilla LSTM, 2-stacked LSTM, 3-stacked LSTM, and CNN-LSTM.
Figure 19
Figure 19
Accuracy and loss examples of training process of the proposed 4-layer CNN-LSTM.
See this image and copyright information in PMC

References

    1. Shih C.S., Chou J.J., Lin K.J. WuKong: Secure Run-Time environment and data-driven IoT applications for Smart Cities and Smart Buildings. J. Internet Serv. Inf. Secur. 2018;8:1–17.
    1. Jobanputra C., Bavishi J., Doshi N. Human Activity Recognition: A Survey. Procedia Comput. Sci. 2019;155:698–703. doi: 10.1016/j.procs.2019.08.100. - DOI
    1. Qi J., Yang P., Hanneghan M., Tang S., Zhou B. A Hybrid Hierarchical Framework for Gym Physical Activity Recognition and Measurement Using Wearable Sensors. IEEE Internet Things J. 2019;6:1384–1393. doi: 10.1109/JIOT.2018.2846359. - DOI
    1. Mekruksavanich S., Jitpattanakul A. Exercise Activity Recognition with Surface Electromyography Sensor using Machine Learning Approach; Proceedings of the 2020 Joint International Conference on Digital Arts, Media and Technology with ECTI Northern Section Conference on Electrical, Electronics, Computer and Telecommunications Engineering (ECTI DAMT and NCON); Pattaya, Thailand. 11–14 March 2020; pp. 75–78.
    1. Atapour C., Agrafiotis I., Creese S. Modeling Advanced Persistent Threats to enhance anomaly detection techniques. J. Wirel. Mob. Netw. Ubiquitous Comput. Dependable Appl. 2018;9:71–102.

LinkOut - more resources