Intelligent ADL Recognition via IoT-Based Multimodal Deep Learning Framework

Abstract

Smart home monitoring systems via internet of things (IoT) are required for taking care of elders at home. They provide the flexibility of monitoring elders remotely for their families and caregivers. Activities of daily living are an efficient way to effectively monitor elderly people at home and patients at caregiving facilities. The monitoring of such actions depends largely on IoT-based devices, either wireless or installed at different places. This paper proposes an effective and robust layered architecture using multisensory devices to recognize the activities of daily living from anywhere. Multimodality refers to the sensory devices of multiple types working together to achieve the objective of remote monitoring. Therefore, the proposed multimodal-based approach includes IoT devices, such as wearable inertial sensors and videos recorded during daily routines, fused together. The data from these multi-sensors have to be processed through a pre-processing layer through different stages, such as data filtration, segmentation, landmark detection, and 2D stick model. In next layer called the features processing, we have extracted, fused, and optimized different features from multimodal sensors. The final layer, called classification, has been utilized to recognize the activities of daily living via a deep learning technique known as convolutional neural network. It is observed from the proposed IoT-based multimodal layered system's results that an acceptable mean accuracy rate of 84.14% has been achieved.

Keywords: IoT; activities of daily living recognition; deep learning; multimodal data; patient monitoring; smart homes.

Figure 1

The architecture diagram for multimodal IoT-based deep learning framework via ADL recognition.

Figure 2

Sample signals after filters applied for motion sensor data.

Figure 3

Detailed view of data segmentation applied over the inertial signal has been presented using multiple colors in the figure. The red dotted box shows single segment of data.

Figure 4

(a) Real video frame and (b) extracted human figure after background extraction for bending activity in Berkeley-MHAD dataset.

Figure 5

(a) Human silhouette (b) 2D stick model, where each red dot represents the body point detected, green lines show the upper body skeleton, and orange lines give the lower body skeleton.

Figure 6

Extracted LPCCs result for the Jumping Jacks ADL over Berkeley-MHAD dataset.

Figure 7

Upward motion direction flow in Jumping in Place ADL.

Figure 8

Features optimization via genetic algorithm explained through a detailed view.

Figure 9

Proposed CNN model for multimodal IoT-based ADL recognition over Berkeley-MHAD.

Figure 10

Sample frame sequences from the Berkeley-MHAD [22] dataset.

Figure 11

Sample frame sequences from Opportunity++ [21] dataset.

Figure 12

Examples of problematic ADL activities over Berkeley-MHAD, where red dotted circles point out the skeleton extraction problems.