Recognition of Activities of Daily Living with Egocentric Vision: A Review

Abstract

Video-based recognition of activities of daily living (ADLs) is being used in ambient assisted living systems in order to support the independent living of older people. However, current systems based on cameras located in the environment present a number of problems, such as occlusions and a limited field of view. Recently, wearable cameras have begun to be exploited. This paper presents a review of the state of the art of egocentric vision systems for the recognition of ADLs following a hierarchical structure: motion, action and activity levels, where each level provides higher semantic information and involves a longer time frame. The current egocentric vision literature suggests that ADLs recognition is mainly driven by the objects present in the scene, especially those associated with specific tasks. However, although object-based approaches have proven popular, object recognition remains a challenge due to the intra-class variations found in unconstrained scenarios. As a consequence, the performance of current systems is far from satisfactory.

Keywords: activity recognition; ambient assisted living; egocentric vision; wearable cameras.

Figure 1

Human behaviour analysis tasks: classification (reprinted from [9]).

Figure 2

Pipeline for human behaviour analysis at the motion level.

Figure 3

Gaze prediction without reference to saliency or the activity model [32]. Egocentric features, which are head/hand motion and hand location/pose, are leveraged to predict gaze. A model that takes account of eye-hand and eye-head coordination, combined with temporal dynamics of gaze, is designed for gaze prediction. Only egocentric videos have been used, and the performance is compared to the ground truth acquired with an eye tracker (reprinted from [32]).

Figure 4

A part-based object model for a stove in an activities of daily living (ADLs) dataset using a HOG descriptor (reprinted and adapted from [11]).

Figure 5

Pixel-level hand detection under varying illumination and hand poses (reprinted from [50]).

Figure 6

Action recognition based on changes in the state of objects (reprinted from [64]).

Figure 7

Flow chart for indoor office task classification (reprinted from [15]).

Figure 8

Temporal pyramid representation of a video sequence (reprinted from [12]).

Figure 9

Detection of hands and objects in the ADL dataset (reprinted from [11]).

Figure 10

Graph-based framework’s model. An activity y is a sequence of actions $a_i$, and each action is represented by objects and hands. During testing, objects and hand labels $h_k$ are assigned to regions $x_k$ (reprinted from [8]).

Figure 11

Visual explanation of the proposed method. The region around the fixation point is extracted and encoded using a gradient-based template. These templates are used to build the vocabulary, which is then applied to generate a BoW representation for an activity (reprinted from [73]).

Figure 12

Various graphical models for activity and object recognition in which A, O, R and V represent activity, object, RFID and video frame, respectively (reprinted from [42]).