Linguistic Summarization of Video for Fall Detection Using Voxel Person and Fuzzy Logic

Abstract

In this paper, we present a method for recognizing human activity from linguistic summarizations of temporal fuzzy inference curves representing the states of a three-dimensional object called voxel person. A hierarchy of fuzzy logic is used, where the output from each level is summarized and fed into the next level. We present a two level model for fall detection. The first level infers the states of the person at each image. The second level operates on linguistic summarizations of voxel person's states and inference regarding activity is performed. The rules used for fall detection were designed under the supervision of nurses to ensure that they reflect the manner in which elders perform these activities. The proposed framework is extremely flexible. Rules can be modified, added, or removed, allowing for per-resident customization based on knowledge about their cognitive and physical ability.

Fig. 1

Voxel person construction. Cameras capture the raw video from different viewpoints, silhouette extraction is performed for each camera, voxel sets are calculated from the silhouettes for each camera, and the voxel sets are intersected to calculate voxel person.

Fig. 2

Fuzzy inference outputs plotted for a voxel person fall. The x-axis is time, measured in frames, and the y-axis is the fuzzy inference outputs. The red curve is upright, the blue curve is in-between, and the green curve is on-the-ground. The frame rate was 3 per second, so the above plot is approximately 23 seconds of activity.

Fig. 3

Color-coding of voxel person according to the membership output values. Voxel persons color is a mixture of the fuzzy rule system outputs. The upright state determines the amount of red, in-between is green, and on-the-ground is blue.

Fig. 4

Activity recognition framework, which utilizes a hierarchy of fuzzy logic based on voxel person representation. The first level is reasoning about the state of the individual. Linguistic summarizations are produced and fuzzy logic is used again to reason about human activity.

Fig. 5

Detection of a large recent change in voxel person’s speed. (a) Motion vector magnitudes are computed, (b) a fixed size window, placed directly before the start of the summarization, is smoothed with a mean filter, and (c) the maximum of the derivative of the filtered motion vector magnitudes is found in the first and second halves of the window. The feature is the ratio of the two maximum values.

Fig. 6

Example images and their corresponding silhouettes from the fall data set. Lying on the couch and sitting on the chair with feet up activities, which could be misinterpreted as a fall, are not recognized as a fall in our system, an advantage of rule-based reasoning and knowledge about three-dimensional voxel person.

Fig. 7

Approximately 11 minutes of video analysis, 2,042 frames total. A total of 4 falls occurred and 38 linguistic summarizations were produced. The upright membership is shown in red, in-between membership is shown in blue, and on-the-ground is shown in green. Dashed vertical purple lines are the manually inserted moments where a fall occurred.

Fig. 8

Fifty-eight frames (approximately 19 seconds) from a sequence where the person fell and was able to get back up. Red is upright, blue is in-between, and green is on-the-ground.

Fig. 9

Sixty-three frames (approximately 21 seconds) where the person fell and tried to get back up three times. Red is upright, blue is in-between, and green is on-the-ground.