Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2014 Aug 5;14(8):14253-77.
doi: 10.3390/s140814253.

Appearance-based multimodal human tracking and identification for healthcare in the digital home

Affiliations

Appearance-based multimodal human tracking and identification for healthcare in the digital home

Mau-Tsuen Yang et al. Sensors (Basel). .

Abstract

There is an urgent need for intelligent home surveillance systems to provide home security, monitor health conditions, and detect emergencies of family members. One of the fundamental problems to realize the power of these intelligent services is how to detect, track, and identify people at home. Compared to RFID tags that need to be worn all the time, vision-based sensors provide a natural and nonintrusive solution. Observing that body appearance and body build, as well as face, provide valuable cues for human identification, we model and record multi-view faces, full-body colors and shapes of family members in an appearance database by using two Kinects located at a home's entrance. Then the Kinects and another set of color cameras installed in other parts of the house are used to detect, track, and identify people by matching the captured color images with the registered templates in the appearance database. People are detected and tracked by multisensor fusion (Kinects and color cameras) using a Kalman filter that can handle duplicate or partial measurements. People are identified by multimodal fusion (face, body appearance, and silhouette) using a track-based majority voting. Moreover, the appearance-based human detection, tracking, and identification modules can cooperate seamlessly and benefit from each other. Experimental results show the effectiveness of the human tracking across multiple sensors and human identification considering the information of multi-view faces, full-body clothes, and silhouettes. The proposed home surveillance system can be applied to domestic applications in digital home security and intelligent healthcare.

PubMed Disclaimer

Figures

Figure 1.
Figure 1.
Framework of the proposed appearance-based multimodal human detection, tracking, and identification system for intelligent healthcare and security in digital homes.
Figure 2.
Figure 2.
Human tracking using a Kinect. (a) Skeleton tracking; (b) Face tracking.
Figure 3.
Figure 3.
FCT-based human identification and tracking.
Figure 4.
Figure 4.
Segmented body silhouettes in various facing directions. (a) Adult; (b) Child.
Figure 5.
Figure 5.
Two data fusion approaches in the proposed system: (a) multisensor fusion for human tracking; (b) multimodal fusion for human identification.
Figure 6.
Figure 6.
Cooperation among detection, tracking, and identification modules.
Figure 7.
Figure 7.
Typical layout of an elderly apartment. Two Kinects and two color cameras were installed to cover most open areas in the apartment.
Figure 8.
Figure 8.
Measurements of distinct sensors: (a) red cross mark for Kinect#1; (b) green plus mark for Kinect#2; (c) blue circle mark for color#1; (d) yellow triangle mark for color#2; (e) multisensor fusion; Thin white curve indicates the Kalman estimated trajectory, and thick purple curve represents the actual trajectory (ground truth).
Figure 9.
Figure 9.
Track-based voting for the human identification in five tracks with distinct IDs. (a) the voting result over time in the first track; (b) the winner ID with the highest votes and its confidence value over time in the first track; (cj) for the second∼fifth track.
Figure 10.
Figure 10.
Constructed FCTs of five adult males with similar body builds.
Figure 11.
Figure 11.
Example images captured in four living rooms in different apartments.

References

    1. Microsoft Corp. Kinect for Xbox 360 [(accessed on 30 May 2013)]. Available online: http://www.xbox.com/en-GB/kinect.
    1. Rice A., Phillips P., Natu V., An X., O'Toole A. Unaware Person Recognition from the Body When Face Identification Fails. Psychol. Sci. 2013;24:2235–2243. - PubMed
    1. Sixsmith A., Johnson N. A smart sensor to detect the falls of the elderly. IEEE Pervasive Comput. 2004;3:42–47.
    1. Tao S., Kudo M., Nonaka H. Privacy-preserved behavior analysis and fall detection by an infrared ceiling sensor network. Sensors. 2012;12:16920–16936. - PMC - PubMed
    1. Ni B., Dat N., Moulin P. RGBD-Camera Based Get-up Event Detection for Hospital Fall Prevention. Proceedings of International Conference on Acoustics, Speech and Signal Processing; Kyoto, Japan. 25–30 March 2012; pp. 1405–1408.

Publication types

LinkOut - more resources