Person Re-Identification with RGB-D and RGB-IR Sensors: A Comprehensive Survey

Abstract

Learning about appearance embedding is of great importance for a variety of different computer-vision applications, which has prompted a surge in person re-identification (Re-ID) papers. The aim of these papers has been to identify an individual over a set of non-overlapping cameras. Despite recent advances in RGB-RGB Re-ID approaches with deep-learning architectures, the approach fails to consistently work well when there are low resolutions in dark conditions. The introduction of different sensors (i.e., RGB-D and infrared (IR)) enables the capture of appearances even in dark conditions. Recently, a lot of research has been dedicated to addressing the issue of finding appearance embedding in dark conditions using different advanced camera sensors. In this paper, we give a comprehensive overview of existing Re-ID approaches that utilize the additional information from different sensor-based methods to address the constraints faced by RGB camera-based person Re-ID systems. Although there are a number of survey papers that consider either the RGB-RGB or Visible-IR scenarios, there are none that consider both RGB-D and RGB-IR. In this paper, we present a detailed taxonomy of the existing approaches along with the existing RGB-D and RGB-IR person Re-ID datasets. Then, we summarize the performance of state-of-the-art methods on several representative RGB-D and RGB-IR datasets. Finally, future directions and current issues are considered for improving the different sensor-based person Re-ID systems.

Keywords: RGB–D sensors; RGB–IR sensors; cross-modal; multi-modal; re-identification; video surveillance.

Figure 1

Sample images showing the challenges related to camera variations and environmental conditions in the Re-ID problem. (a) shows pose and viewpoint variations; (b) background clutter; (c) partial occlusion; and (d) illumination variations. Images were taken from the public standard datasets IAS-Lab [14], i-LIDS [15] and Market-1501 [16].

Figure 2

Sample images showing the RGB, depth, and skeleton information of an individual in the RobotPKU RGBD-ID [33] dataset.

Figure 3

General categories of RGB–D and IR sensor-based person re-identification systems.

Figure 4

Different multi-modal person re-identification approaches (feature/score-level fusion), where each approach uses different modalities. (a) RGB and depth modalities; (b) RGB and skeleton information; (c) depth and skeleton information; (d) RGB, depth, and skeleton information; and (e) RGB, depth, and thermal images [44].

Figure 5

A flowchart of the RGB–depth image-based Re-ID framework. (a) Score-level fusion and (b) feature-level fusion techniques.

Figure 6

A flowchart of the RGB–skeleton information-based Re-ID framework.

Figure 7

A flowchart of the depth–skeleton information-based Re-ID framework.

Figure 8

A flowchart of the RGB–depth–skeleton-based Re-ID.

Figure 9

A flowchart of RGB–depth–skeleton-based Re-ID.

Figure 10

Illustration of cross-modal person Re-ID. (a) Sketch–RGB cross-modal Re-ID uses sketch as a query and RGB as a gallery or vice versa. (b) IR–RGB cross-modal Re-ID uses IR as a query and RGB as a gallery or vice versa. (c) Depth–RGB cross-modal Re-ID uses depth as a query and RGB as a gallery or vice versa.

Figure 11

The general architecture of RGB–depth cross-modal Re-ID.

Figure 12

Sample images showing frontal, rear, and side views of an individual, with the camera installed at a horizontal viewpoint.

Figure 13

Sample image taken from the TVPR dataset showing an overhead view of a person.

Figure 14

Sample images showing three modalities (RGB, depth, and thermal) taken from the RGB-D-T dataset.

Figure 15

Example images showing sketches of different persons and their corresponding RGB images.

Figure 16

The left figure shows sample images from the SYSU-MM01 dataset and the right one shows sample images from the RegDB dataset.