Deep Learning Methods for Remote Heart Rate Measurement: A Review and Future Research Agenda

Abstract

Heart rate (HR) is one of the essential vital signs used to indicate the physiological health of the human body. While traditional HR monitors usually require contact with skin, remote photoplethysmography (rPPG) enables contactless HR monitoring by capturing subtle light changes of skin through a video camera. Given the vast potential of this technology in the future of digital healthcare, remote monitoring of physiological signals has gained significant traction in the research community. In recent years, the success of deep learning (DL) methods for image and video analysis has inspired researchers to apply such techniques to various parts of the remote physiological signal extraction pipeline. In this paper, we discuss several recent advances of DL-based methods specifically for remote HR measurement, categorizing them based on model architecture and application. We further detail relevant real-world applications of remote physiological monitoring and summarize various common resources used to accelerate related research progress. Lastly, we analyze the implications of research findings and discuss research gaps to guide future explorations.

Keywords: deep learning; heart rate measurement; noncontact monitoring; rPPG; remote photoplethysmography.

Figure 1

Principle of remote photoplethysmography (rPPG) based on the dichromatic reflection model (DRM). The digital camera captures the specular and diffuse reflection from ambient light. The specular reflection contains surface information that does not relate to physiological signals, while the diffuse reflection is modulated by blood flow. The rPPG signal can be obtained from further signal processing.

Figure 2

General framework of conventional methods for remote heart rate (HR) measurement. Face detection (e.g., Viola and Jones algorithm) is performed on the video frames, resulting in the red bounding box on the face. Next, regions of interest (ROIs) such as the cheeks marked by the black boxes are selected within the face box. The rPPG signal is extracted from the pixels within the ROIs. Lastly, post-processing techniques, such as frequency analysis (e.g., Fourier transform) and peak detection, are applied on the extracted signal to estimate HR.

Figure 3

Schematic diagram of end-to-end deep learning (DL) methods and hybrid DL methods. End-to-end DL methods directly output the HR or rPPG signal with a single model, while hybrid DL methods utilize DL techniques at various stages.

Figure 4

Architecture of DeepPhys [27].

Figure 5

Architecture of 3D CNN PhysNet [29].

Figure 6

Architecture used in Yu et al. [30].

Figure 7

Architecture of AutoHR [31].

Figure 8

Architecture of ETA-rPPGNet [34].

Figure 9

Architecture of RNN-based PhysNet [29].

Figure 10

Architecture used in Deep-HR [45] for signal optimization.

Figure 11

Architecture used in Bian et al. [47].

Figure 12

Architecture used in Deep-HR [45] for signal extraction.

Figure 13

Architecture of Siamese-rPPG [51].

Figure 14

Architecture of Meta-rPPG [57].

Figure 15

Architecture of PRNet [58].

Figure 16

Architecture of PulseGAN [59].

Figure 17

Architecture used in Yang et al. [64].

Figure 18

General procedure of constructing a spatio-temporal map. Firstly, the images are aligned and ROI selection is performed to obtain ROI images. Then, these ROI images are divided into several ROI blocks. Next, within each block, the average color value is calculated for each color channel. After that, the average color value of each channel at the same block but different frames are concatenated into temporal sequences. Finally, the temporal sequences of each block are placed into rows to form a spatio-temporal map.

Figure 19

Architecture of RhythmNet [68].

Figure 20

Architecture used in Huang et al. [56].