Estimation of Respiratory Rate from Thermography Using Respiratory Likelihood Index

Abstract

Respiration is a key vital sign used to monitor human health status. Monitoring respiratory rate (RR) under non-contact is particularly important for providing appropriate pre-hospital care in emergencies. We propose an RR estimation system using thermal imaging cameras, which are increasingly being used in the medical field, such as recently during the COVID-19 pandemic. By measuring temperature changes during exhalation and inhalation, we aim to track the respiration of the subject in a supine or seated position in real-time without any physical contact. The proposed method automatically selects the respiration-related regions from the detected facial regions and estimates the respiration rate. Most existing methods rely on signals from nostrils and require close-up or high-resolution images, while our method only requires the facial region to be captured. Facial region is detected using YOLO v3, an object detection model based on deep learning. The detected facial region is divided into subregions. By calculating the respiratory likelihood of each segmented region using the newly proposed index, called the Respiratory Quality Index, the respiratory region is automatically selected and the RR is estimated. An evaluation of the proposed RR estimation method was conducted on seven subjects in their early twenties, with four 15 s measurements being taken. The results showed a mean absolute error of 0.66 bpm. The proposed method can be useful as an RR estimation method.

Keywords: deep learning; likelihood index; object detection; signal processing; thermal imaging; vital sign measurement.

Figure 1

Overview of proposed method: Firstly, a rectangular region of a face is detected by YOLO v3 [11]. The detected face region is divided into 4 × 6 subregions. Signal intensity is extracted from each subregion, and signal processing is applied. By calculating the respiratory likelihood of each segmented region using the newly proposed index “RQI (Respiratory Quality Index)”, the respiratory-related region is automatically selected and RR is estimated.

Figure 2

YOLOv3 architecture.

Figure 3

Respiratory-related signal: The signal extracted from the segmented region near the nose and its spectrum.

Figure 4

Noisy signal: The signal extracted from the segmented region far from the nose and its spectrum.

Figure 5

Verification of likelihood index: (a) shows signal obtained from the region where RR estimation error is lower than 1 bpm. (b) shows signal obtained from the region where RR estimation error is higher than 1 bpm. The x-axis represents the difference in respiratory rate [bpm] estimated from the time and frequency domains, respectively. The y-axis represents the Spectrum Index.

Figure 6

Training data: Images obtained from the thermal imaging camera for YOLOv3 training. The images were taken in a seated (a) and supine (b) position.

Figure 7

Bland–Altman plot of RR estimation: This plot shows the difference of RR against the mean on the x-axis. ${\mathrm{RR}}_{\mathrm{Pred}}$ and ${\mathrm{RR}}_{\mathrm{GT}}$ stand for the RR predicted by the proposed system and ground truth, respectively. The bias average is 0.19 bpm and the 95% limits of agreement vary between −1.9 and 2.3 bpm.