Breathing Analysis Using Thermal and Depth Imaging Camera Video Records

Abstract

The paper is devoted to the study of facial region temperature changes using a simple thermal imaging camera and to the comparison of their time evolution with the pectoral area motion recorded by the MS Kinect depth sensor. The goal of this research is to propose the use of video records as alternative diagnostics of breathing disorders allowing their analysis in the home environment as well. The methods proposed include (i) specific image processing algorithms for detecting facial parts with periodic temperature changes; (ii) computational intelligence tools for analysing the associated videosequences; and (iii) digital filters and spectral estimation tools for processing the depth matrices. Machine learning applied to thermal imaging camera calibration allowed the recognition of its digital information with an accuracy close to 100% for the classification of individual temperature values. The proposed detection of breathing features was used for monitoring of physical activities by the home exercise bike. The results include a decrease of breathing temperature and its frequency after a load, with mean values -0.16 °C/min and -0.72 bpm respectively, for the given set of experiments. The proposed methods verify that thermal and depth cameras can be used as additional tools for multimodal detection of breathing patterns.

Keywords: breathing disorders detection; depth sensors; facial temperature distribution; machine learning; multimodal signals; thermography.

Figure 1

Data acquisition presenting (a) specification of the fixed and moving region of interest (ROI); (b) facial regions of different mean temperatures in the selected thermal image frame.

Figure 2

Thermal image analysis presenting (a) the block diagram of the thermal camera; (b) a selected image frame of a compact surface with equal temperature values; and (c) the distribution of values recorded by individual pixels.

Figure 3

Principle of data acquisition by an MS Kinect depth sensor, presenting a selected depth frame with the regions of interest used to detect the chest movement.

Figure 4

Thermal imaging camera accuracy analysis presenting (a) a selected thermal image with the region of interest (ROI) and the temperature bar; (b) areas specifying subregions with temperatures in the range of 26 °C and 28 °C; (c) the ROI temperature surface plot; (d) a global analysis of percentage of thermal pixels in the selected range of 26 °C and 28 °C; and (e) percentage values of thermal pixels in the selected range of 26 °C and 28 °C detected in two selected areas and their evolution for 24 images recorded for two minutes with a sampling period of 5 s.

Figure 5

Principle of the use of the thermal imaging camera for breathing analysis, presenting the time evolution of the mean breathing temperature detected in the selected mouth region (a) during the physical exercise with the higher temperature and breathing frequency; (b) during the restful period with the lower temperature and breathing frequency.

Figure 6

Data processing presenting (a) signals recorded by the thermal imaging camera and the MS Kinect depth sensor; and (b) detection of the breathing frequency from the fixed ROI, the moving ROI, and the MS Kinect depth sensor.

Figure 7

An example of records and results evaluated during two 10 min long segments of the physical exercises followed by two 10 min long resting time periods presenting (a) the time evolution of breathing temperatures; (b) the time evolution of the mean breathing temperature in a time window 60 s long; and (c) associated breathing frequency.