Mobile Robotic Platform for Contactless Vital Sign Monitoring

Abstract

The COVID-19 pandemic has accelerated methods to facilitate contactless evaluation of patients in hospital settings. By minimizing in-person contact with individuals who may have COVID-19, healthcare workers can prevent disease transmission and conserve personal protective equipment. Obtaining vital signs is a ubiquitous task that is commonly done in person by healthcare workers. To eliminate the need for in-person contact for vital sign measurement in the hospital setting, we developed Dr. Spot, a mobile quadruped robotic system. The system includes IR and RGB cameras for vital sign monitoring and a tablet computer for face-to-face medical interviewing. Dr. Spot is teleoperated by trained clinical staff to simultaneously measure the skin temperature, respiratory rate, and heart rate while maintaining social distancing from patients and without removing their mask. To enable accurate, contactless measurements on a mobile system without a static black body as reference, we propose novel methods for skin temperature compensation and respiratory rate measurement at various distances between the subject and the cameras, up to 5 m. Without compensation, the skin temperature MAE is 1.3°C. Using the proposed compensation method, the skin temperature MAE is reduced to 0.3°C. The respiratory rate method can provide continuous monitoring with a MAE of 1.6 BPM in 30 s or rapid screening with a MAE of 2.1 BPM in 10 s. For the heart rate estimation, our system is able to achieve a MAE less than 8 BPM in 10 s measured in arbitrary indoor light conditions at any distance below 2 m.

Figure 1

Operation of Dr. Spot. (a) Dr. Spot with the IR camera, three monochrome cameras, and iPad. (b) Patient screening detection with Dr. Spot. (c) Teleinterview with Dr. Spot. (d) Handheld controller for Dr. Spot with vital sign measurement results. (e) GUI for Dr. Spot showing the heart rate waveform, respiratory rate waveform, and estimated skin temperature.

Figure 2

System overview. (a) Operating principle of Dr. Spot showing IR images and RGB monochrome images. (a-i) IR camera waveform for respiratory rate measurement. (a-ii) RGB monochrome waveform for heart rate measurement. (b) Methodology used to calculate vital signs.

Figure 3

Thermal compensation of the IR camera for skin temperature measurement. (a) Measured skin temperature for subjects at 2 m. Different colors represent different ambient temperatures. (b) Skin temperature for subjects measured from 2 m to 5 m. Measured temperatures for each subject are translated vertically and grouped by ambient temperature. For the same subject and ambient temperature, measured temperatures in (a) represent the same data as the scaled temperatures in (b) at 2 m. (c) Effect of ambient temperature on the relationship between the measured temperature and the distance. (d) Dimensions of the face bounding box vs. subject's distance. (e) Measured and compensated skin temperatures for one subject. (f) Error analysis of measured skin temperatures and compensated skin temperatures for all subjects at 5 m. Data for 5 m at 19°C is imputed from the line of best fit.

Figure 4

Respiratory rate estimation with the IR camera for subjects with facemasks. (a) Estimated RR waveforms for one subject after ten different levels of exercise. IR camera temperature readings are normalized from 0 to 1. (b) Error analysis of RR estimation for waveforms in (a) with peak-to-peak (P2P) and fast Fourier transform (FFT). Various window sizes are used to determine the best parameters for quick screening and continuous monitoring. (c) Error analysis of RR estimation for 10 subjects standing 2 m from the IR camera. P2P and FFT are used at the optimal parameters. (d) Error analysis of RR estimation for 2 subjects standing 5 m from the IR camera. P2P and FFT are used at the optimal parameters.

Figure 5

Heart rate estimation with monochrome cameras. (a) RGB signals captured from the subject's forehead in a normal ambient lighting condition. Pulse signal determined from RGB signals using the POS method. (b) Effect of region of interest (ROI) on HR estimation accuracy. Skin. Seg. means skin segmentation. !forehead means excluding forehead. (c) Effect of detection latency (amount of time spent capturing data before estimating HR) on HR estimation accuracy. (d) Effect of ROI resolution on HR estimation accuracy. The ROI resolution (i.e., forehead resolution) captured by a 5 MP monochrome camera at 1 m, 2 m, and 3 m is displayed.