Monitoring Pulse Rate in the Background Using Front Facing Cameras of Mobile Devices

Abstract

We propose a novel framework to passively monitor pulse rate during the time spent by users on their personal mobile devices. Our framework is based on passively capturing the user's pulse signal using the front-facing camera. Signal capture is performed in the background, while the user is interacting with the device as he/she normally would, e.g., watch movies, read emails, text, and play games. The framework does not require subject participation with the monitoring procedure, thereby addressing the well-known problem of low adherence with such procedures. We investigate various techniques to suppress the impact of spontaneous user motion and fluctuations in ambient light conditions expected in non-participatory environments. Techniques include traditional signal processing, machine learning classifiers, and deep learning methods. Our performance evaluation is based on a clinical study encompassing 113 patients with a history of atrial fibrillation (Afib) who are passively monitored at home using a tablet for a period of two weeks. Our results show that the proposed framework accurately monitors pulse rate, thereby providing a gateway for long-term monitoring without relying on subject participation or the use of a dedicated wearable device.

Fig. 1:

Data collection architecture

Fig. 2:

Typical use case in the data synchronization process

Fig. 3:

Histogram of the number of recordings per hour of the day across all subjects.

Fig. 4:

Histogram of HR measured by ECG across all subjects.

Fig. 5:

Histogram of the number of recordings per subject. The threshold is used for under and oversampling methods.

Fig. 6:

Examples of a VPG signal passively captured in the background with the synchronized ECG signal. (a) High-quality signal and its PSD response (b) Low-quality signal and its PSD.

Fig. 7:

ROC plots show the performance of the SVM, RF, XGBoost, QDA, CNN, and BTC models.

Fig. 8:

False Discovery Rate and Yield as a function of threshold for the classifiers.

Fig. 9:

Bland-Altman plots for various classifiers and a +-5BPM error range indication. (a) SVM, (b) RF, (c) XGBoost, (d) QDA, (e) CNN, and (f) BTC.