Camera-based photoplethysmography in an intraoperative setting

Abstract

Background: Camera-based photoplethysmography (cbPPG) is a measurement technique which enables remote vital sign monitoring by using cameras. To obtain valid plethysmograms, proper regions of interest (ROIs) have to be selected in the video data. Most automated selection methods rely on specific spatial or temporal features limiting a broader application. In this work, we present a new method which overcomes those drawbacks and, therefore, allows cbPPG to be applied in an intraoperative environment.

Methods: We recorded 41 patients during surgery using an RGB and a near-infrared (NIR) camera. A Bayesian skin classifier was employed to detect suitable regions, and a level set segmentation approach to define and track ROIs based on spatial homogeneity.

Results: The results show stable and homogeneously illuminated ROIs. We further evaluated their quality with regards to extracted cbPPG signals. The green channel provided the best results where heart rates could be correctly estimated in 95.6% of cases. The NIR channel yielded the highest contribution in compensating false estimations.

Conclusions: The proposed method proved that cbPPG is applicable in intraoperative environments. It can be easily transferred to other settings regardless of which body site is considered.

Keywords: Camera-based photoplethysmography; Intraoperative monitoring; Level set methods; Remote monitoring; Spatial homogeneity.

Fig. 1

CbPPG setup during surgery. (1) Construction with adjustable arm for the sensing system. (2) Sensing system (enlarged on the right) including NIR illumination, NIR camera, and RGB camera. (3) Recording PC. (4) Patient (face directed towards the cameras). (5) Surgeons and clinical staff

Fig. 2

Example for a segmentation process using level set methods. a Initialization point. b Point during segmentation. c Point when process has converged. The inside region ${\mathrm{\Omega }}_1$ and the outside region ${\mathrm{\Omega }}_2$ are implicitly described and changed by $\mathrm{\Phi }.$ The contour $\mathrm{\Phi }=0$ is depicted separately in the images below the graphs. Please note that t represents the segmentation time for an image and does not refer to the time component in the videos. The eye section was blurred if it was visible

Fig. 3

Program structure of the presented ROI detection and tracking algorithm. a ROI detector which (initially) detects the skin, finds the ROI and registers and adapts the result for the NIR image. b Simplified flowchart of the whole program (detection and tracking) which runs separately for the RGB and NIR video. For some transitions between the program blocks, the data types are given (I: image, $\tilde{I}$: adjusted image, ${\mathrm{\Omega }}_{...}^{...}$: image region, $k$: frame number). * pause after ROI reselection

Fig. 4

Reliability metrics of the ROI selection process. a Number of segments (NoS) per patient in which single ROIs were absent. b NoS in which the ROI detector had to be re-executed. c NoS in which the standard deviation of the mean ROI intensity exceeded 50 (see “Implementation and framework” section). Each boxplot depicts 41 patient-related values

Fig. 5

Selected ROIs for six different patients. The first two columns show the ROIs (only contour) for the RGB and NIR image at the beginning of the recording, the last two columns at a later point. If there was minor or no movement, the results in column 1 and 2 are similar to those in 3 and 4. Please note that in case the patient was identifiable, the eye section in the depicted images was blurred

Fig. 6

Results of cbPPG measures when using the proposed method. a Heart rate detection rate for the red, green, blue and near-infrared channel. b Signal-to-noise ratio (SNR)

Fig. 7

Heart rate detection rate for the green channel in comparison to channel combinations. The combinations are determined assuming that always the correct heart rate (if available) can be selected between the two channels. Each boxplot depicts 41 patient-related values. The outcome of the statistical tests is shown above the boxes (***$p<0.001$)

Fig. 8

Comparison of the proposed method to a GrabCut-based approach. Three examples (RGB video) are depicted in the state of the initial ROI detection. The first column shows the result of the skin classifier. Similar as in our method, it was used as initialization for GrabCut although morphological closing was performed beforehand (see [39]). The last two columns show the final ROIs (only contour) in which the red arrows highlight the lack of performance of GrabCut. Please note that in case the patient was identifiable, the eye section in the depicted images was blurred. Due to eyebrows, eyelashes, and shadowing effects, the region around the eyes usually appears darker than the surrounding area

Fig. 9

Signal examples where artifacts occurred. Related signal segments for the R, G, B and NIR channel where the HR was detected correctly solely in the NIR signal. The ROIs were well-defined in both videos. Light variations in the ambient light caused artifacts to occur in the RGB video while the NIR video remained unaffected (cardiac pulse is visible). Please note that the strength of the pulsatile component usually does not exceed $\pm 15$ units for the set color depth