Spectral imaging enables contrast agent-free real-time ischemia monitoring in laparoscopic surgery

Abstract

Laparoscopic surgery has evolved as a key technique for cancer diagnosis and therapy. While characterization of the tissue perfusion is crucial in various procedures, such as partial nephrectomy, doing so by means of visual inspection remains highly challenging. We developed a laparoscopic real-time multispectral imaging system featuring a compact and lightweight multispectral camera and the possibility to complement the conventional surgical view of the patient with functional information at a video rate of 25 Hz. To enable contrast agent-free ischemia monitoring during laparoscopic partial nephrectomy, we phrase the problem of ischemia detection as an out-of-distribution detection problem that does not rely on data from any other patient and uses an ensemble of invertible neural networks at its core. An in-human trial demonstrates the feasibility of our approach and highlights the potential of spectral imaging combined with advanced deep learning-based analysis tools for fast, efficient, reliable, and safe functional laparoscopic imaging.

Fig. 1.. Our MSI-based approach enables continuous, contrast agent–free, real-time ischemia monitoring in laparoscopic surgery.

The clinical state of the art (brown) to verify successful ischemia induction via clamping of arteries is based on ICG imaging: Ischemic tissue is characterized by a lack of fluorescent signal, whereas perfused tissue fluoresces. In the case of unsuccessful clamping, the test can only be repeated after a washout period of about 30 min. Our approach (green) is based on noninvasive and contrast agent–free MSI at video rate. DL models trained on MSI video sequences before clamping are capable of detecting ischemic tissue areas as outliers in real time, as detailed in (Fig. 3).

Fig. 2.. Our MSI system for ischemia monitoring is comparable to current standard equipment in terms of size and weight.

(A) Proposed spectral imaging–based laparoscope (left) compared to standard RGB laparoscope (right). (B) Schematic representation of the system developed for spectral tissue analysis in laparoscopic surgery (see the “Multispectral imaging system” section). The dimensions of the laptop and the light source have been scaled down for visualization purposes. USB, universal serial bus.

Fig. 3.. We treat ischemia monitoring based on MSI as an OoD task, which only needs training data from one single patient.

Traditional DL methods (brown) require large amounts of patient data to train a model, while our method (green) only needs data from a single patient. Using an ensemble of INNs as a core component, our algorithm computes the likelihood of ischemia based on a short (several seconds) video sequence acquired at the beginning of each surgery. An important feature of our approach is that the entire training and inference process can be performed during a surgical procedure.

Fig. 4.. High tissue heterogeneity across subjects motivated our personalized approach.

Both the (A) PCA and (B) the mixed-model analysis demonstrate the high interpatient variability of spectral tissue measurements. (A) The fact that different tissue states cluster within a subject but do not form a uniform cluster across subjects motivated us to phrase the challenge of ischemia detection as an OoD problem. Solid markers show the cluster centers, transparent markers show the raw data points, and the axis labels denote the explained variance of the corresponding PC. (B) Explained variance for the 16 bands of the MSI camera, depicted in Fig. 6.

Fig. 5.. Our approach is capable of discriminating between ischemic and perfused kidney states.

We calculated the ischemia index for every frame in video sequences of perfused and ischemic kidney separately for each patient (see the “Algorithm for live ischemia monitoring” section) and generated corresponding dot and box plots. The boxes show the interquartile range with the median (solid line) and mean (dashed line), while the whiskers extend up to 1.5 the interquartile range. Min-max normalization was performed for clarity of presentation.

Fig. 6.. Optical properties of our multispectral system.

(A) ℓ₁-Normalized transmission spectra of the laparoscopes, bandpass filter, and C-Mount adapter are shown in the left axis. Extinction coefficients of HbO₂ and Hb are shown in the right axis. The bandpass filter mainly filters light in the low-wavelength region where blood absorbs the most (below ≈600 nm), thus making the spectral power distribution that reaches the camera detector more uniform across wavelengths. (B) Representation of the 4 × 4 mosaic pattern of the multispectral camera sensor. Each colored square represents a different filter; these filters form a 4 × 4 pattern that extends over the whole image. (C) Filter responses of the multispectral camera bands. Some bands, such as 5 to 12, show two peaks in the spectral response, which are referred to as “second-order” peaks.

Fig. 7.. Comparison of a representative patient with patient 7 with respect to the algorithm input.

(A) Example RGB images were selected from the perfused₂ sequence to illustrate the unusual images acquired of the kidney of patient 7. (B) The reflectances of the representative patient (patient 3) differ clearly between perfused and ischemic tissue for the vast majority of camera bands, while no clear separation can be observed in patient 7. The boxes show the interquartile range with the median (solid line), while the whiskers extend up to 1.5 the interquartile range. (C) A KDE performed on the data resulting from a patient-specific PCA provides further explanation for why our method falsely detected the ischemic kidney data of patient 7 as inlier. The axis labels denote the explained variance of the corresponding PC.

Fig. 8.. MSI recording procedure in the OR.

We recorded two MSI sequences [perfused₁ (for training) and perfused₂ (for testing)] before the clamp was applied to the renal artery and one sequence after applying the clamp (for testing). The laparoscope was removed and reinserted in the patient’s abdominal cavity before recording the testing sequence for perfused kidney to obtain the same conditions as for the ischemic kidney.