Decoupling channel count from field of view and spatial resolution in single-sensor imaging systems for fluorescence image-guided surgery

Abstract

Significance: Near-infrared fluorescence image-guided surgery is often thought of as a spectral imaging problem where the channel count is the critical parameter, but it should also be thought of as a multiscale imaging problem where the field of view and spatial resolution are similarly important.

Aim: Conventional imaging systems based on division-of-focal-plane architectures suffer from a strict relationship between the channel count on one hand and the field of view and spatial resolution on the other, but bioinspired imaging systems that combine stacked photodiode image sensors and long-pass/short-pass filter arrays offer a weaker tradeoff.

Approach: In this paper, we explore how the relevant changes to the image sensor and associated image processing routines affect image fidelity during image-guided surgeries for tumor removal in an animal model of breast cancer and nodal mapping in women with breast cancer.

Results: We demonstrate that a transition from a conventional imaging system to a bioinspired one, along with optimization of the image processing routines, yields improvements in multiple measures of spectral and textural rendition relevant to surgical decision-making.

Conclusions: These results call for a critical examination of the devices and algorithms that underpin image-guided surgery to ensure that surgeons receive high-quality guidance and patients receive high-quality outcomes as these technologies enter clinical practice.

Keywords: Image-guided cancer surgery; multiscale spectral imaging; pixelated optical filter; sentinel lymph node mapping; stacked photodiode image sensor; tumor detection.

Fig. 1

The structure and function of an RGB-IR camera and a bioinspired camera. (a) A red/green/blue/near-infrared camera (or RGB-IR camera) is constructed from a single layer of silicon photodiodes topped with an additional layer of four different absorptive filters. The silicon photodiodes exhibit a broadband response spanning the visible (VIS) and near-infrared (NIR), while the absorptive filters exhibit a relatively narrow transmission tuned to blue light, green light, red light, or NIR light. Since each photodiode absorbs all colors of light while each absorptive filter passes a different color of light, the camera can provide, in every $2\times 2\text{\hspace{0.17em}\hspace{0.17em}pixel}$ neighborhood, three observations in the VIS spectrum and one observation in the NIR spectrum. (b) The quantum efficiencies presumed for the RGB-IR camera that was used in this study, illustrating the tetrachromatic (“four-color”) vision offered. The curves have been adopted from Ref. , consistent with the methods in Ref. . (c) A bioinspired long-pass/short-pass camera (or bioinspired camera) makes two substitutions compared with an RGB-IR camera: First, it replaces the single layer of silicon photodiodes with three layers of such photodiodes (each sensitive to shorter wavelength, longer wavelengths, or intermediate wavelengths), and second, it replaces the layer of four different absorptive filters with a layer of two different interferences filters (each serving as a short-pass filter tuned for VIS light or a long-pass filter tuned for NIR light). Since each photodiode absorbs and each interference filter passes a different color of light, the camera can provide, in every $2\times 1\text{\hspace{0.17em}\hspace{0.17em}pixel}$ neighborhood, three observations in the VIS spectrum and three observations in the NIR spectrum. (d) The quantum efficiencies evaluated for the bioinspired camera that was used in this study, illustrating the hexachromatic (“six-color”) vision offered. The curves have been adopted from Ref. .

Fig. 2

The need for demosaicing routines. While blue light, green light, red light, and near-infrared light are reflected from every point in a scene (as illustrated in the unmosaiced image in the top row with all pixels known), these colors of light are not observed at every pixel in a single-sensor imaging system (as illustrated in the mosaiced images in the middle row with some pixels known and others unknown). As a result, demosaicing routines must generate demosaiced images (like those in the bottom row) that can be displayed to an end-user or consumed by a computer. Yet even though the loss of spatial information during the imaging process can be mitigated by the demosaicing routine, the loss of spectral information cannot. Consequently, an RGB-IR camera will only be able to provide three VIS channels and one NIR channel, even after the demosaicing routine has been applied, and a bioinspired camera will still be able to provide three VIS channels and three NIR channels.

Fig. 3

The experimental setup and unmosaiced images for the preclinical dataset along with quantile functions for the perceptual metrics. (a) A mouse with a breast tumor was administered IRDye 800CW Maleimide so that the breast tumor could be identified with near-infrared fluorescence. Raw images were collected using a stacked photodiode image sensor equipped with no filter array, with illumination provided by either a VIS source alone or an NIR source alone. Unmosaiced images were then constructed by stacking the images taken under white light illumination alone and the images taken under near-infrared illumination alone. (In this illustration, the light sources and the image sensor have been positioned for conceptual clarity; during the actual experiment, though, the light sources and the image sensor were all positioned above the mouse.) (b) Three unmosaiced images were included in the dataset: two images showing the tumor before surgery through the skin and one image showing the tumor during surgery through an incision. Each image consisted of one frame and $399\times 144\text{\hspace{0.17em}\hspace{0.17em}pixels}$. (c) Quantile plots for visible color difference (VIS $\mathrm{\Delta }E$), visible dissimilarity index (VIS DSSIM), near-infrared color difference (NIR $\mathrm{\Delta }E$), and near-infrared dissimilarity index (NIR DSSIM) were generated for an RGB-IR imaging system with bilinear demosaicing and a bioinspired imaging system with bilinear demosaicing. Each point indicates the proportion of pixels that have been assigned a value for a metric that is less than or equal to a given value for that metric. NIR $\mathrm{\Delta }E$ could not be computed for the RGB-IR imaging system since only one of three total near-infrared channels could be captured by that camera.

Fig. 4

The experimental setup and unmosaiced images for the clinical dataset along with quantile functions for the perceptual metrics. (a) Seven women with breast cancer were administered indocyanine green and methylene blue so that any sentinel lymph nodes could be identified with near-infrared fluorescence. Raw images were collected using a stacked photodiode image sensor equipped with a bioinspired filter array, with illumination provided by both a VIS light source and an NIR source. Unmosaiced images were then constructed by taking spatial averages within each channel across $2\times 2\text{\hspace{0.17em}\hspace{0.17em}pixel}$ blocks. (In this illustration, the light sources and the image sensor have been positioned for conceptual clarity; during the actual experiment, though, the visible light source was positioned above the patient, whereas the near-infrared light sources and the image sensor were all positioned to the side of the patient.) (b) One unmosaiced video was included in the dataset, showing the surgeon identifying and removing a sentinel lymph node before validating resection with a gamma probe. The video consisted of 384 frames and $640\times 368\text{\hspace{0.17em}\hspace{0.17em}pixels}$; the intensities in the visible frames and the near-infrared frames were scaled up by $1.5\times$ and $3.0\times$, respectively, for display here. (c, top row) Quantile plots for visible color difference (VIS $\mathrm{\Delta }E$), visible dissimilarity index (VIS DSSIM), near-infrared color difference (NIR $\mathrm{\Delta }E$), and near-infrared dissimilarity index (NIR DSSIM) generated for an RGB-IR imaging system with bilinear demosaicing and a bioinspired imaging system with bilinear demosaicing. Each point indicates the proportion of pixels that have been assigned a value for a metric that is less than or equal to a given value for that metric. NIR $\mathrm{\Delta }E$ could not be computed for the RGB-IR imaging system since only one of three total near-infrared channels could be captured by that camera. (c, bottom row) The percent change in the metric that was observed at each quantile after switching the RGB-IR imaging system from bilinear demosaicing to one-dimensional bicubic polynomial demosaicing and after switching the bioinspired imaging system from bilinear demosaicing to two-dimensional bicubic polynomial demosaicing.