Spectrally encoded fiber-based structured lighting probe for intraoperative 3D imaging

Abstract

Three dimensional quantification of organ shape and structure during minimally invasive surgery (MIS) could enhance precision by allowing the registration of multi-modal or pre-operative image data (US/MRI/CT) with the live optical image. Structured illumination is one technique to obtain 3D information through the projection of a known pattern onto the tissue, although currently these systems tend to be used only for macroscopic imaging or open procedures rather than in endoscopy. To account for occlusions, where a projected feature may be hidden from view and/or confused with a neighboring point, a flexible multispectral structured illumination probe has been developed that labels each projected point with a specific wavelength using a supercontinuum laser. When imaged by a standard endoscope camera they can then be segmented using their RGB values, and their 3D coordinates calculated after camera calibration. The probe itself is sufficiently small (1.7 mm diameter) to allow it to be used in the biopsy channel of commonly used medical endoscopes. Surgical robots could therefore also employ this technology to solve navigation and visualization problems in MIS, and help to develop advanced surgical procedures such as natural orifice translumenal endoscopic surgery.

Keywords: (110.6880) Three-dimensional image acquisition; (170.1610) Clinical applications; (170.2150) Endoscopic imaging; (170.3890) Medical optics instrumentation; (330.1710) Color, measurement.

Fig. 1

(a) Broadband laser light from the supercontinuum is dispersed by an SF-11 prism, which is then coupled into the fibers (50 μm core) at the array end of the probe. The projected pattern is a magnified image of the end face of the bundle formed by the projection lens. (b) Emission spectrum of the supercontinuum laser source, with the wavelength range used by the probe indicated between the dashed lines.

Fig. 2

Spot segmentation and 3D calibration. (a) Cartoon image showing three projected spots, having different RGB values. (b) Each RGB triplet is converted to xy coordinates. A line projected through these coordinates from a reference white spot intersects the spectrum locus (dashed) at the dominant wavelength of the pixel. (c) RGB pixels are replaced by the calculated wavelength to form a greyscale ‘λ-space’ image which can be thresholded to find the centroids of spots of a particular wavelength. (d) Epipolar geometry showing different positions of a calibration object (c₁-c₃) and triangulation of points using spot centroids.

Fig. 3

(a) RGB image of pattern recorded by camera. (b) λ-space image with centroids of spots. (c) Plot showing spot wavelength as calculated by the segmentation algorithm against the wavelength measured using a spectrometer. The transmission response of the camera’s filters (normalized to 1) is overlaid and the identity line is shown in black. The error bars indicate ± 1 standard deviation.

Fig. 4

Calculated wavelength of a set of spots projected onto surfaces of different colors. (a) Blue and red card. (b) Ex vivo tissue: porcine intestine (inset, top left) and lamb kidney (inset, bottom right). The error bars represent ± 1 standard deviation.

Fig. 5

Three dimensional calibration and validation. (a) Origin and propagation direction of projected spots with respect to the camera (origin) as determined during calibration routine. (b) Planar object. (c) Cylindrical object, diameter = 81 mm. (d) Cross-section of cylindrical object with least-squares fit.

Fig. 6

Three-dimensional reconstruction of ex vivo tissue. (a) Porcine liver, ‘step’. (b) Ovine kidney, convex curve. (c) Porcine liver, convex curve. (d) Porcine tissue, border between fatty tissue and liver. (e) Porcine liver, ‘valley’.