Multimodal retinal vessel segmentation from spectral-domain optical coherence tomography and fundus photography

Abstract

Segmenting retinal vessels in optic nerve head (ONH) centered spectral-domain optical coherence tomography (SD-OCT) volumes is particularly challenging due to the projected neural canal opening (NCO) and relatively low visibility in the ONH center. Color fundus photographs provide a relatively high vessel contrast in the region inside the NCO, but have not been previously used to aid the SD-OCT vessel segmentation process. Thus, in this paper, we present two approaches for the segmentation of retinal vessels in SD-OCT volumes that each take advantage of complimentary information from fundus photographs. In the first approach (referred to as the registered-fundus vessel segmentation approach), vessels are first segmented on the fundus photograph directly (using a k-NN pixel classifier) and this vessel segmentation result is mapped to the SD-OCT volume through the registration of the fundus photograph to the SD-OCT volume. In the second approach (referred to as the multimodal vessel segmentation approach), after fundus-to-SD-OCT registration, vessels are simultaneously segmented with a k -NN classifier using features from both modalities. Three-dimensional structural information from the intraretinal layers and neural canal opening obtained through graph-theoretic segmentation approaches of the SD-OCT volume are used in combination with Gaussian filter banks and Gabor wavelets to generate the features. The approach is trained on 15 and tested on 19 randomly chosen independent image pairs of SD-OCT volumes and fundus images from 34 subjects with glaucoma. Based on a receiver operating characteristic (ROC) curve analysis, the present registered-fundus and multimodal vessel segmentation approaches [area under the curve (AUC) of 0.85 and 0.89, respectively] both perform significantly better than the two previous OCT-based approaches (AUC of 0.78 and 0.83, p < 0.05). The multimodal approach overall performs significantly better than the other three approaches (p < 0.05).

Fig. 1

Illustration of NCO location and NCO false positives from Niemeijer's OCT-based vessel segmentation approach [18]. (a-b) Central slice of raw SD-OCT volume and vessel-oriented OCT projection image (Section II-B1) obtained from the SD-OCT volume with the highlight of NCO location (yellow arrows) and retinal vessels (purple arrows). A green outline is used to emphasize that the SD-OCT image is three dimensional. (c) Vessel segmentation from Niemeijer's OCT-based vessel segmentation [18]. Note that the typical false positives near the NCO (red arrows).

Fig. 2

Illustration of the high visibility of the NCO in SD-OCT image and that of vessels on fundus image. Left column: vessel-oriented OCT projection image (bottom) (Section II-B1) with the zoomed ONH center (upper). Note that the high NCO contrast as indicated by the yellow arrows. Right column: fundus photograph (bottom) with the zoomed ONH center (upper). Note that the high vessel contrast as indicated by the purple arrows.

Fig. 3

Overview of registered-fundus and multimodal vessel segmentation, where the dashed-blue-line and light-gray-background blocks indicate the registered-fundus vessel segmentation and the dashed-red-line and light-gray-background blocks indicate the multimodal vessel segmentation.

Fig. 4

Example fundus photograph and its vessel segmentation. (a) Original color fundus photograph. (b) Segmented vesselness map of the original fundus photograph.

Fig. 5

Illustration of vessel-oriented OCT projection image creation. (a) Central slice of the raw SD-OCT volume. (b) Central slice of the flattened SD-OCT volume with four segmented retinal surfaces indicated. (c) 3-D visualization of segmented surfaces. (d) Vessel-oriented OCT projection image obtained from the layer indicated by the yellow arrows in (b).

Fig. 6

Illustration of registered-fundus vessel segmentation. (a) Original color fundus image. (b) Segmented vesselness map of the original fundus image. (c) Vessel-oriented OCT projection image. (d) Preliminary OCT vessel segmentation [18]. (e) Cropped fundus-to-OCT registered vessel image.

Fig. 7

Illustration of NCO segmentation. (a) Central slice of the raw SD-OCT volume. (b) Central slice of the flattened SD-OCT volume with three radially interpolated surfaces. (c) NCO-aimed OCT projection image obtained from the layer indicated by the yellow arrows in b. (d) NCO (outer boundary) and optic cup (inner boundary) segmentation overlapping with the NCO-aimed projection image.

Fig. 8

Illustration of multimodal retinal vessel segmentation. (a) Registered fundus to OCT image. (b) Vessel-oriented OCT projection image. (c) A schematic illustration of the Gabor wavelet responses and the NCO-based templates oriented at 20 degrees. Blue arrow = NCO contour. Purple arrows = template pair centered on the NCO boundary. (d) Vessel segmentation from the multimodal approach.

Fig. 9

ROC curves of four different vessel segmentation approaches for all the 19 test eyes. ROC curves in the region (a) inside the NCO, (b) outside the NCO, and (c) entire region.

Fig. 10

Example comparison of the four different vessel segmentation algorithms. (a) Cropped fundus registered image. (b) Vessel-oriented OCT projection image. (c-f) Vessel segmentation from Niemeijer's previous OCT [18], our previous unimodal OCT [22], registered-fundus, and multimodal approach respectively. The red arrows indicate the false positives or vessel breaks due to the presence of the optic disc/NCO boundary.

Fig. 11

Example comparison of the four different vessel segmentation algorithms. (a) Cropped fundus registered image. (b) Vessel-oriented OCT projection image. (c-f) Vessel segmentation from Niemeijer's previous OCT [18], our previous unimodal OCT [22], registered-fundus, and multimodal approach respectively. The red arrows indicate the false positives or vessel breaks due to the presence of the optic disc/NCO boundary. The green arrow indicates the false positive from the choroidal vessels.

Fig. 12

Example “worst-case” performance of multimodal approach. (a) Original color fundus image. (b) Registered fundus image. (c) Vessel-oriented OCT projection image. (d) Multimodal vessel segmentation result. Note that the motion artifacts within the SD-OCT volume as illustrated with yellow arrows made registration difficult and correspondingly resulted in a relatively poor segmentation result compared to the other results in the dataset. However, such a segmentation result would still be expected to be usable in many applications.