Registration of fluorescein angiography and optical coherence tomography images of curved retina via scanning laser ophthalmoscopy photographs

Abstract

Accurate and automatic registration of multimodal retinal images such as fluorescein angiography (FA) and optical coherence tomography (OCT) enables utilization of supplementary information. FA is a gold standard imaging modality that depicts neurovascular structure of retina and is used for diagnosing neurovascular-related diseases such as diabetic retinopathy (DR). Unlike FA, OCT is non-invasive retinal imaging modality that provides cross-sectional data of retina. Due to differences in contrast, resolution and brightness of multimodal retinal images, the images resulted from vessel extraction of image pairs are not exactly the same. Also, prevalent feature detection, extraction and matching schemes do not result in perfect matches. In addition, the relationships between retinal image pairs are usually modeled by affine transformation, which cannot generate accurate alignments due to the non-planar retina surface. In this paper, a precise registration scheme is proposed to align FA and OCT images via scanning laser ophthalmoscopy (SLO) photographs as intermediate images. For this purpose, first a retinal vessel segmentation is applied to extract main blood vessels from the FA and SLO images. Next, a novel global registration is proposed based on the Gaussian model for curved surface of retina. For doing so, first a global rigid transformation is applied to FA vessel-map image using a new feature-based method to align it with SLO vessel-map photograph, in a way that outlier matched features resulted from not-perfect vessel segmentation are completely eliminated. After that, the transformed image is globally registered again considering Gaussian model for curved surface of retina to improve the precision of the previous step. Eventually a local non-rigid transformation is exploited to register two images perfectly. The experimental results indicate the presented scheme is more precise compared to other registration methods.

Fig. 1.

Block diagram of overall process of registration.

Fig. 2.

(a) SLO image. (b) the OCT B-scan corresponds to the yellow line in the SLO image.

Fig. 3.

Our data set includes 36 pairs of FA and SLO images. Odd columns illustrate FA images of different field of view (30 and 55 degrees), even columns depict the SLO images corresponding to the left side column.

Fig. 4.

Vessel Extraction. (a) FA image. (b) vessels extracted from FA. (c) SLO image. (d) vessels extracted from SLO.

Fig. 5.

(a) Fast corner detector and Bresenham circle. (b) FAST corners in FA vessel map (green dots). (c) FAST corners in SLO vessel map (green dots).

Fig. 6.

SURF features (a) 150 strongest SURF features on SLO vessel map (green circles). (b) 150 strongest SURF features on FA vessel map (green circles).

Fig. 7.

HOG visualization. (a) SLO vessel map. (b) HOG visualization of (a). (c) FA vessel map. (d) HOG visualization of (c).

Fig. 8.

Final MSAC step refines the previously-matched features. (a) matched features after joining separately refined SURF and HOG matches (still contains few outliers). (b) obtaining perfect matches after final MSAC step.

Fig. 9.

Two different camera viewpoints of the retina.

Fig. 10.

Qualitative evaluation of three global registration schemes (Yellow rectangles highlight the differences). (a) global registration using first step(false color). (b) global registration exploiting quadratic model for retina surface(false color). (c) global registration using Gaussian model for retina surface(false color). In false color method, first and second images are shown in green and magenta, and the overlapped regions are depicted in white.

Fig. 11.

Vessel maps and images after different steps of the proposed registration. (a) vessel maps after first step of global registration (false color). (b) vessel maps after applying Gaussian model to the first step of global registration (false color). (c) vessel maps after local registration step (false color) (arrows point out the differences in registration accuracy and yellow circles show the key points). (d-f) images correspond to the vessel maps (checker board). (g-i) zoomed images correspond to the ROI specified via red rectangle in second row (checker board). (j-l) zoomed images correspond to the ROI specified via magenta rectangle in second row (checker board). (m-o) zoomed images correspond to the ROI specified via yellow rectangle in second row (checker board). In checker board method, image is composed of alternating rectangular regions from two images.

Fig. 12.

All data set images (zoomed ROI) after local registration step (checker board). Since registration of pair of images 36 is failed, this image is not shown.

Fig. 13.

The box-plot of RMS errors of global and local registration steps.

Fig. 14.

Qualitative evaluation of proposed method. (a) matching features using SURF on image case 3. (b) matching features using proposed method. (c) registration failure utilizing SURF features.(in color) (d) registration using proposed method. (e) matching features using FAST-HOG on image case 8. (f) matching features using proposed method. (g) registration failure utilizing FAST-HOG. (in color) (h) registration using proposed method.

Fig. 15.

MAs in B-scans of diabetic retinopathy. (a) MA in FA image. (b) corresponding B-scan of the yellow line in (a). (c) zoomed complete ring sign in image (b). (d) MA in an FA image. (e) corresponding B-scan of the yellow line in (d). (f) zoomed incomplete ring sign in image (e). (g) MA in an FA image.(h) corresponding B-scan of the yellow line in (g). (i) MA with no sign.

Fig. 16.

leakage in B-scan of diabetic retinopathy (a) FA image with leakage. Red arrow and circle show part of leakage. (b) corresponding B-scan of the yellow line in (a). yellow arrows indicate the blob shaped (cystoid) area corresponds to red circle in (a). (c) zoomed image of the blob shaped (cystoid) area in (b).