MF-AV-Net: an open-source deep learning network with multimodal fusion options for artery-vein segmentation in OCT angiography

Abstract

This study is to demonstrate the effect of multimodal fusion on the performance of deep learning artery-vein (AV) segmentation in optical coherence tomography (OCT) and OCT angiography (OCTA); and to explore OCT/OCTA characteristics used in the deep learning AV segmentation. We quantitatively evaluated multimodal architectures with early and late OCT-OCTA fusions, compared to the unimodal architectures with OCT-only and OCTA-only inputs. The OCTA-only architecture, early OCT-OCTA fusion architecture, and late OCT-OCTA fusion architecture yielded competitive performances. For the 6 mm×6 mm and 3 mm×3 mm datasets, the late fusion architecture achieved an overall accuracy of 96.02% and 94.00%, slightly better than the OCTA-only architecture which achieved an overall accuracy of 95.76% and 93.79%. 6 mm×6 mm OCTA images show AV information at pre-capillary level structure, while 3 mm×3 mm OCTA images reveal AV information at capillary level detail. In order to interpret the deep learning performance, saliency maps were produced to identify OCT/OCTA image characteristics for AV segmentation. Comparative OCT and OCTA saliency maps support the capillary-free zone as one of the possible features for AV segmentation in OCTA. The deep learning network MF-AV-Net used in this study is available on GitHub for open access.

Fig. 1.

Representative OCTA images and ground truth AV maps. (A) 6 mm×6 mm OCTA image. (B) 3 mm×3 mm OCTA image with green dashed box overlayed on the 6 mm×6 mm OCTA image. (C) 6 mm×6 mm ground truth AV map. (D) 3 mm×3 mm ground truth with green dashed box overlayed on the 6 mm×6 mm ground truth AV map. (E) 3 mm×3 mm OCTA image. (F) 3 mm×3 mm ground truth AV map. (G) enlarged orange dashed box of OCTA image shown in E (H) enlarged orange dashed box of ground truth shown in F. Green points in H indicate some of the last arterial and venous branch points.

Fig. 2.

An illustration of the MF-AV-Net architectures.

Fig. 3.

Comparative illustration of the AV segmentation performance achieved by different architectures trained on the 6 mm×6 mm dataset (each row shows a different sample). Yellow windows indicate some areas in which late fusion architecture has correctly segmented arteries or veins comparing to OCTA-only architecture or early fusion architecture. White windows indicate some areas in which early fusion architecture has failed to successfully segment arteries or veins whereas OCTA-only and late fusion architectures have successfully segmented.

Fig. 4.

Illustration of the ground truths with the lower level of vascular details beside OCT images, original ground truths and the OCT-only architecture predicted AV-maps.

Fig. 5.

Comparative illustration of the AV segmentation performance achieved by different architectures trained on the 3 mm×3 mm dataset (each row shows a different sample).

Fig. 6.

Comparative illustration of saliency maps for AV classification of individual pixels (A1) Representative AV-map predicted by OCTA-only architecture with three individual pixels B (yellow), C (blue), and D (gray) on the artery, vein, and background, respectively. (A2) Corresponding en face OCTA image. Yellow arrows indicate the capillary-free zone around the artery. (A3) Corresponding AV-map predicted by OCT-only architecture with three individual pixels B (yellow), C (blue), and D (gray) on the artery, vein, and background, respectively. (A4) Corresponding en face OCT image. (B1), (C1), and (D1) are the saliency maps for the pixels B, C, and D in A1, respectively. (B2), (C2), and (D2) are the saliency maps in B1, C1, and D1 overlayed on the OCTA image A2. (B3), (C3), and (D3) are the saliency maps for the pixels B, C, and D in A3, respectively. (B4), (C4), and (D4) are the saliency maps in B3, C3, and D3 overlayed on the OCT image A4.

Fig. 7.

Comparative illustration of saliency maps for AV segmentation of segments in different classes (A1) Representative AV-map predicted by OCTA-only architecture with three segments B (yellow), C (blue), and D (gray) of the artery, vein, and background, respectively. (A2) Corresponding en face OCTA image. (A3) Corresponding AV-map predicted by OCT-only architecture with three segments B (yellow), C (blue), and D (gray) of the artery, vein, and background, respectively. (A4) Corresponding en face OCT image. (B1), (C1), and (D1) are the saliency maps for segments the B, C, and D in A1, respectively. (B2), (C2), and (D2) are the saliency maps in B1, C1, and D1 overlayed on the OCTA image A2. (B3), (C3), and (D3) are the saliency maps for the segments B, C, and D in A3, respectively. (B4), (C4), and (D4) are the saliency maps described in B3, C3, and D3 overlayed on the OCT image A4.

Fig. 8.

Comparative illustration of saliency maps for AV segmentation of segments in different classes (A1) Representative AV-map predicted by early fusion architecture with three segments B (yellow), C (blue), and D (gray) of the artery, vein, and background, respectively. (A2) Corresponding en face OCTA image. (A3) Corresponding en face OCT image. (A4) Corresponding AV-map predicted by late fusion architecture with three segments B (yellow), C (blue), and D (gray) of the artery, vein, and background, respectively. (B1), (C1), and (D1) are the saliency maps of early fusion architecture for segments B, C, and D in A1 overlayed on the OCT image, respectively. (B2), (C2), and (D2) are the saliency maps of early fusion architecture for segments B, C, and D in A1 overlayed on the OCTA image, respectively. (B3), (C3), and (D3) are the saliency maps of late fusion architecture for segments B, C, and D in A4 overlayed on the OCT image, respectively. (B4), (C4), and (D4) are the saliency maps of late fusion architecture for segments B, C, and D in A4 overlayed on the OCTA image, respectively.

Fig. 9.

Comparative illustration of saliency maps for 3 mm×3 mm dataset (A1) Representative AV-map predicted by OCT-only architecture with two segments B (yellow) and C (blue) of the artery and vein, respectively. (A2) Corresponding AV-map predicted by OCTA-only architecture (A3) Corresponding AV-map predicted by early fusion architecture (A4) Corresponding en face OCTA image. (A5) Corresponding en face OCT image. (A6) Corresponding AV-map predicted by late fusion architecture (B1) and (C1) are the saliency maps of OCT-only architecture for segments B and C in A1 overlayed on the OCT image, respectively. (B2) and (C2) are the saliency maps of OCTA-only architecture for segments B and C in A2 overlayed on the OCTA image, respectively. (B3), (C3), (B4), and (C4) are the saliency maps of early fusion architecture for segments B and C in A3 overlayed on the OCT and OCTA images, respectively. (B5), (C5), (B6), and (C6) are the saliency maps of late fusion architecture for segments B and C in A6 overlayed on the OCT and OCTA images, respectively.