Multitask Learning for Activity Detection in Neovascular Age-Related Macular Degeneration

Abstract

Purpose: The purpose of this study was to provide a comparison of performance and explainability of a multitask convolutional deep neuronal network to single-task networks for activity detection in neovascular age-related macular degeneration (nAMD).

Methods: From 70 patients (46 women and 24 men) who attended the University Eye Hospital Tübingen, 3762 optical coherence tomography B-scans (right eye = 2011 and left eye = 1751) were acquired with Heidelberg Spectralis, Heidelberg, Germany. B-scans were graded by a retina specialist and an ophthalmology resident, and then used to develop a multitask deep learning model to predict disease activity in neovascular age-related macular degeneration along with the presence of sub- and intraretinal fluid. We used performance metrics for comparison to single-task networks and visualized the deep neural network (DNN)-based decision with t-distributed stochastic neighbor embedding and clinically validated saliency mapping techniques.

Results: The multitask model surpassed single-task networks in accuracy for activity detection (94.2% vs. 91.2%). The area under the curve of the receiver operating curve was 0.984 for the multitask model versus 0.974 for the single-task model. Furthermore, compared to single-task networks, visualizations via t-distributed stochastic neighbor embedding and saliency maps highlighted that multitask networks' decisions for activity detection in neovascular age-related macular degeneration were highly consistent with the presence of both sub- and intraretinal fluid.

Conclusions: Multitask learning increases the performance of neuronal networks for predicting disease activity, while providing clinicians with an easily accessible decision control, which resembles human reasoning.

Translational relevance: By improving nAMD activity detection performance and transparency of automated decisions, multitask DNNs can support the translation of machine learning research into clinical decision support systems for nAMD activity detection.

Figure 1.

Exemplary retinal images (B-scans) with neovascular age-related macular degeneration (nAMD). (A) No nAMD activity. (B) nAMD activity due to subretinal fluid (SRF). (C) nAMD activity due to intraretinal fluid (IRF). (D) nAMD activity due to both SRF and IRF.

Figure 2.

A deep neural network for simultaneous detection of subretinal and intraretinal fluid as well as the nAMD activity from OCT B-scans. Given a B-scan, convolutional stack of the InceptionV3 architecture extracts 2048 feature maps. These are average and max pooled, and fed into a fully connected (dense) layer with 1024 units for shared representation. Then, task-specific heads specialize into individual tasks and single units with sigmoid function achieve binary classification based on 256 task-specific features.

Figure 3.

Performance curves of the selected models on the test images. Area under the curve (AUC) values given for models also summarize the overall performance into one number (higher is better). (A) Receiver Operating Characteristics (ROC) curves. (B) Precision-recall curves.

Figure 4.

Visualization of data via t-SNE of ensemble-based representations. Only the test data are shown. (A) Low dimensional embedding of images based on the 1024-dimensional features from the pre-penultimate layers of single-task networks. Colored with respect to the task-specific labels. (B) Same as in A but with respect to 1024 features from the shared representation layer of multitask networks. (C) Same map as in B but colored with respect to correct and wrong predictions. (D) Same map as in B but colored with respect to uncertainty minimum-maximum normalized to [0, 1].

Figure 5.

Layer-wise visualization of test data via t-SNE. Starting just before the first inception module (A) and reading out feature representations yielded by every other module (B-F) along with the last inception module (G), the shared representation layer (H) and the nAMD activity detection head's penultimate layer (I), we performed t-SNE with the aforementioned settings. Useful representations emerged toward the end of convolutional stack and the task-specific representation allowed the best separation of nAMD active cases from those inactive.

Figure 6.

Exemplary saliency maps for four optical coherence tomography (OCT) images. The first column displays the OCT B-scan with the corresponding labeling of a retinal specialist. Second to fourth columns show saliency maps and the network's confidence for active nAMD (yellow), subretinal fluid (SRF; cyan), and intraretinal fluid (IRF; magenta). Note, that saliency maps are only shown in case of confidence >0.5.

Figure 7.

Exemplary saliency maps as in Figure 6 but results were obtained from single-task models.