A multimodal visual-language foundation model for computational ophthalmology

Abstract

Early detection of eye diseases is vital for preventing vision loss. Existing ophthalmic artificial intelligence models focus on single modalities, overlooking multi-view information and struggling with rare diseases due to long-tail distributions. We propose EyeCLIP, a multimodal visual-language foundation model trained on 2.77 million ophthalmology images from 11 modalities with partial clinical text. Our novel pretraining strategy combines self-supervised reconstruction, multimodal image contrastive learning, and image-text contrastive learning to capture shared representations across modalities. EyeCLIP demonstrates robust performance across 14 benchmark datasets, excelling in disease classification, visual question answering, and cross-modal retrieval. It also exhibits strong few-shot and zero-shot capabilities, enabling accurate predictions in real-world, long-tail scenarios. EyeCLIP offers significant potential for detecting both ocular and systemic diseases, and bridging gaps in real-world clinical applications.

Fig. 1. Study diagram.

a Using an extensive multimodal database across nine provinces in China, we matched the multi-examination images from the same patient, and cleaned the medical reports using a keyword mapping dictionary containing medical terminology to generate hierarchical keyword text labels. b EyeCLIP was pretrained using self-supervised reconstruction, multi-examination contrastive learning, and hierarchical text-image contrastive learning to leverage real-world multi-examination clinical data fully. c Downstream multi-country datasets for EyeCLIP validation, including zero-shot, few-shot, and supervised finetuning scenarios. d Radar plot outlines the performance of EyeCLIP and baseline models across various downstream tasks. EyeCLIP significantly outperforms the baseline models across diverse tasks, including zero-shot classification, multimodal retrieval, visual question answering (VQA), and supervised systemic disease prediction.

Fig. 2. Zero-shot performance on downstream ocular diseases datasets.

a AUROC. b AUPR. Error bars represent 95% confidence intervals, and the centers correspond to computed values of each metric. EyeCLIP achieved significantly better zero-shot performance than other models for both AUROC and AUPR. AUROC = area under the receiver operator characteristic curve, AUPR = area under the precision-recall curve. EyeCLIP outperforms the second-best model FLAIR, a pretrained vision-language model for universal retinal fundus image understanding. Notably, FLAIR was pretrained on public datasets, with its performance evaluated through internal validation. In contrast, EyeCLIP, which was not trained on these public datasets, demonstrated its performance through external validation, highlighting its strong generalizability.

Fig. 3. Few-shot classification experiments.

We investigated the label efficiency of different pretrained models in a few-shot setting, varying the number of training labels per class (nc = 1, 2, 4, 8, 16) in the APTOS2019 (a), MESSIDOR2 (b), IDRID (c), GLAUCOMA FUNDUS (d), PAPILA (e), JSIEC (f), RETINA (g), OCTDL (h), and OCTID (i) dataset. For each nc, we sampled five different sets of training examples and trained a weakly supervised model. Boxes indicate quartile values, and whiskers extend to data points within 1.5× the interquartile range. EyeCLIP achieves significantly better performance (in terms of the mean AUROC of five runs) than other encoders for different sizes of training sets and across all datasets. AUROC = area under the receiver operator characteristic curve. AUPR results can be found in Supplementary Fig. 1.

Fig. 4. Performance of EyeCLIP across ocular, systemic, and rare disease prediction tasks.

a Supervised full-data finetuning on ocular disease tasks. EyeCLIP is on par with the 2^nd best model RETFound on APTOS2019, MESSIDOR2, OCTID (P > 0.05), and surpasses all models on the other eight datasets. b Supervised full-data finetuning on systemic disease prediction. EyeCLIP surpasses all other models. (P < 0.05). c Few-shot finetuning on rare disease classification. EyeCLIP surpasses all other models. (P < 0.05). Boxes indicate quartile values, and whiskers extend to data points within 1.5× the interquartile range. Detailed statistics can be found in Supplementary Tables 4-5. AUROC = area under the receiver operator characteristic curve, AUPR area under the precision-recall curve.

Fig. 5. Zero-shot multimodal retrieval performance.

a Model comparison on two datasets with image-text pairs, AngioReport and Retina Image Bank. Similarity in the embedding space was computed between the query image and all text samples in the database. The top-K most similar texts were retrieved. We report Recall@K for K ∈ {1, 5, 10} and the mean recall, which averages over K. We compared different models in text-to-image (1^st column), image-to-image (2^nd column) and image-to-text (3^rd column). EyeCLIP outperforms other baselines on all retrieval tasks. Error bars indicate 95% confidence intervals. b Schematic illustrates zero-shot cross-modal retrieval. c, d Examples of images in the top one retrieved result from the Retina Image Bank. More examples can be found in Supplementary Fig. 3.