Generalist Vision Foundation Models for Medical Imaging: A Case Study of Segment Anything Model on Zero-Shot Medical Segmentation

Abstract

Medical image analysis plays an important role in clinical diagnosis. In this paper, we examine the recent Segment Anything Model (SAM) on medical images, and report both quantitative and qualitative zero-shot segmentation results on nine medical image segmentation benchmarks, covering various imaging modalities, such as optical coherence tomography (OCT), magnetic resonance imaging (MRI), and computed tomography (CT), as well as different applications including dermatology, ophthalmology, and radiology. Those benchmarks are representative and commonly used in model development. Our experimental results indicate that while SAM presents remarkable segmentation performance on images from the general domain, its zero-shot segmentation ability remains restricted for out-of-distribution images, e.g., medical images. In addition, SAM exhibits inconsistent zero-shot segmentation performance across different unseen medical domains. For certain structured targets, e.g., blood vessels, the zero-shot segmentation of SAM completely failed. In contrast, a simple fine-tuning of it with a small amount of data could lead to remarkable improvement of the segmentation quality, showing the great potential and feasibility of using fine-tuned SAM to achieve accurate medical image segmentation for a precision diagnostics. Our study indicates the versatility of generalist vision foundation models on medical imaging, and their great potential to achieve desired performance through fine-turning and eventually address the challenges associated with accessing large and diverse medical datasets in support of clinical diagnostics.

Keywords: Segment Anything Model (SAM); deep Learning; foundation models; large AI models; medical image segmentation; zero-shot segmentation.

Figure 1

Successful segmentation examples of SAM. Eight distinct modalities labeled with (A–H) are included, corresponding to dermoscope, fundus, CT, MRI, RGB endoscope, X-ray, endoscopic OCT, and ophthalmic OCT. Each set of images comprises four images, containing two pairs of SAM segmentation versus corresponding ground truth (GT).

Figure 2

Failure segmentation examples of SAM. Eight distinct modalities labeled with (A–H) correspond to dermoscope, fundus, CT, MRI, RGB endoscope, X-ray, endoscopic OCT, and ophthalmic OCT. Each set of images consists of two paired SAM segmentation and ground truth (GT).

Figure 3

A failure sample of SAM on segmenting retinal vessels. The first row from left to right is: the initial input image, ground truth mask, and the input image superimposed with the ground truth mask. The second row from left to right shows three SAM segmented images with the score of 1.007, 0.993, and 0.673, respectively.

Figure 4

Segmentation samples of SAM fine-tuned on retinal vessels. Each row from left to right is the initial input image, ground truth mask and prediction of fine-tuned SAM. The column from top to bottom shows retinal images from four different datasets.