Interactive Medical Image Segmentation Using Deep Learning With Image-Specific Fine Tuning

Abstract

Convolutional neural networks (CNNs) have achieved state-of-the-art performance for automatic medical image segmentation. However, they have not demonstrated sufficiently accurate and robust results for clinical use. In addition, they are limited by the lack of image-specific adaptation and the lack of generalizability to previously unseen object classes (a.k.a. zero-shot learning). To address these problems, we propose a novel deep learning-based interactive segmentation framework by incorporating CNNs into a bounding box and scribble-based segmentation pipeline. We propose image-specific fine tuning to make a CNN model adaptive to a specific test image, which can be either unsupervised (without additional user interactions) or supervised (with additional scribbles). We also propose a weighted loss function considering network and interaction-based uncertainty for the fine tuning. We applied this framework to two applications: 2-D segmentation of multiple organs from fetal magnetic resonance (MR) slices, where only two types of these organs were annotated for training and 3-D segmentation of brain tumor core (excluding edema) and whole brain tumor (including edema) from different MR sequences, where only the tumor core in one MR sequence was annotated for training. Experimental results show that: 1) our model is more robust to segment previously unseen objects than state-of-the-art CNNs; 2) image-specific fine tuning with the proposed weighted loss function significantly improves segmentation accuracy; and 3) our method leads to accurate results with fewer user interactions and less user time than traditional interactive segmentation methods.

Fig. 1.

The proposed Bounding box and Image-specific Fine-tuning-based Segmentation (BIFSeg). 2D images are shown as examples. During training, each instance is cropped with its bounding box, and the CNN is trained for binary segmentation. In the testing stage, image-specific fine-tuning with optional scribbles and a weighted loss function is used. Note that the object class (e.g. maternal kidneys) for testing may have not been present in the training set.

Fig. 2.

Our resolution-preserving networks with dilated convolution for 2D segmentation (a) and 3D segmentation (b). The numbers in dark blue boxes denote convolution kernel sizes and numbers of output channels, and the numbers on the top of these boxes denote dilation parameters.

Fig. 3.

An example of weight map for image-specific fine-tuning. The weight is 0 for pixels with high uncertainty (black), for scribbles (white), and 1 for the remaining pixels (gray).

Fig. 4.

Evolution of cross entropy loss on training and validation data during the training stage of different networks for 2D fetal MRI segmentation. Fetal lungs and maternal kidneys were not present in the training set.

Fig. 5.

Visual comparison of initial segmentation of multiple organs from fetal MRI with a bounding box. All the methods use the same bounding box for each test instance. Note that fetal lungs and maternal kidneys are previously unseen objects but P-Net works well on them.

Fig. 6.

Visual comparison of P-Net and three unsupervised refinement methods for fetal MRI segmentation. The foreground probability is visualized by heatmap.

Fig. 7.

Visual comparison of P-Net and three supervised refinement methods for fetal MRI segmentation. The same initial segmentation and scribbles are used for P-Net + CRF, BIFSeg(-w) and BIFSeg.

Fig. 8.

Unsupervised and supervised fine-tuning results of BIFSeg for the same instance of previously unseen maternal kidneys. (a) shows the user-provided bounding box. (b) is the initial output of P-Net and (e) is the result of unsupervised fine-tuning. (c) and (d) show user-provided scribbles for supervised fine-tuning, and (f) and (g) are their corresponding results.

Fig. 9.

User time and Dice score of different interactive methods for fetal MRI segmentation. denotes previously unseen objects for BIFSeg.

Fig. 10.

Visual comparison of initial segmentation of brain tumors from a 3D bounding box. The whole tumor in FLAIR is previously unseen in the training set. All these methods use the same bounding box for each test image.

Fig. 11.

Visual comparison of PC-Net and unsupervised refinement methods without additional scribbles for 3D brain tumor segmentation. The same initial segmentation obtained by PC-Net is used by different refinement methods. (a) Tumor core in T1c (previously seen). (b) Whole tumor in FLAIR (previously unseen).

Fig. 12.

Visual comparison of PC-Net and three supervised refinement methods with scribbles for 3D brain tumor segmentation. The refinement methods use the same initial segmentation and set of scribbles. (a) Tumor core in T1c (previously seen). (b) Whole tumor in FLAIR (previously unseen).

Fig. 13.

User time and Dice score of different interactive methods for 3D brain tumor segmentation. denotes previously unseen objects for BIFSeg.