DeepIGeoS: A Deep Interactive Geodesic Framework for Medical Image Segmentation

Abstract

Accurate medical image segmentation is essential for diagnosis, surgical planning and many other applications. Convolutional Neural Networks (CNNs) have become the state-of-the-art automatic segmentation methods. However, fully automatic results may still need to be refined to become accurate and robust enough for clinical use. We propose a deep learning-based interactive segmentation method to improve the results obtained by an automatic CNN and to reduce user interactions during refinement for higher accuracy. We use one CNN to obtain an initial automatic segmentation, on which user interactions are added to indicate mis-segmentations. Another CNN takes as input the user interactions with the initial segmentation and gives a refined result. We propose to combine user interactions with CNNs through geodesic distance transforms, and propose a resolution-preserving network that gives a better dense prediction. In addition, we integrate user interactions as hard constraints into a back-propagatable Conditional Random Field. We validated the proposed framework in the context of 2D placenta segmentation from fetal MRI and 3D brain tumor segmentation from FLAIR images. Experimental results show our method achieves a large improvement from automatic CNNs, and obtains comparable and even higher accuracy with fewer user interventions and less time compared with traditional interactive methods.

Fig. 1.

Overview of the proposed interactive segmentation framework with two stages. Stage 1: P-Net automatically proposes an initial segmentation. Stage 2: R-Net refines the segmentation with user interactions indicating mis-segmentations. CRF-Net(f) is our proposed back-propagatable CRF that uses freeform pairwise potentials. It is extended to be CRF-Net(fu) that employs user interactions as hard constraints.

Fig. 2.

Input of R-Net in Stage 2. (a) The user provides clicks/scribbles to correct foreground (red) and background (cyan) on the initial automatic segmentation. (d) and (e) are geodesic distance maps based on foreground and background interactions, respectively. The original image (b) is combined with the initial automatic segmentation (c) and the geodesic distance maps (d), (e) by channel-concatenation and used as the input of R-Net.

Fig. 3.

The CNN structure of 2D/3D P-Net with CRF-Net(f). The parameters of convolution layers are (kernel size, output channels, dilation) in dark blue rectangles. Block 1 to block 6 are resolution-preserving. 2D/3D R-Net uses the same structure as 2D/3D P-Net except its input has three additional channels shown in Fig. 2 and the CRF-Net(f) is replaced by the CRF-Net(fu) (Section 3.3).

Fig. 4.

The Pairwise-Net for pairwise potential function $f({\tilde{\mathbf{f}}}_{ij},d_{ij})$. ${\tilde{\mathbf{f}}}_{ij}$ is the difference of features between a pixel pair $i$ and $j$. $d_{ij}$ is the euclidean distance between them.

Fig. 5.

Simulated user interactions on training images for placenta (a) and brain tumor (b, c). Green: automatic segmentation given by P-Net with CRF-Net(f). Yellow: ground truth. Red (cyan): simulated clicks on under-segmentation (over-segmentation).

Fig. 6.

Visual comparison of different networks in Stage 1 of 2D placenta segmentation. The last row shows interactively refined results by DeepIGeoS.

Fig. 7.

Visual comparison of different CRFs in Stage 1 of 2D placenta segmentation. The last row shows interactively refined results by DeepIGeoS.

Fig. 8.

Visual comparison of different refinement methods in Stage 2 of 2D placenta segmentation. The first column shows the initial automatic segmentation obtained by 2D P-Net + CRF-Net(f), on which user interactions are added for refinement. The remaining columns show refined results. 2D R-Net(Euc) is a counterpart of the proposed 2D R-Net and it uses euclidean distance. White arrows show the difference in local details.

Fig. 9.

Visual comparison of DeepIGeoS and other interactive methods for 2D placenta segmentation. The first row shows initial scribbles (except for DeepIGeoS) and the resulting segmentation. The second row shows final refined results with the entire set of scribbles. The user decided on the level of interaction required to achieve a visually acceptable result.

Fig. 10.

Quantitative comparison of 2D placenta segmentation by different interactive methods in terms of Dice, ASSD, total interactions (scribble length) and user time.

Fig. 11.

Visual comparison of different networks in Stage 1 of 3D brain tumor segmentation. The last row shows interactively refined results by DeepIGeoS.

Fig. 12.

Visual comparison of different CRFs in Stage 1 of 3D brain tumor segmentation. The last column shows interactively refined results by DeepIGeoS.

Fig. 13.

Visual comparison of different refinement methods in Stage 2 of 3D brain tumor segmentation. The initial segmentation is obtained by 3D P-Net + CRF-Net(f), on which user interactions are given. 3D R-Net(Euc) is a counterpart of the proposed 3D R-Net and it uses euclidean distance.

Fig. 14.

Visual comparison of 3D brain tumor segmentation using GeoS, ITK-SNAP, and DeepIGeoS that is based on 3D P-Net.

Fig. 15.

Quantitative evaluation of 3D brain tumor segmentation by DeepIGeoS, GeoS and ITK-SNAP.