Deep convolutional neural network and 3D deformable approach for tissue segmentation in musculoskeletal magnetic resonance imaging

Abstract

Purpose: To describe and evaluate a new fully automated musculoskeletal tissue segmentation method using deep convolutional neural network (CNN) and three-dimensional (3D) simplex deformable modeling to improve the accuracy and efficiency of cartilage and bone segmentation within the knee joint.

Methods: A fully automated segmentation pipeline was built by combining a semantic segmentation CNN and 3D simplex deformable modeling. A CNN technique called SegNet was applied as the core of the segmentation method to perform high resolution pixel-wise multi-class tissue classification. The 3D simplex deformable modeling refined the output from SegNet to preserve the overall shape and maintain a desirable smooth surface for musculoskeletal structure. The fully automated segmentation method was tested using a publicly available knee image data set to compare with currently used state-of-the-art segmentation methods. The fully automated method was also evaluated on two different data sets, which include morphological and quantitative MR images with different tissue contrasts.

Results: The proposed fully automated segmentation method provided good segmentation performance with segmentation accuracy superior to most of state-of-the-art methods in the publicly available knee image data set. The method also demonstrated versatile segmentation performance on both morphological and quantitative musculoskeletal MR images with different tissue contrasts and spatial resolutions.

Conclusion: The study demonstrates that the combined CNN and 3D deformable modeling approach is useful for performing rapid and accurate cartilage and bone segmentation within the knee joint. The CNN has promising potential applications in musculoskeletal imaging. Magn Reson Med 79:2379-2391, 2018. © 2017 International Society for Magnetic Resonance in Medicine.

Keywords: CNN; MRI; deep learning; deformable model; musculoskeletal imaging; segmentation.

FIG. 1.

An illustration of the SegNet CNN architecture. SegNet contains an encoder network and a decoder network. The final output of this network is high resolution pixel-wise tissue classification.

FIG. 2.

Flowchart of our proposed fully automated segmentation method. The complete process contains a training phase and a testing phase. SegNet is first trained with training images and labels. The well-trained SegNet and 3D simplex deformable modeling are then applied in the testing phase to carry out fully automated segmentation for the test images. In the testing phase, the 3D pixel-wise tissue labels from the SegNet output are passed to iterative processing filter to fill holes and remove small isolated objects using a connected-component filter. The processed labels are then sent to 3D simplex deformable process for each individual objects, and the final 3D segmentation is generated by combining all the deformed objects.

FIG. 3.

Example of the pixel-wise class possibility maps obtained from soft-max layer in SegNet. (a) A sagittal slice fat-suppressed T₁-weighted SPGR image for one subject in the SKI10 image data set. (b–f) Pixel-wise probability maps (%) for background, femur, femoral cartilage, tibia, and tibial cartilage, respectively. (g) Pixel-wise classification map where each individual pixel has the class index with highest class probability.

FIG. 4.

Example of the segmentation performance for one subject in the SKI10 image data set with severe cartilage loss. (a) A sagittal slice fat-suppressed T₁-weighted SPGR image with ROIs for femoral cartilage (green) and tibial cartilage (red). (b) Reference mask of the manual drawn segmentation carried out by experts from the SKI10 challenge.(c) Segmentation result directly from SegNet. (d) 3D rendering of the segmentation directly result from SegNet. (e and f) The intermediate filtered images obtained from Perona-Malik anisotropic diffusion filter and recursive Gaussian filter in 3D simplex deformable modeling process. (g) Final segmentation result obtained from our proposed method. (h) 3D rendering of the final segmentation result.

FIG. 5.

Example of the segmentation performance for one subject in the SKI10 image data set with a bone edema lesion. (a) A sagittal slice fat-suppressed T₁-weighted SPGR image with ROIs for femoral cartilage (green) and tibial cartilage (red). (b) Reference mask of the manual drawn segmentation carried out by experts from the SKI10 challenge. (c) Segmentation result directly from SegNet. (d) Final segmentation result obtained from our proposed method. (e) 3D rendering of the segmentation result directly from SegNet. (f) 3D rendering of the final segmentation result.

FIG. 6.

Example of the segmentation performance for one subject in the fat-suppressed 3D-FSE image data set. (a) A sagittal slice fat-suppressed 3D-FSE image. (b) Reference mask of the manual drawn segmentation. (c) Final segmentation result obtained from our proposed method with SegNet. (d) 3D rendering of the final segmentation result.

FIG. 7.

Example of the segmentation performance for one subject in the T₂ mapping image data set. Top: multiple sagittal slices T₂ maps generated using the multiple-echo single-exponential signal fitting. Middle: reference mask of the manual drawn segmentation using the DST method. Bottom: final segmentation result obtained from our proposed method with SegNet.