Segmentation of the Proximal Femur from MR Images using Deep Convolutional Neural Networks

Abstract

Magnetic resonance imaging (MRI) has been proposed as a complimentary method to measure bone quality and assess fracture risk. However, manual segmentation of MR images of bone is time-consuming, limiting the use of MRI measurements in the clinical practice. The purpose of this paper is to present an automatic proximal femur segmentation method that is based on deep convolutional neural networks (CNNs). This study had institutional review board approval and written informed consent was obtained from all subjects. A dataset of volumetric structural MR images of the proximal femur from 86 subjects were manually-segmented by an expert. We performed experiments by training two different CNN architectures with multiple number of initial feature maps, layers and dilation rates, and tested their segmentation performance against the gold standard of manual segmentations using four-fold cross-validation. Automatic segmentation of the proximal femur using CNNs achieved a high dice similarity score of 0.95 ± 0.02 with precision = 0.95 ± 0.02, and recall = 0.95 ± 0.03. The high segmentation accuracy provided by CNNs has the potential to help bring the use of structural MRI measurements of bone quality into clinical practice for management of osteoporosis.

Figure 1

Overview of the proposed learning algorithm for an automatic segmentation of the proximal femur. Training CNN yields automatic proximal segmentation model that is used in model evaluation on a test dataset. The output of the model is the probability of the bone which is used to obtain the proximal femur segmentation mask using a threshold.

Figure 2

ROC and Precision-Recall Curve for 2D and 3D CNN segmentation models. Left panel shows the receiver operating characteristics (ROC) curves of different CNNs modeled in this work. The number of initial feature maps (F) and layers (L) in the contracting/expanding paths are presented in the legend with the area under the curve (AUC). Right panel shows the precision- recall curves of modeled CNNs. In the legend, cross-validation average precision (AP) is presented for comparison of different models. * indicates the 2D CNN with unpadded convolutions.

Figure 3

Box plots for dice score, precision and recall that are obtained from the binary segmentation map from each individual. F is the number of initial feature maps, L is the number of layers, PP is the post-processing. * indicates the 2D CNN with unpadded convolutions.

Figure 4

An example of the results using 2D CNN and 3D CNN. 3T MRI of the proximal femur (a) is shown with the ground truth/hand segmentation mask (e). The white dashed line represents the location where the sagittal view is displayed from the coronal view. The probability map produced by 2D CNN is presented in (b) and corresponding segmentation mask after post-processing is presented in (f). Red arrow in (b) indicates a location which was misclassified by the 2D CNN. Using padded convolution provided superior segmentation (b vs c). Some of the misclassified regions in (b) are removed by using the padded convolution; however, there are still regions that are misclassified as indicated by the red arrow in (c). Misclassified regions were removed by post-processing using proximal femur connectivity and size prior information (f and g). Probability map produced by 3D CNN is presented in (d) and corresponding segmentation mask obtained by thresholding without post-processing is presented in (h).

Figure 5

Examples of the suboptimal segmentation results. First row images are from a subject who has a bone cysts in the proximal femur. 3T MRI of the proximal femur in (a) is shown with the ground truth/hand segmentation mask overlaid in (b). Both 2D (c) and 3D (d) CNN were not capable of segmenting the proximal femur of this subject with high accuracy. Second row images are from an acquisition where there is a foldover artifact (indicated by the white arrow) that is not affecting the hand segmentation (f). However, foldover artifacts are affecting the accuracy of automatic proximal femur segmentations of both 2D and 3D CNN (g,h). These segmentation results remained suboptimal with minor improvements when dilated convolutions are used.

Figure 6

CNN architecture of one of the 3D CNNs used in the paper. Blue rectangles represent feature maps with the size and the number of feature maps indicated. Different operations in the network are depicted by color-coded arrows. The architecture represented here contains 32 feature maps in the first and last layer of the network and 4 layers in the contracting/expanding paths. In 3D CNN-dilated, dilated convolutions with multiple dilation rates are performed and concatenated (as indicated by green dashed boxes) at the center layer of the original 3D CNN.

Figure 7

Examples of train and validation loss plots for two different CNNs. Weighted cross-entropy loss was minimized using the Adam algorithm. As indicated by the x-axis, the number of epochs used for training different CNNs differs due to early stopping criteria used during cross-validation. In both cases, as expected, the loss in train dataset is lower that the validation set when the training is stopped.