Development and Validation of a Convolutional Neural Network for Automated Detection of Scaphoid Fractures on Conventional Radiographs

Abstract

Purpose: To compare the performance of a convolutional neural network (CNN) to that of 11 radiologists in detecting scaphoid bone fractures on conventional radiographs of the hand, wrist, and scaphoid.

Materials and methods: At two hospitals (hospitals A and B), three datasets consisting of conventional hand, wrist, and scaphoid radiographs were retrospectively retrieved: a dataset of 1039 radiographs (775 patients [mean age, 48 years ± 23 {standard deviation}; 505 female patients], period: 2017-2019, hospitals A and B) for developing a scaphoid segmentation CNN, a dataset of 3000 radiographs (1846 patients [mean age, 42 years ± 22; 937 female patients], period: 2003-2019, hospital B) for developing a scaphoid fracture detection CNN, and a dataset of 190 radiographs (190 patients [mean age, 43 years ± 20; 77 female patients], period: 2011-2020, hospital A) for testing the complete fracture detection system. Both CNNs were applied consecutively: The segmentation CNN localized the scaphoid and then passed the relevant region to the detection CNN for fracture detection. In an observer study, the performance of the system was compared with that of 11 radiologists. Evaluation metrics included the Dice similarity coefficient (DSC), Hausdorff distance (HD), sensitivity, specificity, positive predictive value (PPV), and area under the receiver operating characteristic curve (AUC).

Results: The segmentation CNN achieved a DSC of 97.4% ± 1.4 with an HD of 1.31 mm ± 1.03. The detection CNN had sensitivity of 78% (95% CI: 70, 86), specificity of 84% (95% CI: 77, 92), PPV of 83% (95% CI: 77, 90), and AUC of 0.87 (95% CI: 0.81, 0.91). There was no difference between the AUC of the CNN and that of the radiologists (0.87 [95% CI: 0.81, 0.91] vs 0.83 [radiologist range: 0.79-0.85]; P = .09).

Conclusion: The developed CNN achieved radiologist-level performance in detecting scaphoid bone fractures on conventional radiographs of the hand, wrist, and scaphoid.Keywords: Convolutional Neural Network (CNN), Deep Learning Algorithms, Machine Learning Algorithms, Feature Detection-Vision-Application Domain, Computer-Aided DiagnosisSee also the commentary by Li and Torriani in this issue.Supplemental material is available for this article.©RSNA, 2021.

Keywords: Computer-Aided Diagnosis; Convolutional Neural Network (CNN); Deep Learning Algorithms; Feature Detection-Vision-Application Domain; Machine Learning Algorithms.

Figure 1:

Illustration shows an overview of the scaphoid fracture detection pipeline, which consists of segmentation and detection convolutional neural networks (CNN). A class activation map is calculated and visualized as a heatmap for fracture localization.

Figure 2:

(A) Graph shows the receiver operating characteristic (ROC) curves for the detection convolutional neural network (CNN) (blue line), the average performance for detection of fractures for radiologists (orange line), and the radiologists with the highest and lowest areas under the curve (AUC) (striped and dotted orange line) on dataset 3 (95 fracture cases, 95 nonfracture cases). The shaded band represents the 95% CI. The black line represents no ability to discriminate between fractures and nonfractures. (B) Graph shows the same ROC curve for the detection CNN with individual operating points of the CNN and radiologists on dataset 3.

Figure 3:

Scatterplot shows the confidence scores for the detection convolutional neural network (CNN) versus the average confidence score of the radiologists per fracture case (n = 95) and nonfracture case (n = 95) in dataset 3. Fracture and nonfracture cases are marked in red and blue, respectively (ground truth). The confidence scores range from 0 to 1, where 1 means absolute certainty that a fracture is present. Example radiographs in which the CNN and radiologists showed high disagreement are shown next to the plot: (A) A 17-year-old male with a proximal scaphoid fracture, (B) a 21-year-old man with a fracture of the distal pole of the scaphoid (volar and radial side), (C) a 45-year-old woman with an intact scaphoid and a slightly irregular radial side, and (D) 13-year-old girl with an intact scaphoid with no irregularities.

Figure 4:

Examples of class activation maps (CAMs) for localizing fractures. The interpretation of the CAMs is performed on the basis of the color coding of a heatmap, in which pixel regions with a warm color signify a greater influence on the final decision of the network than regions with a cold color. The yellow arrows on the input images indicate the fracture lines, and they are only shown for reference.