External Validation of Deep Learning Algorithm for Detecting and Visualizing Femoral Neck Fracture Including Displaced and Non-displaced Fracture on Plain X-ray

Abstract

This study aimed to develop a method for detection of femoral neck fracture (FNF) including displaced and non-displaced fractures using convolutional neural network (CNN) with plain X-ray and to validate its use across hospitals through internal and external validation sets. This is a retrospective study using hip and pelvic anteroposterior films for training and detecting femoral neck fracture through residual neural network (ResNet) 18 with convolutional block attention module (CBAM) + + . The study was performed at two tertiary hospitals between February and May 2020 and used data from January 2005 to December 2018. Our primary outcome was favorable performance for diagnosis of femoral neck fracture from negative studies in our dataset. We described the outcomes as area under the receiver operating characteristic curve (AUC), accuracy, Youden index, sensitivity, and specificity. A total of 4,189 images that contained 1,109 positive images (332 non-displaced and 777 displaced) and 3,080 negative images were collected from two hospitals. The test values after training with one hospital dataset were 0.999 AUC, 0.986 accuracy, 0.960 Youden index, and 0.966 sensitivity, and 0.993 specificity. Values of external validation with the other hospital dataset were 0.977, 0.971, 0.920, 0.939, and 0.982, respectively. Values of merged hospital datasets were 0.987, 0.983, 0.960, 0.973, and 0.987, respectively. A CNN algorithm for FNF detection in both displaced and non-displaced fractures using plain X-rays could be used in other hospitals to screen for FNF after training with images from the hospital of interest.

Keywords: AI; Artificial intelligence; Deep learning; Femur; Fracture; Machine learning.

Fig. 1

Flowchart of data collection and analyzing in the study

Fig. 2

Architecture and network comparison of deep learning neural network for detection of FNF. a Architecture of ResNet18 with attention module. ResNet18 is composed of 8 residual blocks, every two blocks belong to the same stage and have the same channels of output. b Diagram of CBAM +  + . The two outputs pooled along the channel axis are resized by additional pooling operations according to stage number and forwarded to a convolution layer. c the AUC of internal validation with Hospital A dataset and the number of Network parameters. The parameter number of ResNet18 with CBAM +  + is the lowest and the AUC value is the highest

Fig. 3

Receiver operator characteristics (ROC) curves and test comparing two AUCs of (1) Test after training and internal validation with A set, (2) External Validation with B set after training and internal validation with A set (3) Test after training and internal validation with A + B sets. *p < 0.05 is statistically significant

Fig. 4

Visualization with Grad-Class Activation mapping (CAM) results in external validation test. A Correct detection images. The images in 1st and 2nd row (true positive images) are the original plain C-ray images and CAM applied images with FNF, whereas the images in 3rd and 4th row (true negative images) are the original plain X-ray images and CAM applied images without fracture. B False detection images. The images in in 1st row are false-positive images, whereas the images in in 2nd, 3rd, and 4th row are false-negative images with unidentified areas highlighted by CAM in images