Musculoskeletal radiologist-level performance by using deep learning for detection of scaphoid fractures on conventional multi-view radiographs of hand and wrist

Abstract

Objectives: To assess how an artificial intelligence (AI) algorithm performs against five experienced musculoskeletal radiologists in diagnosing scaphoid fractures and whether it aids their diagnosis on conventional multi-view radiographs.

Methods: Four datasets of conventional hand, wrist, and scaphoid radiographs were retrospectively acquired at two hospitals (hospitals A and B). Dataset 1 (12,990 radiographs from 3353 patients, hospital A) and dataset 2 (1117 radiographs from 394 patients, hospital B) were used for training and testing a scaphoid localization and laterality classification component. Dataset 3 (4316 radiographs from 840 patients, hospital A) and dataset 4 (688 radiographs from 209 patients, hospital B) were used for training and testing the fracture detector. The algorithm was compared with the radiologists in an observer study. Evaluation metrics included sensitivity, specificity, positive predictive value (PPV), area under the characteristic operating curve (AUC), Cohen's kappa coefficient (κ), fracture localization precision, and reading time.

Results: The algorithm detected scaphoid fractures with a sensitivity of 72%, specificity of 93%, PPV of 81%, and AUC of 0.88. The AUC of the algorithm did not differ from each radiologist (0.87 [radiologists' mean], p ≥ .05). AI assistance improved five out of ten pairs of inter-observer Cohen's κ agreements (p < .05) and reduced reading time in four radiologists (p < .001), but did not improve other metrics in the majority of radiologists (p ≥ .05).

Conclusions: The AI algorithm detects scaphoid fractures on conventional multi-view radiographs at the level of five experienced musculoskeletal radiologists and could significantly shorten their reading time.

Key points: • An artificial intelligence algorithm automatically detects scaphoid fractures on conventional multi-view radiographs at the same level of five experienced musculoskeletal radiologists. • There is preliminary evidence that automated scaphoid fracture detection can significantly shorten the reading time of musculoskeletal radiologists.

Keywords: Artificial intelligence; Clinical decision support system; Fractures, bone; Multicenter study; Scaphoid bone.

Fig. 1

Flowchart for the inclusion and exclusion of samples in dataset 4 (test data). The number of studies at each step is denoted with n. DBC, diagnosis-treatment combination; ICD-10, International Classification of Diseases Version 10; JBZ, Jeroen Bosch Hospital. ²Studies with ICD-10 diagnosis code S62.00 were included. ³There was no patient overlap between the samples from the non-fracture and fracture category. ⁴Studies were excluded when the wrist was in cast, the scaphoid was incompletely depicted, or when there was severe scapholunate advanced collapse

Fig. 2

Overview of the scaphoid fracture detection artificial intelligence (AI) pipeline, which consisted of four convolutional neural networks (CNNs): a scaphoid localization CNN, scaphoid laterality classification CNN, and two scaphoid fracture detection CNNs for processing frontal or oblique view radiographs (including anterior-posterior/posterior-anterior [AP/PA] and ulnar-deviated AP/PA views) and lateral view radiographs separately

Fig. 3

a Receiver operating characteristic (ROC) curve (blue) with operating point of the automated scaphoid fracture detection results based on all available radiographic views from dataset 4 (65 fracture cases, 154 non-fracture cases; each case represents one hand from one patient). The corresponding mean localization precision curve (orange) is shown as well. The shaded bands represent 95% confidence intervals. The black line represents no ability to discriminate between fracture and non-fracture cases. b Receiver operating characteristic (ROC) curves of the automated scaphoid fracture detection results for multiple input configurations on the same dataset. All 219 cases included at least one posterior-anterior (PA) view (65 cases with fracture), 55 cases included at least one ulnar-deviated PA view (30 cases with fracture), 159 cases included at least one oblique view (52 cases with fracture), and 156 cases included at least one lateral view (63 with fracture)

Fig. 4

a Receiver operating characteristic (ROC) curves of the results of the scaphoid fracture detection algorithm and those of the musculoskeletal (MSK) radiologists without artificial intelligence (AI) assistance on dataset 4 (65 fracture cases, 154 non-fracture cases; each case represents one hand from one patient). b ROC curves of the results of the scaphoid fracture detection algorithm and those of the MSK radiologists with AI assistance on the same dataset. The corner of each plot is magnified for easier comparison of the curves. The black line represents no ability to discriminate between fracture and non-fracture cases.

Fig. 5

False positive (FP) and false negative (FN) detections made by the scaphoid fracture detection artificial intelligence (AI) algorithm that none of the five musculoskeletal radiologists made. The AI fracture score per scaphoid region (ranging from 0 [no fracture] to 1 [fracture], rounded to two decimals) is shown below each image. The yellow arrows indicate the fracture locations and are only shown for reference. False positive case descriptions (from top to bottom): 13-year-old male and 77-year-old female with an intact scaphoid, 81-year-old male with rheumatoid arthritis with an old healed waist scaphoid fracture. False negative case descriptions (from top to bottom): 67-year-old female with a slightly displaced transverse waist scaphoid fracture (transition middle one-third to distal one-third), 37-year-old male with a waist oblique scaphoid fracture (transition proximal one-third to middle one-third), 77-year-old female with a waist scaphoid fracture (middle one-third)

Fig. 6

False positive (FP) and false negative (FN) detections made by the majority of the five musculoskeletal radiologists that the artificial intelligence (AI) algorithm did not make. The proportion of radiologists making the FP or FN detection is shown in the upper right corner of each image. The corresponding fracture scores per scaphoid region of the radiologists (mean score of responsible radiologists per region) and AI (ranging from 0 [no fracture] to 1 [fracture], rounded to two decimals) are shown below each image. The yellow arrows indicate the fracture locations and are only shown for reference. Case descriptions first row (left to right): 27-year-old female, 12-year-old male, and 50-year-old female with an intact scaphoid. Case descriptions second row (left to right): 74-year-old female and 59-year-old female with an intact scaphoid, 79-year-old female with calcium pyrophosphate deposition arthritis with calcifications surrounding the triangular fibrocartilage complex. Case descriptions third row (left to right): 69-year-old female with osteophyte and subchondral cyst formation, 45-year-old female with an intact scaphoid, 74-year-old male with radiocarpal and scapho-trapezium/trapezoid joint arthritis. Case descriptions fourth row (left to right): 23-year-old male with a waist scaphoid fracture (middle one-third), 60-year-old female with a waist scaphoid fracture (distal one-third), 12-year-old female with a waist scaphoid fracture (middle one-third)