Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 Apr 25;6(1):tzae011.
doi: 10.1093/bjro/tzae011. eCollection 2024 Jan.

Improving traumatic fracture detection on radiographs with artificial intelligence support: a multi-reader study

Rikke Bachmann  1 , Gozde Gunes  2 , Stine Hangaard  3 , Andreas Nexmann  1 , Pavel Lisouski  1 , Mikael Boesen  4   5 , Michael Lundemann  1 , Scott G Baginski  6
Affiliations

Improving traumatic fracture detection on radiographs with artificial intelligence support: a multi-reader study

Rikke Bachmann et al. BJR Open. .

Abstract

Objectives: The aim of this study was to evaluate the diagnostic performance of nonspecialist readers with and without the use of an artificial intelligence (AI) support tool to detect traumatic fractures on radiographs of the appendicular skeleton.

Methods: The design was a retrospective, fully crossed multi-reader, multi-case study on a balanced dataset of patients (≥2 years of age) with an AI tool as a diagnostic intervention. Fifteen readers assessed 340 radiographic exams, with and without the AI tool in 2 different sessions and the time spent was automatically recorded. Reference standard was established by 3 consultant radiologists. Sensitivity, specificity, and false positives per patient were calculated.

Results: Patient-wise sensitivity increased from 72% to 80% (P < .05) and patient-wise specificity increased from 81% to 85% (P < .05) in exams aided by the AI tool compared to the unaided exams. The increase in sensitivity resulted in a relative reduction of missed fractures of 29%. The average rate of false positives per patient decreased from 0.16 to 0.14, corresponding to a relative reduction of 21%. There was no significant difference in average reading time spent per exam. The largest gain in fracture detection performance, with AI support, across all readers, was on nonobvious fractures with a significant increase in sensitivity of 11 percentage points (pp) (60%-71%).

Conclusions: The diagnostic performance for detection of traumatic fractures on radiographs of the appendicular skeleton improved among nonspecialist readers tested AI fracture detection support tool showed an overall reader improvement in sensitivity and specificity when supported by an AI tool. Improvement was seen in both sensitivity and specificity without negatively affecting the interpretation time.

Advances in knowledge: The division and analysis of obvious and nonobvious fractures are novel in AI reader comparison studies like this.

Keywords: artificial intelligence; diagnostic performance; fracture detection; multi-reader study.

PubMed Disclaimer

Conflict of interest statement

R.B., M.L., P.L., and A.N. are employees of Radiobotics. M.B. is a member of the clinical advisory board of Radiobotics.

Figures

Figure 1.
Figure 1.
Study workflow.
Figure 2.
Figure 2.
ROC and free-response receiver operating characteristic curves are shown in (A) and (B), respectively. The coloured circles show the unaided performance and the arrow-end show the aided performance for each reader. Arrows pointing towards the upper left corner indicate readers who improved both the sensitivity and specificity. The dashed lines were constructed by artificially varying the probability threshold of the AI tool. The solid grey circles represent the standalone performance of the AI tool. The ROC-AUC was 0.94 (CI, 0.90-0.97). Abbreviations: ROC-AUC = area under the ROC curve; ROC = receiver operating characteristic.
Figure 3.
Figure 3.
Anterior-posterior (A) and lateral (C) radiographs of nondisplaced fracture of the olecranon (white arrows). The fracture was missed by 11 readers in the unaided exam, but detected by all readers in the aided exam, with support from the AI tool (B, D).
Figure 4.
Figure 4.
Stacked bar plot showing the average number of missed fractures for each professional group and the AI tool, in unaided and aided nonobvious fractures as well as unaided and unaided obvious fractures. Error bars show the measure of spread (±1 SD) from the average values. Abbreviation: AI = artificial intelligence.
Figure 5.
Figure 5.
Anterior-posterior radiograph (A) of a nonobvious hip fracture unaided. The subcapital hip fracture (B, white arrow) was missed by 9 readers in the unaided exam, but detected by all readers in the aided exam, with support from the AI tool (C).
Figure 6.
Figure 6.
A lateral paediatric hand radiograph (A) with a Salter Harris type II fracture of the base of fifth proximal phalanx (B, white arrow). The fracture was detected by the AI tool (C). The fracture was missed by 11 readers in the unaided exam, but detected by 14 readers in the aided exam with support from the AI tool.
Figure 7.
Figure 7.
An anterior-posterior radiograph of the left shoulder and clavicle of a 30-year-old woman. A markedly displaced fracture of the clavicle (white arrow) was missed by the AI tool. The fracture was detected by 11 readers in the unaided session and 14 readers in the aided session.
See this image and copyright information in PMC

References

    1. Weinick RM, Burns RM, Mehrotra A.. Many emergency department visits could be managed at urgent care centers and retail clinics. Health Aff (Millwood). 2010; 29(9):1630-1636. - PMC - PubMed
    1. Court-Brown CM, Caesar B.. Epidemiology of adult fractures: a review. Injury. 2006;37(8):691-697. - PubMed
    1. Shibuya N, Davis ML, Jupiter DC.. Epidemiology of foot and ankle fractures in the United States: an analysis of the National Trauma Data Bank (2007 to 2011). J Foot Ankle Surg. 2014;2953(5):606-608. - PubMed
    1. Petinaux B, Bhat R, Boniface K, Aristizabal J.. Accuracy of radiographic readings in the emergency department. Am J Emerg Med. 2011;29(1):18-25. - PubMed
    1. Kranz R, Cosson P.. Anatomical and/or pathological predictors for the “incorrect” classification of red dot markers on wrist radiographs taken following trauma. Br J Radiol. 2015;88(1046):20140503. - PMC - PubMed