Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2023 Mar;96(1143):20211104.
doi: 10.1259/bjr.20211104. Epub 2023 Feb 6.

Lessons learned from independent external validation of an AI tool to detect breast cancer using a representative UK data set

Dominic Cushnan  1 , Kenneth C Young  2   3 , Dominic Ward  2 , Mark D Halling-Brown  2   3 , Stephen Duffy  4 , Rosalind Given-Wilson  5 , Matthew G Wallis  6 , Louise Wilkinson  7 , Iain Lyburn  8   9   10 , Richard Sidebottom  8   11 , Rita McAvinchey  12 , Emma B Lewis  2 , Alistair Mackenzie  2 , Lucy M Warren  2
Affiliations

Lessons learned from independent external validation of an AI tool to detect breast cancer using a representative UK data set

Dominic Cushnan et al. Br J Radiol. 2023 Mar.

Abstract

Objective: To pilot a process for the independent external validation of an artificial intelligence (AI) tool to detect breast cancer using data from the NHS breast screening programme (NHSBSP).

Methods: A representative data set of mammography images from 26,000 women attending 2 NHS screening centres, and an enriched data set of 2054 positive cases were used from the OPTIMAM image database. The use case of the AI tool was the replacement of the first or second human reader. The performance of the AI tool was compared to that of human readers in the NHSBSP.

Results: Recommendations for future external validations of AI tools to detect breast cancer are provided. The tool recalled different breast cancers to the human readers. This study showed the importance of testing AI tools on all types of cases (including non-standard) and the clarity of any warning messages. The acceptable difference in sensitivity and specificity between the AI tool and human readers should be determined. Any information vital for the clinical application should be a required output for the AI tool. It is recommended that the interaction of radiologists with the AI tool, and the effect of the AI tool on arbitration be investigated prior to clinical use.

Conclusion: This pilot demonstrated several lessons for future independent external validation of AI tools for breast cancer detection.

Advances in knowledge: Knowledge has been gained towards best practice procedures for performing independent external validations of AI tools for the detection of breast cancer using data from the NHS Breast Screening Programme.

PubMed Disclaimer

Conflict of interest statement

conflictNone of the authors had any conflicts of interest with the AI vendor in the validation.

Figures

Figure 1.
Figure 1.
(a) Normal screening workflow. (b) Workflow when AI tool replaces one of the human readers (independent AI reader). AI, artificial intelligence.
See this image and copyright information in PMC

References

    1. Taylor-Phillips S, Stinton C. Double reading in breast cancer screening: considerations for policy-making. Br J Radiol 2020; 93: 20190610. doi: 10.1259/bjr.20190610 - DOI - PMC - PubMed
    1. NHS Breast Screening Programme . Interval cancers explained. Available from: https://assets.publishing.service.gov.uk/government/uploads/system/uploa...
    1. McKinney SM, Sieniek M, Godbole V, Godwin J, Antropova N, Ashrafian H, et al. . International evaluation of an AI system for breast cancer screening. Nature 2020; 577: 89–94. doi: 10.1038/s41586-019-1799-6 - DOI - PubMed
    1. Salim M, Wåhlin E, Dembrower K, Azavedo E, Foukakis T, Liu Y, et al. . External evaluation of 3 commercial artificial intelligence algorithms for independent assessment of screening mammograms. JAMA Oncol 2020; 6: 1581–88. doi: 10.1001/jamaoncol.2020.3321 - DOI - PMC - PubMed
    1. Schaffter T, Buist DSM, Lee CI, Nikulin Y, Ribli D, Guan Y, et al. . Evaluation of combined artificial intelligence and radiologist assessment to interpret screening mammograms. JAMA Netw Open 2020; 3: e200265. doi: 10.1001/jamanetworkopen.2020.0265 - DOI - PMC - PubMed