Evolution of an Artificial Intelligence-Powered Application for Mammography

Abstract

Background: The implementation of radiological artificial intelligence (AI) solutions remains challenging due to limitations in existing testing methodologies. This study assesses the efficacy of a comprehensive methodology for performance testing and monitoring of commercial-grade mammographic AI models. Methods: We utilized a combination of retrospective and prospective multicenter approaches to evaluate a neural network based on the Faster R-CNN architecture with a ResNet-50 backbone, trained on a dataset of 3641 mammograms. The methodology encompassed functional and calibration testing, coupled with routine technical and clinical monitoring. Feedback from testers and radiologists was relayed to the developers, who made updates to the AI model. The test dataset comprised 112 medical organizations, representing 10 manufacturers of mammography equipment and encompassing 593,365 studies. The evaluation metrics included the area under the curve (AUC), accuracy, sensitivity, specificity, technical defects, and clinical assessment scores. Results: The results demonstrated significant enhancement in the AI model's performance through collaborative efforts among developers, testers, and radiologists. Notable improvements included functionality, diagnostic accuracy, and technical stability. Specifically, the AUC rose by 24.7% (from 0.73 to 0.91), the accuracy improved by 15.6% (from 0.77 to 0.89), sensitivity grew by 37.1% (from 0.62 to 0.85), and specificity increased by 10.7% (from 0.84 to 0.93). The average proportion of technical defects declined from 9.0% to 1.0%, while the clinical assessment score improved from 63.4 to 72.0. Following 2 years and 9 months of testing, the AI solution was integrated into the compulsory health insurance system. Conclusions: The multi-stage, lifecycle-based testing methodology demonstrated substantial potential in software enhancement and integration into clinical practice. Key elements of this methodology include robust functional and diagnostic requirements, continuous testing and updates, systematic feedback collection from testers and radiologists, and prospective monitoring.

Keywords: artificial intelligence; mammography; radiology; software; software validation.

Figure 1

Model architecture. * Module components or actions performed.

Figure 2

Overview of the study design.

Figure 3

Overview of the study timeline.

Figure 4

Additional image series with graphical masks highlighting pathological findings.

Figure 5

ROC curves for (a) version 0.14.0 and (b) version 0.15.0 of the Celsus Mammography AI model: results of the first and second calibration tests.

Figure 6

(a) Variations in A and B defects during the technical monitoring; (b) changes in the average rate of technical defects observed during the technical monitoring.

Figure 7

Changes in the clinical assessment score during the clinical monitoring.

Figure 8

False-positive findings by the AI model: (a) benign vascular calcifications misclassified as malignant calcification, and (b) normal fibroglandular tissue misclassified as a malignant mass.

Figure 9

True-positive results from the AI model, illustrating the detected irregular masses with indistinct margins in the right and left breasts.

Figure 10

(a) ROC curve for Celsus Mammography (version 0.16.0) from the third calibration test; (b) ROC curve for Celsus Mammography (version 0.17.0) from the fourth calibration test.