Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Review
. 2024 Mar 4;1(1):ubae003.
doi: 10.1093/bjrai/ubae003. eCollection 2024 Jan.

Artificial intelligence in medicine: mitigating risks and maximizing benefits via quality assurance, quality control, and acceptance testing

Usman Mahmood  1 , Amita Shukla-Dave  1   2 , Heang-Ping Chan  3 , Karen Drukker  4 , Ravi K Samala  5 , Quan Chen  6 , Daniel Vergara  7 , Hayit Greenspan  8 , Nicholas Petrick  5 , Berkman Sahiner  5 , Zhimin Huo  9 , Ronald M Summers  10 , Kenny H Cha  5 , Georgia Tourassi  11 , Thomas M Deserno  12 , Kevin T Grizzard  13 , Janne J Näppi  14 , Hiroyuki Yoshida  14 , Daniele Regge  15   16 , Richard Mazurchuk  17 , Kenji Suzuki  18 , Lia Morra  19 , Henkjan Huisman  20 , Samuel G Armato 3rd  4 , Lubomir Hadjiiski  3
Affiliations
Review

Artificial intelligence in medicine: mitigating risks and maximizing benefits via quality assurance, quality control, and acceptance testing

Usman Mahmood et al. BJR Artif Intell. .

Abstract

The adoption of artificial intelligence (AI) tools in medicine poses challenges to existing clinical workflows. This commentary discusses the necessity of context-specific quality assurance (QA), emphasizing the need for robust QA measures with quality control (QC) procedures that encompass (1) acceptance testing (AT) before clinical use, (2) continuous QC monitoring, and (3) adequate user training. The discussion also covers essential components of AT and QA, illustrated with real-world examples. We also highlight what we see as the shared responsibility of manufacturers or vendors, regulators, healthcare systems, medical physicists, and clinicians to enact appropriate testing and oversight to ensure a safe and equitable transformation of medicine through AI.

Keywords: acceptance testing; artificial intelligence; deep learning; machine learning; quality assurance; quality control; radiology.

PubMed Disclaimer

Conflict of interest statement

U.M., A.S.-D., H.-P.C., R.K.S., D.V., H.G., N.P., B.S., Z.H., K.C., G.T., T.M.D., D.R., R.M., and L.H. have nothing to disclose. K.D. receives royalties from Hologic. Q.C. has received compensations from Carina Medical LLC, not related to this work, provides consulting services for Reflexion Medical, not related to this work. R.M.S. received royalties for patents or software licenses from iCAD, Philips, ScanMed, Translation Holdings, PingAn, and MGB, and received research support from PingAn through a Cooperative Research and Development Agreement, not related to this work. J.J.N. has received royalties from Hologic and from MEDIAN Technologies, through the University of Chicago licensing, not related to this work. H.Y. has received royalties from licensing fees to Hologic and Medians Technologies through the University of Chicago licensing, not related to this work. K.S. provides consulting services for Canon Medical, not related to this work. L.M. has received funding from HealthTriagesrl, not related to this work. H.H. has received funding from Siemens Healthineers for a scientific research project, not related to this work. SG.A. III has received royalties and licensing fees for computer-aided diagnosis through the University of Chicago Consultant, Novartis, not related to this work.

Figures

Figure 1.
Figure 1.
Flowchart illustrating the interconnected processes of quality assurance (QA), acceptance testing (AT), and quality control (QC) for AI tools in medical settings. The figure delineates the key steps emphasizing the cyclical nature of these processes for continuous improvement and patient safety. Some information that may be necessary for AT or the overall QA program must be obtained and reviewed before purchasing the AI tool. At installation of any AI tool, the necessary information must be provided to the team performing AT and ongoing QC procedures.
Figure 2.
Figure 2.
Overview of the different information sources involved in AI development, regulatory review, and clinical installation. Upon model completion (left), the locked model either undergoes regulatory review, additional retrospective or prospective multi-institutional validation (middle), or local clinical installation (right). Finally, before the tool is deployed for clinical use, testing with a site-specific, locally curated test set, the composition of which could be facilitated by vendor transparency, is essential.
Figure 3.
Figure 3.
Example considerations for quality assurance (QA) in the 3 phases of AI tool integration into the clinical workflow: pre-install QA preparation, post-install acceptance testing, and routine quality control.
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

See all "Cited by" articles

References

    1. Davis MA, Lim N, Jordan J, Yee J, Gichoya JW, Lee R.. Imaging artificial intelligence: a framework for radiologists to address health equity, from the AJR special series on DEI. AJR. 2023;221(3):302-308. - PubMed
    1. Geis JR, Brady AP, Wu CC, et al.Ethics of artificial intelligence in radiology: summary of the joint European and North American multisociety statement. Radiology. 2019;293(2):436-440. - PubMed
    1. Lundström C, Lindvall M.. Mapping the landscape of care providers’ quality assurance approaches for AI in diagnostic imaging. J Digit Imaging. 2023;36(2):379-387. - PMC - PubMed
    1. Mezrich JL. Is artificial intelligence (AI) a pipe dream? Why legal issues present significant hurdles to AI autonomy. AJR Am J Roentgenol. 2022;219(1):152-156. - PubMed
    1. Pesapane F, Volonté C, Codari M, Sardanelli F.. Artificial intelligence as a medical device in radiology: ethical and regulatory issues in Europe and the United States. Insights Imaging. 2018;9(5):745-753. - PMC - PubMed

LinkOut - more resources