Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2020 Jan;7(1):016502.
doi: 10.1117/1.JMI.7.1.016502. Epub 2020 Feb 11.

Integrating AI into radiology workflow: levels of research, production, and feedback maturity

Engin Dikici  1 , Matthew Bigelow  1 , Luciano M Prevedello  1 , Richard D White  1 , Barbaros S Erdal  1
Affiliations

Integrating AI into radiology workflow: levels of research, production, and feedback maturity

Engin Dikici et al. J Med Imaging (Bellingham). 2020 Jan.

Abstract

We present a roadmap for integrating artificial intelligence (AI)-based image analysis algorithms into existing radiology workflows such that (1) radiologists can significantly benefit from enhanced automation in various imaging tasks due to AI, and (2) radiologists' feedback is utilized to further improve the AI application. This is achieved by establishing three maturity levels where (1) research enables the visualization of AI-based results/annotations by radiologists without generating new patient records; (2) production allows the AI-based system to generate results stored in an institution's picture-archiving and communication system; and (3) feedback equips radiologists with tools for editing the AI inference results for periodic retraining of the deployed AI systems, thereby allowing continuous organic improvement of AI-based radiology-workflow solutions. A case study (i.e., detection of brain metastases with T1-weighted contrast-enhanced three-dimensional MRI) illustrates the deployment details of a particular AI-based application according to the aforementioned maturity levels. It is shown that the given AI application significantly improves with feedback coming from radiologists; the number of incorrectly detected brain metastases (false positives) decreases from 14.2 to 9.12 per patient with the number of subsequently annotated datasets increasing from 93 to 217 as a result of radiologist adjudication.

Keywords: AI-based image analysis; digital imaging and communications in medicine; picture archiving and communication system; radiology workflow.

PubMed Disclaimer

Figures

Fig. 1
Fig. 1
A simplified view of an example radiology workflow.
Fig. 2
Fig. 2
Research workflow.
Fig. 3
Fig. 3
Production workflow.
Fig. 4
Fig. 4
Feedback workflow.
Fig. 5
Fig. 5
DICOM-transfer to the AI system. (a) DICOM-transfer can be utilized to send the MRI series. (b) The final destination should be the DICOM node where the AI system is located.
Fig. 6
Fig. 6
Switching from PACS to the advanced viewer. (a) From the PACS, the neuroradiologist switches to the advanced viewer with the click of a button located at the user toolbar (shown with the yellow arrow). (b) A separate viewer overlays the result, GSPS object, on the medical image (GSPS circle overlays are pointed with red arrows in the figure).
Fig. 7
Fig. 7
Viewing the AI results from PACS workspace.
Fig. 8
Fig. 8
ZFP medical-image viewer showing AI results.
Fig. 9
Fig. 9
AFP for datasets. Average number of false-positives per patient (i.e., incorrectly detected BM lesions for each patient) in relation to the sensitivity is illustrated for each CV fold and the mean of all folds (black curve) for (a) Dataset01, (b) Dataset02, and (c) Dataset03. (d) The mean AFP curves of all datasets are shown together.
Fig. 10
Fig. 10
AFP versus sensitivity.
See this image and copyright information in PMC

References

    1. Sharma N., Aggarwal L. M., “Automated medical image segmentation techniques,” J. Med. Phys./Assoc. Med. Phys. India 35(1), 3 (2010). 10.4103/0971-6203.58777 - DOI - PMC - PubMed
    1. Wyawahare M. V, et al. , “Image registration techniques: an overview,” Int. J. Signal Process. Image Process. Pattern Recognit. 2(3), 11–28 (2009).
    1. Deepa S. N., Devi B. A., “A survey on artificial intelligence approaches for medical image classification,” Indian J. Sci. Technol. 4(11), 1583–1595 (2011). 10.17485/ijst/2011/v4i11/30291 - DOI
    1. Jiang F., et al. , “Artificial intelligence in healthcare: past, present and future,” Stroke Vasc. Neurol. 2(4), 230–243 (2017). 10.1136/svn-2017-000101 - DOI - PMC - PubMed
    1. Ramesh A. N., et al. , “Artificial intelligence in medicine,” Ann. R. Coll. Surg. Engl. 86(5), 334–338 (2004). 10.1308/147870804290 - DOI - PMC - PubMed