Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2022 Feb 17;1(2):e0000016.
doi: 10.1371/journal.pdig.0000016. eCollection 2022 Feb.

To explain or not to explain?-Artificial intelligence explainability in clinical decision support systems

Julia Amann  1 , Dennis Vetter  2   3 , Stig Nikolaj Blomberg  4 , Helle Collatz Christensen  4 , Megan Coffee  5 , Sara Gerke  6 , Thomas K Gilbert  7 , Thilo Hagendorff  8 , Sune Holm  9 , Michelle Livne  10 , Andy Spezzatti  11 , Inga Strümke  12   13 , Roberto V Zicari  14   15 , Vince Istvan Madai  16   17   18 ; Z-Inspection initiative
Affiliations

To explain or not to explain?-Artificial intelligence explainability in clinical decision support systems

Julia Amann et al. PLOS Digit Health. .

Abstract

Explainability for artificial intelligence (AI) in medicine is a hotly debated topic. Our paper presents a review of the key arguments in favor and against explainability for AI-powered Clinical Decision Support System (CDSS) applied to a concrete use case, namely an AI-powered CDSS currently used in the emergency call setting to identify patients with life-threatening cardiac arrest. More specifically, we performed a normative analysis using socio-technical scenarios to provide a nuanced account of the role of explainability for CDSSs for the concrete use case, allowing for abstractions to a more general level. Our analysis focused on three layers: technical considerations, human factors, and the designated system role in decision-making. Our findings suggest that whether explainability can provide added value to CDSS depends on several key questions: technical feasibility, the level of validation in case of explainable algorithms, the characteristics of the context in which the system is implemented, the designated role in the decision-making process, and the key user group(s). Thus, each CDSS will require an individualized assessment of explainability needs and we provide an example of how such an assessment could look like in practice.

PubMed Disclaimer

Conflict of interest statement

VIM reported receiving personal fees from ai4medicine outside the submitted work. There is no connection, commercial exploitation, transfer or association between the projects of ai4medicine and the results presented in this work.

Figures

Fig 1
Fig 1. Terminology.
Given that there is no commonly accepted terminology, we defined the following terms for this work: When referring to the general concept of explaining machine learning models, we will call it explainability. This can be achieved either by using inherently interpretable algorithms (which provide interpretations) or by using a black-box algorithm and an additional explanation algorithm (which provides explanations). These interpretations or explanations are what the user is interacting with.
Fig 2
Fig 2. Z-Inspection® process.
This figure has been reproduced from Zicari RV, Brodersen J, Brusseau J, Düdder B, Eichhorn T, Ivanov T, et al. Z-Inspection®: A Process to Assess Trustworthy AI. IEEE Trans Technol Soc. 2021 Jun;2(2):83–97. [56].
Fig 3
Fig 3. Ethical issue identified during the initial assessment of the use case [31].
See this image and copyright information in PMC

References

    1. Kubben P, Dumontier M, Dekker A. Fundamentals of Clinical Data Science [Internet]. Cham: Springer International Publishing; 2019. [cited 2021 Mar 23]. Available from: http://link.springer.com/10.1007/978-3-319-99713-1 - DOI - PubMed
    1. Fauw JD, Ledsam JR, Romera-Paredes B, Nikolov S, Tomasev N, Blackwell S, et al.. Clinically applicable deep learning for diagnosis and referral in retinal disease. Nat Med. 2018. Sep;24(9):1342–50. doi: 10.1038/s41591-018-0107-6 - DOI - PubMed
    1. Liu X, Faes L, Kale AU, Wagner SK, Fu DJ, Bruynseels A, et al.. A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis. Lancet Digit Health. 2019. Oct 1;1(6):e271–97. doi: 10.1016/S2589-7500(19)30123-2 - DOI - PubMed
    1. Beede E, Baylor E, Hersch F, Iurchenko A, Wilcox L, Ruamviboonsuk P, et al. A Human-Centered Evaluation of a Deep Learning System Deployed in Clinics for the Detection of Diabetic Retinopathy. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems [Internet]. New York, NY, USA: Association for Computing Machinery; 2020 [cited 2021 May 7]. p. 1–12. (CHI ‘20). Available from: 10.1145/3313831.3376718 - DOI
    1. Ching T, Himmelstein DS, Beaulieu-Jones BK, Kalinin AA, Do BT, Way GP, et al.. Opportunities and obstacles for deep learning in biology and medicine. J R Soc Interface. 2018. Apr 30;15(141):20170387. doi: 10.1098/rsif.2017.0387 - DOI - PMC - PubMed

LinkOut - more resources