Explainable AI: A review of applications to neuroimaging data

Abstract

Deep neural networks (DNNs) have transformed the field of computer vision and currently constitute some of the best models for representations learned via hierarchical processing in the human brain. In medical imaging, these models have shown human-level performance and even higher in the early diagnosis of a wide range of diseases. However, the goal is often not only to accurately predict group membership or diagnose but also to provide explanations that support the model decision in a context that a human can readily interpret. The limited transparency has hindered the adoption of DNN algorithms across many domains. Numerous explainable artificial intelligence (XAI) techniques have been developed to peer inside the "black box" and make sense of DNN models, taking somewhat divergent approaches. Here, we suggest that these methods may be considered in light of the interpretation goal, including functional or mechanistic interpretations, developing archetypal class instances, or assessing the relevance of certain features or mappings on a trained model in a post-hoc capacity. We then focus on reviewing recent applications of post-hoc relevance techniques as applied to neuroimaging data. Moreover, this article suggests a method for comparing the reliability of XAI methods, especially in deep neural networks, along with their advantages and pitfalls.

Keywords: artificial intelligence (AI); deep learning; explainable AI; interpretability; medical imaging; neural networks; neuroimaging.

Figure 1

(A) Explainable AI methods taxonomy. (B) Functional approaches attempt to disclose the algorithm's mechanistic aspects. (C) Archetypal approaches, like generative methods, seek to uncover input patterns that yield the best model response. (D) Post-hoc perturbation relevance approaches generally change the inputs or the model's components and then attributing relevance proportionally to the amount of the change in model output. (E) Post-hoc decomposition relevance approaches are propagation-based techniques explaining an algorithm's decisions by redistributing the function value (i.e., the neural network's output) to the input variables, often in a layer-by-layer fashion.

Figure 2

The flow diagram of the methodology and selection processes used in this systematic review follows the PRISMA statement (Moher et al., 2009).

Figure 3

Study characteristics. (A) Categorization of included studies, (B) XAI in medical imaging, and (C) a bubble plot that shows mentioned studies by type of XAI method, imaging modality, sample size, and publication trend in recent years.

Figure 4

Co-occurrence network of the commonly used words in reviewed studies.

Figure 5

Assessing the risk of bias using the Cochrane Collaboration's tool.

Figure 6

Requirement for interpretability in medical intelligent systems.