Explainable artificial intelligence models using real-world electronic health record data: a systematic scoping review

Abstract

Objective: To conduct a systematic scoping review of explainable artificial intelligence (XAI) models that use real-world electronic health record data, categorize these techniques according to different biomedical applications, identify gaps of current studies, and suggest future research directions.

Materials and methods: We searched MEDLINE, IEEE Xplore, and the Association for Computing Machinery (ACM) Digital Library to identify relevant papers published between January 1, 2009 and May 1, 2019. We summarized these studies based on the year of publication, prediction tasks, machine learning algorithm, dataset(s) used to build the models, the scope, category, and evaluation of the XAI methods. We further assessed the reproducibility of the studies in terms of the availability of data and code and discussed open issues and challenges.

Results: Forty-two articles were included in this review. We reported the research trend and most-studied diseases. We grouped XAI methods into 5 categories: knowledge distillation and rule extraction (N = 13), intrinsically interpretable models (N = 9), data dimensionality reduction (N = 8), attention mechanism (N = 7), and feature interaction and importance (N = 5).

Discussion: XAI evaluation is an open issue that requires a deeper focus in the case of medical applications. We also discuss the importance of reproducibility of research work in this field, as well as the challenges and opportunities of XAI from 2 medical professionals' point of view.

Conclusion: Based on our review, we found that XAI evaluation in medicine has not been adequately and formally practiced. Reproducibility remains a critical concern. Ample opportunities exist to advance XAI research in medicine.

Keywords: Explainable artificial intelligence (XAI); deep learning; electronic health records; interpretable machine learning; real-world data.

Figure 1.

The PRISMA diagram depicts the number of records identified, included and excluded, and the reasons for exclusions.

Figure 2.

Publication trend of XAI studies using EHR data between January 1st, 2009 and May 1st, 2019.

Figure 3.

Explainable artificial intelligence (XAI) methods vs machine learning (ML) methods used for interpretability enhancement. The links between ML and XAI methods illustrate that ML method was used with that specific XAI method in a paper. The thicker the links are, the more frequent that combination of ML and XAI method has been practiced. A thicker box around the XAI method shows that it has been applied by more of the articles.