WindowSHAP: An efficient framework for explaining time-series classifiers based on Shapley values

Abstract

Unpacking and comprehending how black-box machine learning algorithms (such as deep learning models) make decisions has been a persistent challenge for researchers and end-users. Explaining time-series predictive models is useful for clinical applications with high stakes to understand the behavior of prediction models, e.g., to determine how different variables and time points influence the clinical outcome. However, existing approaches to explain such models are frequently unique to architectures and data where the features do not have a time-varying component. In this paper, we introduce WindowSHAP, a model-agnostic framework for explaining time-series classifiers using Shapley values. We intend for WindowSHAP to mitigate the computational complexity of calculating Shapley values for long time-series data as well as improve the quality of explanations. WindowSHAP is based on partitioning a sequence into time windows. Under this framework, we present three distinct algorithms of Stationary, Sliding and Dynamic WindowSHAP, each evaluated against baseline approaches, KernelSHAP and TimeSHAP, using perturbation and sequence analyses metrics. We applied our framework to clinical time-series data from both a specialized clinical domain (Traumatic Brain Injury - TBI) as well as a broad clinical domain (critical care medicine). The experimental results demonstrate that, based on the two quantitative metrics, our framework is superior at explaining clinical time-series classifiers, while also reducing the complexity of computations. We show that for time-series data with 120 time steps (hours), merging 10 adjacent time points can reduce the CPU time of WindowSHAP by 80 % compared to KernelSHAP. We also show that our Dynamic WindowSHAP algorithm focuses more on the most important time steps and provides more understandable explanations. As a result, WindowSHAP not only accelerates the calculation of Shapley values for time-series data, but also delivers more understandable explanations with higher quality.

Keywords: Explainable artificial intelligence; Model interpretation; Shapley value; Time-series data.

Figure 1.

Conceptual Demonstration of KernelSHAP vs WindowSHAP for a classification model for an individual instance, predicting whether there is an anomaly in a synthetically generated sequence. The top picture shows the sequence and its Shapley values derived from the KernelSHAP while the bottom depicts the Shapley values from WindowSHAP. While KernelSHAP is spreading the Shapley values all over the sequence, our approach focuses more on the part of the sequence that is more important, avoiding calculating Shapley values for each single time step.

Figure 2.

A visualization of time-axis partitioning in the Stationary WindowSHAP algorithm. The windows are non-overlapping, contiguous, and of the same length, except possibly the last window being smaller.

Figure 3.

Demonstration of Sliding WindowSHAP: (a) depicts a single iteration in which the entire sequence is divided into two time intervals, inside and outside of the time window. (b) shows the final windowing result after all iterations have been completed and a Shapley value has been produced for each time window.

Figure 4.

Demonstration of Dynamic WindowSHAP algorithm for a sequence. The algorithm stops in the fourth iteration because all the Shapley values for time windows are less than the threshold $\delta$

Figure 5.

Evaluation metrics for all explanation algorithms. Each row of figures shows the result for one of the prediction models. The x axis in all figures represents the percentile p that is used in the metrics definitions. The $\text{y}$ axis represents the change in the quality metric after perturbing the most crucial time points. Error bars are shown as the mean ± standard errors of the mean of binary loss function.

Figure 6.

Visualization of RAM usage and CPU time of different algorithms under WindowSHAP framework. Columns (a), (b), and (c) represent Stationary, Sliding, and Dynamic WindowSHAP algorithms respectively.

Figure 7.

Heatmaps depicting the importance of all time steps for the important features for a certain patient record from the MIMIC-III dataset. The top 15 variables depicted on the y axis are ranked according to their importance. The darker the color is, the higher the absolute value of the assigned Shapley value is.

Figure 8.

The explanations of the heart rate variable for a patient in MIMIC mortality prediction model. The left and right plots represent visual explanations of WindowSHAP and KernelSHAP, respectively.