Limitations of Out-of-Distribution Detection in 3D Medical Image Segmentation

Abstract

Deep learning models perform unreliably when the data come from a distribution different from the training one. In critical applications such as medical imaging, out-of-distribution (OOD) detection methods help to identify such data samples, preventing erroneous predictions. In this paper, we further investigate OOD detection effectiveness when applied to 3D medical image segmentation. We designed several OOD challenges representing clinically occurring cases and found that none of the methods achieved acceptable performance. Methods not dedicated to segmentation severely failed to perform in the designed setups; the best mean false-positive rate at a 95% true-positive rate (FPR) was 0.59. Segmentation-dedicated methods still achieved suboptimal performance, with the best mean FPR being 0.31 (lower is better). To indicate this suboptimality, we developed a simple method called Intensity Histogram Features (IHF), which performed comparably or better in the same challenges, with a mean FPR of 0.25. Our findings highlight the limitations of the existing OOD detection methods with 3D medical images and present a promising avenue for improving them. To facilitate research in this area, we release the designed challenges as a publicly available benchmark and formulate practical criteria to test the generalization of OOD detection beyond the suggested benchmark. We also propose IHF as a solid baseline to contest emerging methods.

Keywords: anomaly detection; computed tomography; magnetic resonance imaging; out-of-distribution detection; segmentation.

Figure 1

Examples of CT images (representative axial slices) from different simulated OOD sources in our benchmark.

Figure 2

Examples of MRI images (representative axial slices) from different simulated OOD sources in our benchmark.

Figure 3

The proposed OOD detection method, called Intensity Histogram Features (IHF). It consists of three steps: calculating a m-dimensional vector as histogram bin values from the preprocessed image (Step 1), fitting and applying PCA to the occuring data, and calculating the Mahalanobis distance between a test vector and ID sample distribution (Step 3). We apply IHF to the 3D images and illustrate the process using 2D axial slices for simplicity. (* PCA is fitted once on all training data.)

Figure 4

Dependence of IHF on its two hyperparameters: the number of histogram bins (m) and explained variance in PCA (v). We give the results for both IHF variants and CT and MRI setups.

Figure 5

FPR under synthetically distorted data for every distortion severity level. Blue line indicates method’s average trend across presented challenges with 95% confidence interval. The other UE methods (MCD, Ensemble and G-ODIN) are excluded since their average trend is similar to Entropy.