DeepAMO: a multi-slice, multi-view anthropomorphic model observer for visual detection tasks performed on volume images

Abstract

Purpose: We propose a deep learning-based anthropomorphic model observer (DeepAMO) for image quality evaluation of multi-orientation, multi-slice image sets with respect to a clinically realistic 3D defect detection task. Approach: The DeepAMO is developed based on a hypothetical model of the decision process of a human reader performing a detection task using a 3D volume. The DeepAMO is comprised of three sequential stages: defect segmentation, defect confirmation (DC), and rating value inference. The input to the DeepAMO is a composite image, typical of that used to view 3D volumes in clinical practice. The output is a rating value designed to reproduce a human observer's defect detection performance. In stages 2 and 3, we propose: (1) a projection-based DC block that confirms defect presence in two 2D orthogonal orientations and (2) a calibration method that "learns" the mapping from the features of stage 2 to the distribution of observer ratings from the human observer rating data (thus modeling inter- or intraobserver variability) using a mixture density network. We implemented and evaluated the DeepAMO in the context of ${}^{99m}\mathrm{Tc}$ -DMSA SPECT imaging. A human observer study was conducted, with two medical imaging physics graduate students serving as observers. A $5\times 2$ -fold cross-validation experiment was conducted to test the statistical equivalence in defect detection performance between the DeepAMO and the human observer. We also compared the performance of the DeepAMO to an unoptimized implementation of a scanning linear discriminant observer (SLDO). Results: The results show that the DeepAMO's and human observer's performances on unseen images were statistically equivalent with a margin of difference ( $\Delta \mathrm{AUC}$ ) of 0.0426 at $p<0.05$ , using 288 training images. A limited implementation of an SLDO had a substantially higher AUC (0.99) compared to the DeepAMO and human observer. Conclusion: The results show that the DeepAMO has the potential to reproduce the absolute performance, and not just the relative ranking of human observers on a clinically realistic defect detection task, and that building conceptual components of the human reading process into deep learning-based models can allow training of these models in settings where limited training images are available.

Keywords: deep learning; model observer; task-based image quality assessment.

Fig. 1

A sample 48-slice image shown in the volumetric display format routinely used in clinical practice at BCH. The red arrow indicates the location of the functional defect.

Fig. 2

A schematic of the proposed model observer, DeepAMO. $I$ is the multi-slice, multi-view input image, $T_k^j$ is the triad, where $k\in (c,s,t)$ represents the slicing direction and $j\in [1,N-1]$, where $N$ is the number of slices in each orientation. ${\mathrm{SM}}_k^j$ is the output segmentation mask for each triad $T_k^j$. ${\mathrm{TVD}}_k$ is the TVD seen in each slicing direction computed by summing the corresponding ${\mathrm{SSM}}_k$. ${\mathrm{SSM}}_k$ is the summed segmentation mask along each slicing direction $k$. ${\mathrm{HP}}_k$ and ${\mathrm{VP}}_k$ are horizontal and VP of the corresponding ${\mathrm{SSM}}_k$. ${\mathrm{DC}}_{\mathrm{cs}}$, ${\mathrm{DC}}_{\mathrm{ct}}$, and ${\mathrm{DC}}_{\mathrm{st}}$ are the three defect confirmation scalars from the defect confirmation network. Note that one triad is fed to the segmentation at a time.

Fig. 3

An illustration of the process of confirming the defect from different views using projection and dot product in 3D space.

Fig. 4

Segmentation network architecture used in this study.

Fig. 5

A sample image of the GUI used in the human observer study for DeepAMO.

Fig. 6

A pictorial illustration of the rejectable and unrejectable case in equivalence hypothesis testing.

Fig. 7

(a), (b) The defect-present and defect-absent composite image at two different randomly sampled defect locations, respectively. The red arrows mark the exact location of the defect inside each slice.

Fig. 8

Images of the seven anthropomorphic DOM channels used in this work. (a) The frequency channels and (b) the spatial domain templates. From left to right, the start frequencies and widths of the channels were 0.5, 1, 2, 4, 8, 16, and $32\text{cycles}/\text{pixel}$. The spatial templates are the analytic inverse Fourier transforms of the frequency channels sampled at the image pixel size.

Fig. 9

Plots of histograms of the rating values of the simulated feature vectors (test data only) and predicted rating values on these data given by the DeepAMO. The plots show the class 0 and 1 (defect present and absent, respectively) as well as the calculated AUC value.

Fig. 10

Histograms of predicted rating values given by DeepAMO on unseen human observer data from the third trial of the $5\times 2$-fold cross validation experiment (other trials have similar patterns). Note that multiple predicted rating values were generated for each test image during testing of the DeepAMO to reduce sampling error. The histograms of the other half of human observer data used for training the DeepAMO are not shown in the plot.