Inter-laboratory comparison of channelized hotelling observer computation

Abstract

Purpose: The task-based assessment of image quality using model observers is increasingly used for the assessment of different imaging modalities. However, the performance computation of model observers needs standardization as well as a well-established trust in its implementation methodology and uncertainty estimation. The purpose of this work was to determine the degree of equivalence of the channelized Hotelling observer performance and uncertainty estimation using an intercomparison exercise.

Materials and methods: Image samples to estimate model observer performance for detection tasks were generated from two-dimensional CT image slices of a uniform water phantom. A common set of images was sent to participating laboratories to perform and document the following tasks: (a) estimate the detectability index of a well-defined CHO and its uncertainty in three conditions involving different sized targets all at the same dose, and (b) apply this CHO to an image set where ground truth was unknown to participants (lower image dose). In addition, and on an optional basis, we asked the participating laboratories to (c) estimate the performance of real human observers from a psychophysical experiment of their choice. Each of the 13 participating laboratories was confidentially assigned a participant number and image sets could be downloaded through a secure server. Results were distributed with each participant recognizable by its number and then each laboratory was able to modify their results with justification as model observer calculation are not yet a routine and potentially error prone.

Results: Detectability index increased with signal size for all participants and was very consistent for 6 mm sized target while showing higher variability for 8 and 10 mm sized target. There was one order of magnitude between the lowest and the largest uncertainty estimation.

Conclusions: This intercomparison helped define the state of the art of model observer performance computation and with thirteen participants, reflects openness and trust within the medical imaging community. The performance of a CHO with explicitly defined channels and a relatively large number of test images was consistently estimated by all participants. In contrast, the paper demonstrates that there is no agreement on estimating the variance of detectability in the training and testing setting.

Keywords: channelized hotelling observer; computed tomography; image quality; intercomparison; model observers.

Figure 1

Cylindrical water tank phantom. Diameter: 20 cm; length: 25 cm. [Color figure can be viewed at wileyonlinelibrary.com]

Figure 2

200 × 200 pixel size ROIs for (a) 6 mm, (b) 8 mm and (c) 10 mm signal size. These images were obtained by increasing signal contrast for visualization purposes.

Figure 3

Detectability indexes for CHO D‐DOG with Dataset1 computed by each participant laboratories for (a) 6 mm, (b) 8 mm and (c) 10 mm signal size in increasing order. The dotted line represents the median value for final estimation of d′. For laboratories that corrected their estimation, the first estimation of d′ is plotted as a triangle marker. Error bars represent the 95% confidence interval for the mean d′. For the laboratories that provided standard uncertainties, the values were multiplied by a coverage factor k = 2 and are drawn as plus/minus this new value. [Color figure can be viewed at wileyonlinelibrary.com]

Figure 4

CHO D‐DOG with Dataset1 95% confidence interval length of the mean d′ computed by each participant laboratory for (a) 6 mm, (b) 8 mm and (c) 10 mm signal size in increasing order. For the laboratories that provided 1 standard‐deviation uncertainty, the values have been adjusted as described in the text. [Color figure can be viewed at wileyonlinelibrary.com]

Figure 5

Effect of the number of samples N used to train the CHO on d′ for independent and resubstitution sampling methods with 10 mm signal size. Error bars represent the exact 95% interval as defined by Wunderlich et al. The dotted lines are present to facilitate the reading of the graph. Courtesy of F. Samuelson and R. Zeng from FDA/CDRH. [Color figure can be viewed at wileyonlinelibrary.com]

Figure 6

CHO D‐DOG with Dataset2 d′ for 8 mm signal size in increasing order. The detectability index was computed by the exercise co‐ordinator from decision variable responses provided by each participant laboratory using the ground truth of the respecting Dataset. The detectability index was estimated as the distance between the mean signal present and absent distribution in sigma units. The dotted line represents the median value. Uncertainty estimates were computed by the co‐ordinator by bootstrapping the test cases from the decision variable responses provided by each participant with 1000 iterations. Errors bars represent two standard deviations from the bootstrapped d′ distribution. [Color figure can be viewed at wileyonlinelibrary.com]

Figure 7

Detectability indexes for human observers with Dataset1 for participating laboratories for (a) 6 mm, (b) 8 mm, and (c) 10 mm signal size. The dotted line represents the median value. To derive d′ and u(d′) from MAFC, the hit/miss values were bootstrapped. Averaged PC was converted to d′ and errors bars represent u(d′) as 2 standard deviations from the bootstrapped distribution. [Color figure can be viewed at wileyonlinelibrary.com]

Figure 8

Covariance matrices K in channels space estimated by each participant for 8 mm signal size. In this representation, the top left pixel is the variance associated to the output of the lowest frequency channel and the bottom right pixel corresponds to the output of the highest frequency channel. All the other pixels describe the inter‐class covariance. As the exercise used 10 channels D‐DOG, K is a 10‐by‐10 matrix with the following array format: $K=\left[\begin{array}{ccc}{K}_{1,1}& \cdots & K_{1,10}\\ ⋮& \ddots & ⋮\\ K_{10,1}& \cdots & K_{10,10}\end{array}\right].$ [Color figure can be viewed at wileyonlinelibrary.com]