Lack of agreement between radiologists: implications for image-based model observers

Abstract

We tested the agreement of radiologists' rankings of different reconstructions of breast computed tomography images based on their diagnostic (classification) performance and on their subjective image quality assessments. We used 102 pathology proven cases (62 malignant, 40 benign), and an iterative image reconstruction (IIR) algorithm to obtain 24 reconstructions per case with different image appearances. Using image feature analysis, we selected 3 IIRs and 1 clinical reconstruction and 50 lesions. The reconstructions produced a range of image quality from smooth/low-noise to sharp/high-noise, which had a range in classifier performance corresponding to AUCs of 0.62 to 0.96. Six experienced Mammography Quality Standards Act (MQSA) radiologists rated the likelihood of malignancy for each lesion. We conducted an additional reader study with the same radiologists and a subset of 30 lesions. Radiologists ranked each reconstruction according to their preference. There was disagreement among the six radiologists on which reconstruction produced images with the highest diagnostic content, but they preferred the midsharp/noise image appearance over the others. However, the reconstruction they preferred most did not match with their performance. Due to these disagreements, it may be difficult to develop a single image-based model observer that is representative of a population of radiologists for this particular imaging task.

Keywords: breast cancer; breast computed tomography; diagnostic performance; model observers; reader study.

Fig. 1

The IIR algorithm reconstructed images with different image appearances. (a) An image reconstructed by f TV-LSQ $(v1,v2)$. This reconstruction maintains the gray-scale information. (b) An image reconstructed by f TV-δLSQ $(v3,v4)$. This reconstruction provides the edge information. (c)–(e) Final reconstructed images obtained by combining two reconstructions (a) and (b) with different weights ($\mathrm{c}=1$, $\mathrm{d}=3$, and $\mathrm{e}=5$).

Fig. 2

The scatter plots of (a) the image noise versus image sharpness and (b) the image sharpness versus the AUC of trained classifiers for all reconstructions that were considered in this study. To compute AUC values, we trained and tested linear discriminant analysis (LDA) classifiers using the quantitative image features extracted from the segmented breast lesions, and then conducted ROC analysis. Section 2.3 explains the details of the ROC analysis on the trained LDA classifiers. The four reconstructions that were selected for the reader study were highlighted with the large markers.

Fig. 3

The layout of the viewer used for the reader study 1. Radiologists were able to review entire breast volume by dynamically moving through slices in three different cross-sectional views. Target lesion was highlighted and centered in the viewer.

Fig. 4

The layout of the viewer used for the second reader study. Each cross-sectional view of four selected reconstructions was grouped. Target lesion was highlighted. For this reader study, radiologists reviewed the center slice of each view of four selected reconstructions and ranked them in terms of which reconstruction provided the best diagnostic information (or simply their preference of one reconstruction algorithm over others).