How subjective CT image quality assessment becomes surprisingly reliable: pairwise comparisons instead of Likert scale

Abstract

Objectives: The aim of this study is to improve the reliability of subjective IQ assessment using a pairwise comparison (PC) method instead of a Likert scale method in abdominal CT scans.

Methods: Abdominal CT scans (single-center) were retrospectively selected between September 2019 and February 2020 in a prior study. Sample variance in IQ was obtained by adding artificial noise using dedicated reconstruction software, including reconstructions with filtered backprojection and varying iterative reconstruction strengths. Two datasets (each n = 50) were composed with either higher or lower IQ variation with the 25 original scans being part of both datasets. Using in-house developed software, six observers (five radiologists, one resident) rated both datasets via both the PC method (forcing observers to choose preferred scans out of pairs of scans resulting in a ranking) and a 5-point Likert scale. The PC method was optimized using a sorting algorithm to minimize necessary comparisons. The inter- and intraobserver agreements were assessed for both methods with the intraclass correlation coefficient (ICC).

Results: Twenty-five patients (mean age 61 years ± 15.5; 56% men) were evaluated. The ICC for interobserver agreement for the high-variation dataset increased from 0.665 (95%CI 0.396-0.814) to 0.785 (95%CI 0.676-0.867) when the PC method was used instead of a Likert scale. For the low-variation dataset, the ICC increased from 0.276 (95%CI 0.034-0.500) to 0.562 (95%CI 0.337-0.729). Intraobserver agreement increased for four out of six observers.

Conclusion: The PC method is more reliable for subjective IQ assessment indicated by improved inter- and intraobserver agreement.

Clinical relevance statement: This study shows that the pairwise comparison method is a more reliable method for subjective image quality assessment. Improved reliability is of key importance for optimization studies, validation of automatic image quality assessment algorithms, and training of AI algorithms.

Key points: • Subjective assessment of diagnostic image quality via Likert scale has limited reliability. • A pairwise comparison method improves the inter- and intraobserver agreement. • The pairwise comparison method is more reliable for CT optimization studies.

Keywords: Computed tomography (X-ray); Interobserver variability; Intraobserver variability.

Fig. 1

Flow diagram with selection of the high- and low-variation datasets

Fig. 2

User interface of the in-house developed software for (a) the pairwise comparison method; b the Likert scale method. c Examples of 3 CT images ranked by an observer using the pairwise comparison method with corresponding Likert scores

Fig. 3

Boxplots representing the spread in ranks of the image quality assessment by all six observers using the pairwise comparison method. The boxes are sorted by the median of the six ranks received by the observers and colored by the median Likert score given by the observers. Results are given for the (a) high- and (b) low-variation datasets

Fig. 4

Confusion matrices of random combinations of two observers for the Likert scores of the high-variation dataset (a–d) and low-variation dataset (e–h). The horizontal and vertical axes give the Likert scores given by the observers. A diagonal matrix would indicate complete agreement

Fig. 5

The agreement between two assessments using the pairwise comparison method for every single observer. For each scan, two dots are shown corresponding to the ranking in assessments 1 and 2. The color represents the mean Likert score given by the observer in the two assessments. The scans are sorted by the median rank. As an example, for observer 5, nearly every scan received a Likert score 4 (purple), despite the images consistently having distinct rankings in both assessments

Fig. 6

Confusion matrices of the repeated assessment for every single observer (intraobserver agreement). The horizontal (upper) and vertical axes give the Likert scores given by the observers. A diagonal matrix would indicate total agreement