Quantitative image quality evaluation of MR images using perceptual difference models

Abstract

The authors are using a perceptual difference model (Case-PDM) to quantitatively evaluate image quality of the thousands of test images which can be created when optimizing fast magnetic resonance (MR) imaging strategies and reconstruction techniques. In this validation study, they compared human evaluation of MR images from multiple organs and from multiple image reconstruction algorithms to Case-PDM and similar models. The authors found that Case-PDM compared very favorably to human observers in double-stimulus continuous-quality scale and functional measurement theory studies over a large range of image quality. The Case-PDM threshold for nonperceptible differences in a 2-alternative forced choice study varied with the type of image under study, but was approximately 1.1 for diffuse image effects, providing a rule of thumb. Ordering the image quality evaluation models, we found in overall Case-PDM approximately IDM (Sarnoff Corporation) approximately SSIM [Wang et al. IEEE Trans. Image Process. 13, 600-612 (2004)] > mean squared error NR [Wang et al. (2004) (unpublished)] > DCTune (NASA) > IQM (MITRE Corporation). The authors conclude that Case-PDM is very useful in MR image evaluation but that one should probably restrict studies to similar images and similar processing, normally not a limitation in image reconstruction studies.

Figure 1

Block diagram of the perceptual difference model (Case-PDM) is shown in (a). The inputs of the model are two images, a reference image (b) and a test image (c). The output is a spatial map (d) showing the perceived difference between two images. PDM could be used to tell the visual difference between two input images, as shown in the overlaid display in (e).

Figure 2

Raw images used for human observer experiments. Of them, (a)–(c) are for the DSCQS experiment; (d) and (e) are for the FMT experiment; and (b) and (f) are for the 2-AFC experiment.

Figure 3

Experiment to evaluate correlation to human subject. Data are from SENSE (a), SPIRAL (b), and GRAPPA (c) reconstructions in the DSCQS experiment with correlation coefficients of 0.94, 0.97, and 0.91 correspondingly. Cross points represent responses from radiologists, open square and triangle points represent responses from image engineers, and solid circle points represent average responses. The average human subject data (solid circle) were fitted to y=ax+b, and the functions were represented by the straight lines in the figures.

Figure 4

Image quality evaluation with different raw brain images (brain slices 1 and 2) identical processing (GRAPPA reconstruction). Difference-in-quality scores, averaged over all subjects were shown in (a). The numbers 1–6 and 7–12 on the x axis correspond to brain slice 1 and brain slice 2, respectively. Each point on the curves indicates the score for the difference-in-quality between the corresponding row and column stimuli in the 12×12 stimulus matrix. Averaged subjective quality curves obtained by using Anderson’s FMT are shown in (b). Error bars represent standard deviations.

Figure 5

Image quality evaluation with identical raw brain images (brain slice 1) different processing (GRAPPA and WGRAPPA reconstructions). Difference-in-quality scores, averaged over all subjects were shown in (a). The numbers 1–6 and 7–12 on the x-axis correspond to GRAPPA reconstruction method and WGRAPPA reconstruction method, respectively. Each point on the curves indicates the score for the difference-in-quality between the corresponding row and column stimuli in the 12×12 stimulus matrix. Averaged subjective quality curves obtained by using Anderson’s FMT are shown in (b). Error bars represent standard deviations.

Figure 6

One result from the 2-AFC experiment. Reconstruction artifacts were included into a brain MR image, and the relationship between d^′ and Case-PDM predictions is shown. One could observe that Case-PDM scores show good correlation with d^′, but the relationship is not linear. The threshold value was represented with a star in the plot.