In situ study of the impact of inter- and intra-reader variability on region of interest (ROI) analysis in preclinical molecular imaging

Abstract

We estimated reader-dependent variability of region of interest (ROI) analysis and evaluated its impact on preclinical quantitative molecular imaging. To estimate reader variability, we used five independent image datasets acquired each using microPET and multispectral fluorescence imaging (MSFI). We also selected ten experienced researchers who utilize molecular imaging in the same environment that they typically perform their own studies. Nine investigators blinded to the data type completed the ROI analysis by drawing ROIs manually that delineate the tumor regions to the best of their knowledge and repeated the measurements three times, non-consecutively. Extracted mean intensities of voxels within each ROI are used to compute the coefficient of variation (CV) and characterize the inter- and intra-reader variability. The impact of variability was assessed through random samples iterated from normal distributions for control and experimental groups on hypothesis testing and computing statistical power by varying subject size, measured difference between groups and CV. The results indicate that inter-reader variability was 22.5% for microPET and 72.2% for MSFI. Additionally, mean intra-reader variability was 10.1% for microPET and 26.4% for MSFI. Repeated statistical testing showed that a total variability of CV < 50% may be needed to detect differences < 50% between experimental and control groups when six subjects (n = 6) or more are used and statistical power is adequate (80%). Surprisingly high variability has been observed mainly due to differences in the ROI placement and geometry drawn between readers, which may adversely affect statistical power and erroneously lead to negative study outcomes.

Keywords: Molecular imaging; microPET; multispectral fluorescence imaging; preclinical; region of interest analysis; variability.

Figure 1

Numerous sources of variability in typical pre-clinical molecular imaging, where the σ’s are the variation (standard deviations) from each step in the analytic pipeline with respect to the measured mean value (μ).

Figure 3

Inter- and intra-reader variability study performed using image data acquired from MSFI. Left top, selected raw mixed image data, Left bottom, Composite unmixed image sample with tumor (rainbow) and tissue auto fluorescence (grey). Right, comparison of extracted mean ROI value of each reader for the randomly selected five data sets.

Figure 2

Estimate of inter- and intra-reader variability study using microPET image data. Left, a sample of selected microPET image acquired at our imaging facility for other projects. Right, comparison of extracted mean ROI value showing significant variation with nearly consistent pattern of either under- or over-estimation of mean ROI value of respective reader.

Figure 4

Computed probability of statistically significant biological response under the alternative hypothesis (i.e., statistical power) as function of CV assuming different mean relevant difference in biological response with n=6 (left) and different number of subjects with diff=50% (middle). Right, Illustrating combined variability effects due to biological factors and image analysis with n=6 and diff=50%.