Assessment of reliability of multi-site neuroimaging via traveling phantom study

Abstract

This paper describes a framework for quantitative analysis of neuroimaging data of traveling human phantoms used for cross-site validation. We focus on the analysis of magnetic resonance image data including intra- and inter-site comparison. Locations and magnitude of geometric deformation is studied via unbiased atlas building and metrics on deformation fields. Variability of tissue segmentation is analyzed by comparison of volumes, overlap of tissue maps, and a new Kullback-Leibler divergence on tissue probabilities, with emphasis on comparing probabilistic rather than binary segmentations. We show that results from this information theoretic measure are highly correlated with overlap. Reproducibility of automatic, atlas-based segmentation of subcortical structures is examined by comparison of volumes, shape overlap and surface distances. Variability among scanners of the same type but also differences to a different scanner type are discussed. The results demonstrate excellent reliability across multiple sites that can be achieved by the use of the today's scanner generation and powerful automatic analysis software. Knowledge about such variability is crucial for study design and power analysis in new multi-site clinical studies.

Fig. 1

Example images and distributions of the L² norm and the log determinant of the Jacobian for the eight scans of one phantom.

Fig. 2

Quantitative evaluation of image deformation. Left: 90 percentiles for the L2 norm (in mm). Right: 5 and 95 percentiles for the log det Jacobian (log volume change).

Fig. 3

Left: Tissue volumes for all the scanners normalized relative to scanner 𝒜 averages for each phantom. Right: Coefficient of variation for tissue volumes.

Fig. 4

Probabilistic overlap measure POV for the tissue segmentation of each case compared to the tissue segmentation of the atlas. Right: KL divergence measure of reliability of tissue probability maps. Total D_KL and class-specific $D_{KL}^{(c)}$ for each scan are listed. Correlation between POV and D_KL is −0.992 for wm, gm and csf combined (see text).

Fig. 5

Segmentation of subcortical structures averaged over scanner type 𝒜 and both phantoms. Left: 3D display of binarized probabilistic template. Middle: Coefficient of variation for resulting volumes. Right: Overlap of binary segmentation.

Fig. 6

Reliability of segmentation of subcortical structures. Left: Probabilistic overlap coefficient. Right: Surface distances (in mm) per structure for scanners 𝒜 and ℬ relative to estimated truth.