Mapping reliability in multicenter MRI: voxel-based morphometry and cortical thickness

Abstract

Multicenter structural MRI studies can have greater statistical power than single-center studies. However, across-center differences in contrast sensitivity, spatial uniformity, etc., may lead to tissue classification or image registration differences that could reduce or wholly offset the enhanced statistical power of multicenter data. Prior work has validated volumetric multicenter MRI, but robust methods for assessing reliability and power of multisite analyses with voxel-based morphometry (VBM) and cortical thickness measurement (CORT) are not yet available. We developed quantitative methods to investigate the reproducibility of VBM and CORT to detect group differences and estimate heritability when MRI scans from different scanners running different acquisition protocols in a multicenter setup are included. The method produces brain maps displaying information such as lowest detectable effect size (or heritability) and effective number of subjects in the multicenter study. We applied the method to a five-site multicenter calibration study using scanners from four different manufacturers, running different acquisition protocols. The reliability maps showed an overall good comparability between the sites, providing a reasonable gain in sensitivity in most parts of the brain. In large parts of the cerebrum and cortex scan pooling improved heritability estimates, with "effective-N" values upto the theoretical maximum. For some areas, "optimal-pool" maps indicated that leaving out a site would give better results. The reliability maps also reveal which brain regions are in any case difficult to measure reliably (e.g., around the thalamus). These tools will facilitate the design and analysis of multisite VBM and CORT studies for detecting group differences and estimating heritability.

Figure 1

Left: Uncertainty in the determination of h ${}_{\text{lim}}^{\text{2}}$ (pool), as a function of the number of calibration subjects n _c, for k‐site multicenter studies (k = 2, 3, 4, 5, 6, 8) with n _j MZ and DZ twin pairs per site, around h ${}_{\text{lim}}^{\text{2}}$ = 0.51. We kept = 160 for all values of k. The uncertainty was determined in a simulation experiment as the mean absolute difference between the experimentally determined values and the true, i.e., simulated, values of h ${}_{\text{lim}}^{\text{2}}$. The gray‐area symbols represent uncertainties in h ${}_{\text{lim}}^{\text{2}}$ for which b _j = 1 was set; they are only displayed if the uncertainty was smaller than those obtained with free b _j (for the same n _c; same symbol shape). Right: scaled uncertainties of h ${}_{\text{lim}}^{\text{2}}$. The uncertainties turn out to follow the same curve (drawn line) after scaling of n _c and k: Uncertainty ≈ 0.098(n _c√k − 3.69)^−2/3.

Figure 2

Transverse slices of the model brain showing the lowest detectable effect size h ${}_{\text{lim}}^{\text{2}}$ for the single sites (from top downward, U0, J, L, H), and the 4‐site multicenter pool (bottom, P) (left column; see Fig. 6 for this slice's anatomy). Distributions of voxelwise lowest detectable heritabilities for the whole cerebrum (second column). View of the left hemisphere of the cerebrum with vertexwise lowest detectable effect sizes (d _lim) (third column). Distributions of vertexwise lowest detectable effect sizes for the whole cerebral cortex (right column). Calculations were done with z _αβ = 3.939 (twin; one‐sided) and 4.132 (group; two‐sided), and n _j = 40 for all sites.

Figure 3

Top row: Transverse slice of the brain showing gain in lowest detectable heritability h ${}_{\text{lim}}^{\text{2}}$ (twin study) of the 4‐pool versus the average single site (left), and gain in lowest detectable effect size d _lim (group study, right). Negative values reflect a gain in sensitivity when pooling data from the four sites. Bottom row: Lateral view of left cerebral hemisphere showing gain in lowest detectable h ${}_{\text{lim}}^{\text{2}}$ (left) and gain in lowest detectable effect size d _lim (right).

Figure 4

Distribution of gain in lowest detectable heritability h ${}_{\text{lim}}^{\text{2}}$ (twin study): h ${}_{\text{lim}}^{\text{2}}$ (pool)—h ${}_{\text{lim}}^{\text{2}}$ (average single site) and lowest detectable effect size d _lim (group study): d _lim(pool)—d _lim(average single site). Negative values reflect a gain in sensitivity when pooling data from the four sites.

Figure 5

View of the left cerebral cortex showing lowest detectable thickness differences between two groups (patients and control subjects) for the multicenter 4‐pool. A logarithmic color scale is used.

Figure 6

Transverse slices of the model brain (left) showing the multicenter N _effs for a twin study (middle) and a disease effect (group) study (right).

Figure 7

Pie charts displaying the distribution of gain factors N _eff(pool)/N _eff (single site) (% voxels or vertices), for VBM (top row) and CORT (bottom row); twin (left column) and group (right column) studies. Gain factors are calculated voxel‐/vertexwise with respect to the average of the single sites. Undefined: all single site N _effs were zero (gray). Note that for heritability (twin) studies gain factors larger than 4 are possible. For group studies, this percentage refers to a gain factor of exactly 4 (red).

Figure 8

Transverse slice of the model brain showing the best combinations of sites for each voxel, for a multicenter twin study. Each color represents one of the five possible combinations of sites: all four sites, and four combinations leaving out one site at the time.