Multisite reliability of MR-based functional connectivity

Abstract

Recent years have witnessed an increasing number of multisite MRI functional connectivity (fcMRI) studies. While multisite studies provide an efficient way to accelerate data collection and increase sample sizes, especially for rare clinical populations, any effects of site or MRI scanner could ultimately limit power and weaken results. Little data exists on the stability of functional connectivity measurements across sites and sessions. In this study, we assess the influence of site and session on resting state functional connectivity measurements in a healthy cohort of traveling subjects (8 subjects scanned twice at each of 8 sites) scanned as part of the North American Prodrome Longitudinal Study (NAPLS). Reliability was investigated in three types of connectivity analyses: (1) seed-based connectivity with posterior cingulate cortex (PCC), right motor cortex (RMC), and left thalamus (LT) as seeds; (2) the intrinsic connectivity distribution (ICD), a voxel-wise connectivity measure; and (3) matrix connectivity, a whole-brain, atlas-based approach to assessing connectivity between nodes. Contributions to variability in connectivity due to subject, site, and day-of-scan were quantified and used to assess between-session (test-retest) reliability in accordance with Generalizability Theory. Overall, no major site, scanner manufacturer, or day-of-scan effects were found for the univariate connectivity analyses; instead, subject effects dominated relative to the other measured factors. However, summaries of voxel-wise connectivity were found to be sensitive to site and scanner manufacturer effects. For all connectivity measures, although subject variance was three times the site variance, the residual represented 60-80% of the variance, indicating that connectivity differed greatly from scan to scan independent of any of the measured factors (i.e., subject, site, and day-of-scan). Thus, for a single 5min scan, reliability across connectivity measures was poor (ICC=0.07-0.17), but increased with increasing scan duration (ICC=0.21-0.36 at 25min). The limited effects of site and scanner manufacturer support the use of multisite studies, such as NAPLS, as a viable means of collecting data on rare populations and increasing power in univariate functional connectivity studies. However, the results indicate that aggregation of fcMRI data across longer scan durations is necessary to increase the reliability of connectivity estimates at the single-subject level.

Figure 1

Map of edges showing significant effects (p<0.05, FDR-corrected) on seed-based connectivity for each individual site, subject, and scanner manufacturer regressor. No day effects were found. Only one case is shown for scanner manufacturer because GLM estimates are identical for each regressor when there are only two regressors. Contrasts were made between individual regressors (e.g., subject 1) and the Figure 1 grand mean of that group of regressors (e.g., all subjects). For each group of regressors, brighter colors represent edges affected by multiple cases (e.g., for the subject group, an orange edge indicates that the contrast was significant in that edge for four out of eight subjects)

Figure 2

Decision Study violin plots showing the distribution of G-coefficients for seed-based connectivity obtained from increasing amounts of data. The x-axis reflects the number of days over which data is averaged. The mean (diamond) and standard deviation (bars) are shown. Results categorized as follows: poor<0.4, fair=0.4–0.59, good=0.6–0.74, excellent>0.74 (Cicchetti and Sparrow, 1981).

Figure 3

Map of voxels showing significant effects (p<0.05, FDR-corrected) on ICD for each individual site, subject, and scanner manufacturer regressor. No day effects were found. Only one case is shown for scanner manufacturer because GLM estimates are identical for each regressor when there are only two regressors. Contrasts were made between individual regressors (e.g., subject 1) and the grand mean of that group of regressors (e.g., all subjects). For each group of regressors, brighter colors represent voxels affected by multiple cases (e.g., for the subject group, an orange edge indicates that the contrast was significant in that edge for four out of eight subjects).

Figure 4

Decision Study violin plots showing the distribution of G-coefficients for ICD obtained from increasing amounts of data. The x-axis reflects the number of days over which data is averaged. The mean (diamond) and standard deviation (bars) are shown. Results categorized as follows: poor<0.4, fair=0.4–0.59, good=0.6–0.74, excellent>0.74 (Cicchetti and Sparrow, 1981).

Figure 5

Summary map of inter-lobe edges showing significant effects (p<0.05, FDR-corrected) on matrix connectivity for each individual site, subject, and scanner manufacturer regressor. No day effects were found. Only one case is shown for scanner manufacturer because GLM estimates are identical for each regressor when there are only two regressors. These maps correspond with Figures 1 and 3, but are summarized for visualization purposes. 278 regions are organized into 10 roughly anterior-to-posterior lobes: prefrontal cortex (PFC), motor cortex (Mot), insula (Ins), parietal cortex (Par), temporal cortex (Tmp), occipital cortex (Occ), limbic system (Lmb), cerebellum (Cbl), subcortex (Sub), and brainstem (Bst). A single inter-lobe edge in the summary map represents the mean number of affected cases for all edges between the two lobes. For example, the inter-lobe edge between right and left motor cortex under the “3–4 subjects different” heading indicates that, on average, edges between right and left motor cortex are unique to 3–4 subjects. Brighter (more yellow) colors also represent inter-lobe edges affected by multiple cases.

Figure 6

Decision Study violin plots showing the distribution of G-coefficients for matrix connectivity obtained from increasing amounts of data. The x-axis reflects the number of days over which data is averaged. The mean (diamond) and standard deviation (bars) are shown. Results categorized as follows: poor<0.4, fair=0.4–0.59, good=0.6–0.74, excellent>0.74 (Cicchetti and Sparrow, 1981).