Reliability of functional magnetic resonance imaging activation during working memory in a multi-site study: analysis from the North American Prodrome Longitudinal Study

Abstract

Multi-site neuroimaging studies offer an efficient means to study brain functioning in large samples of individuals with rare conditions; however, they present new challenges given that aggregating data across sites introduces additional variability into measures of interest. Assessing the reliability of brain activation across study sites and comparing statistical methods for pooling functional data are critical to ensuring the validity of aggregating data across sites. The current study used two samples of healthy individuals to assess the feasibility and reliability of aggregating multi-site functional magnetic resonance imaging (fMRI) data from a Sternberg-style verbal working memory task. Participants were recruited as part of the North American Prodrome Longitudinal Study (NAPLS), which comprises eight fMRI scanning sites across the United States and Canada. In the first study sample (n=8), one participant from each home site traveled to each of the sites and was scanned while completing the task on two consecutive days. Reliability was examined using generalizability theory. Results indicated that blood oxygen level-dependent (BOLD) signal was reproducible across sites and was highly reliable, or generalizable, across scanning sites and testing days for core working memory ROIs (generalizability ICCs=0.81 for left dorsolateral prefrontal cortex, 0.95 for left superior parietal cortex). In the second study sample (n=154), two statistical methods for aggregating fMRI data across sites for all healthy individuals recruited as control participants in the NAPLS study were compared. Control participants were scanned on one occasion at the site from which they were recruited. Results from the image-based meta-analysis (IBMA) method and mixed effects model with site covariance method both showed robust activation in expected regions (i.e. dorsolateral prefrontal cortex, anterior cingulate cortex, supplementary motor cortex, superior parietal cortex, inferior temporal cortex, cerebellum, thalamus, basal ganglia). Quantification of the similarity of group maps from these methods confirmed a very high (96%) degree of spatial overlap in results. Thus, brain activation during working memory function was reliable across the NAPLS sites and both the IBMA and mixed effects model with site covariance methods appear to be valid approaches for aggregating data across sites. These findings indicate that multi-site functional neuroimaging can offer a reliable means to increase power and generalizability of results when investigating brain function in rare populations and support the multi-site investigation of working memory function in the NAPLS study, in particular.

Keywords: G-theory; Multi-site; Reliability; Working memory; fMRI.

Figure 1

Traveling participant total percent correct responses by site, in the order that sites were visited. Each marker corresponds to a different participant.

Figure 2

Traveling participant mean response times (ms) by site, in the order that sites were visited. Each marker corresponds to a different participant.

Figure 3

Traveling participant mean percent signal change ± standard error in dorsolateral prefrontal cortex (DLPFC), anterior cingulate cortex (ACC), superior parietal cortex (SP), and supplementary motor cortex (SM) averaged across participants for each site for the left (A) and right (B) hemisphere and averaged across sites for each participant in the left (C) and right (D) hemisphere. For subplots A and B, each bar within a given ROI corresponds to a different site. For subplots C and D, each bar within a given ROI corresponds to a different traveling participant.

Figure 3

Traveling participant mean percent signal change ± standard error in dorsolateral prefrontal cortex (DLPFC), anterior cingulate cortex (ACC), superior parietal cortex (SP), and supplementary motor cortex (SM) averaged across participants for each site for the left (A) and right (B) hemisphere and averaged across sites for each participant in the left (C) and right (D) hemisphere. For subplots A and B, each bar within a given ROI corresponds to a different site. For subplots C and D, each bar within a given ROI corresponds to a different traveling participant.

Figure 3

Traveling participant mean percent signal change ± standard error in dorsolateral prefrontal cortex (DLPFC), anterior cingulate cortex (ACC), superior parietal cortex (SP), and supplementary motor cortex (SM) averaged across participants for each site for the left (A) and right (B) hemisphere and averaged across sites for each participant in the left (C) and right (D) hemisphere. For subplots A and B, each bar within a given ROI corresponds to a different site. For subplots C and D, each bar within a given ROI corresponds to a different traveling participant.

Figure 3

Traveling participant mean percent signal change ± standard error in dorsolateral prefrontal cortex (DLPFC), anterior cingulate cortex (ACC), superior parietal cortex (SP), and supplementary motor cortex (SM) averaged across participants for each site for the left (A) and right (B) hemisphere and averaged across sites for each participant in the left (C) and right (D) hemisphere. For subplots A and B, each bar within a given ROI corresponds to a different site. For subplots C and D, each bar within a given ROI corresponds to a different traveling participant.

Figure 4

Proportion of variance attributable to each variance component for response accuracy and response time for traveling participants.

Figure 5

Proportion of variance attributable to each variance component for activation in left and right hemisphere anterior cingulate cortex (ACC), dorsolateral prefrontal cortex (DLPFC), supplementary motor cortex (SM), insula (IN), inferior temporal cortex (IT), superior parietal cortex (SP), occipital cortex (OCC), thalamus (T), basal ganglia (BG), and cerebellum (C) for the correct trials versus rest contrast, and for deactivation in medial frontal gyrus (MFG) and posterior cingulate cortex (PCC) for the rest versus correct trials contrast.

Figure 6

Functional activation group maps for control participants for all correct trials versus rest using image-based meta-analysis method (cluster peak Z score range: 19.7 – 21.8) and mixed effects model with site covariance method (cluster peak Z score range: 13.6 – 19.1).