Reliability of functional magnetic resonance imaging activation during working memory in a multisite study: Clarification and implications for statistical power

Abstract

In this technical note, we clarify the meaning of the generalizability-theory based coefficients reported in our multisite reliability study of fMRI measures of regional brain activation during working memory processing (Forsyth et al., Neuroimage 2014;97:51-52). While the original paper reported generalizability and dependability coefficients based on the design of our traveling subjects study (in which each subject was scanned twice at each of eight sites), those coefficients are of limited applicability outside of the reliability study context. Here we report generalizability and dependability coefficients that represent the reliability one can expect for a multisite study in which a given subject is scanned once on a scanner drawn randomly from the pool of available scanners (i.e., analogous to the more typical multisite study design). We also characterize the implications of a multisite versus single site study design for statistical power, including a figure that shows sample size requirements to detect activation in two key nodes of the working memory circuitry given observed differences in reliability of measurement between single and multisite designs.

Figure

Statistical power as a function of sample size across multiple effect sizes (Cohen’s d for one group all correct trials versus rest contrast) for left dorsolateral prefrontal cortex (A) and left superior parietal cortex (B). The red lines represent power for multisite studies while the blue lines represent power for single-site studies, with nominal effect sizes adjusted downward for observed reliabilities in the multisite and single-site contexts, respectively. For DLPFC, although higher levels of power are achieved with smaller sample sizes in the single-site compared with multisite context, multisite studies achieve acceptable levels of power (≥ 0.8) with moderate to large effect sizes (ES ≥ 0.5) beginning at sample sizes of approximately 125 subjects. For PC, where there is no difference between multisite versus single site reliability (and therefore the red and blue lines overlap), power is adequate to detect moderate to large effect sizes (ES ≥ 0.5) beginning at sample sizes of approximately 50 subjects in either the single or multi-side study context.