Replicability and generalizability in population psychiatric neuroimaging

Abstract

Studies linking mental health with brain function in cross-sectional population-based association studies have historically relied on small, underpowered samples. Given the small effect sizes typical of such brain-wide associations, studies require samples into the thousands to achieve the statistical power necessary for replicability. Here, we detail how small sample sizes have hampered replicability and provide sample size targets given established association strength benchmarks. Critically, while replicability will improve with larger samples, it is not guaranteed that observed effects will meaningfully apply to target populations of interest (i.e., be generalizable). We discuss important considerations related to generalizability in psychiatric neuroimaging and provide an example of generalizability failure due to "shortcut learning" in brain-based predictions of mental health phenotypes. Shortcut learning is a phenomenon whereby machine learning models learn an association between the brain and an unmeasured construct (the shortcut), rather than the intended target of mental health. Given the complex nature of brain-behavior interactions, the future of epidemiological approaches to brain-based studies of mental health will require large, diverse samples with comprehensive assessment.

Fig. 1. Sampling variability of correlations.

Sampling variability (± of correlation on y-axis; magnitude of colored cells) of bivariate correlations (r) as a function of sample size (x-axis), ranging from N  = 25 to N = 500,000. Larger values represent higher sampling variability of a bivariate correlation across 1000 equivalently-sized subsamples. Sampling variability was quantified as the 99th percent confidence interval around an effect across 1000 bootstrapped samples.

Fig. 2. Effects of sample size on an exemplar brain-behavior association.

A Effect size distribution (bivariate |r|) of resting-state functional connectivity edges with fluid intelligence for curated samples from the HCP (N = 900), ABCD (N = 3928) and UK Biobank (N = 32,572). Estimates of statistical power (y-axis) as a function of sample size (x-axis) for resting-state functional connectivity (RSFC) and fluid intelligence associations across all connections (brain features same across data sets), using (B) HCP (N = 900), (C) ABCD (N = 3928), and (D) UKB (N = 32,572), as reference datasets. Associations that passed statistical significance testing (q < 0.05, FDR corrected) in the respective full reference sample were included. In all panels, the black line (‘best’) represents the strongest association, the red dotted line (“mean”) represents the average statistical power across all significant (q < 0.05) RSFC with fluid intelligence associations, and the blue dotted line (“min”) represents the statistical power for the weakest suprathreshold association. Gray lines represent the statistical power for each statistically significant RSFC with fluid intelligence association. Each panel was scaled to the ABCD sample size for side-by-side comparison. Note the rightward shift for ABCD and UKB relative to HCP, demonstrating effects sizes are likely inflated even when using samplings in the hundreds (HCP) or under 10,000 (ABCD).

Fig. 3. Mental health effect sizes.

Univariate correlations (top 1%) between resting-state functional connectivity edges and an array of mental health variables. See Table S1 for the list of ordered variables.

Fig. 4. Generalizability of a brain-based model of mental health symptoms.

Out-of-sample correlation (r_oos; y-axis) of brain-based (RSFC) prediction of mental health symptoms (BIS-BAS) for varying samples drawn from the ABCD training dataset with varying levels of disturbances in initiating and maintaining sleep (i.e., sleep disturbances; x-axis). All training samples contained N = 400 individuals; the testing sample always contained N = 1964 individuals. Moving to the right on the x-axis indicates greater inclusion of individuals with sleep disturbances in the training sample. For example, “7+” includes individuals from the full range of sleep disturbances (from <2 to 7+). Increasing the range of sleep disturbances in the training sample improved the generalizability of brain-based models of mental health.