Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Comment
. 2023 Mar;615(7951):E4-E7.
doi: 10.1038/s41586-023-05745-x. Epub 2023 Mar 8.

Multivariate BWAS can be replicable with moderate sample sizes

Tamas Spisak  1   2 , Ulrike Bingel  3 , Tor D Wager  4
Affiliations
Comment

Multivariate BWAS can be replicable with moderate sample sizes

Tamas Spisak et al. Nature. 2023 Mar.
No abstract available

PubMed Disclaimer

Conflict of interest statement

The authors declare no competing interests.

Figures

Fig. 1
Fig. 1. Examples of multivariate BWAS providing unbiased effect sizes and high replicability with low to moderate sample sizes.
a, Discovery sample effects in multivariate BWAS are inflated only if estimates are obtained without cross-validation (CV). b, Cross-validation fully eliminates in-sample effect-size inflation and, as a consequence, provides higher replicability. Data are from the Human Connectome Project (HCP1200, PTN release, n = 1,003). Each point in a and b corresponds to one bootstrap subsample, as in figure 4b of Marek et al.. The dotted lines denote the threshold for P = 0.05 with n = 495. Mean multivariate brain–behavioural phenotype associations across 100 bootstrap samples at n = 200 and for the full sample are denoted by red and purple dots. c, The inflation of in-sample effect size obtained without cross-validation (red) is reduced, but does not disappear, at higher sample sizes. Conversely, cross-validated estimates (blue) are slightly pessimistic with low sample sizes and become quickly unbiased as sample size is increased. d, Without cross-validation, in-sample effect-size estimates are non-zero (r ≈ 0.5, red), even when predicting permuted outcome data. Cross-validation eliminates systematic bias across all sample sizes (blue). The dashed lines in c and d denote 95% parametric confidence intervals, and the shaded areas denote bootstrap- and permutation-based confidence intervals. e,f, Cross-validated analysis reveals that sufficient in-sample power (e) and out-of-sample replication probability (Prep) (f) can be achieved for a variety of phenotypes at low or moderate sample sizes. 80% power and Prep are achievable in <500 participants for 3 out of 6 phenotypes (coloured bars) using the prediction algorithm of Marek et al. (e and f (top), the sample size required for 80% power or Prep is shown). The remaining three phenotypes require sample sizes of >500 (bars with arrows). Power and Prep can be substantially improved with a ridge regression-based model recommended in some comparison studies, (e and f (bottom), with 80% power and Prep with sample sizes as low as n = 100 and n = 75, respectively, when predicting cognitive ability, and sample sizes between 75 and 375 for other investigated variables (fluid intelligence, episodic memory and cognitive flexibility), except inhibition assessed with the flanker task, which replicated with n = 375 but did not reach 80% power with n = 500. g, We estimated interactions between sample size and publication bias by computing effect size inflation (rdiscovery − rreplication) only for those bootstrap cases in which prediction performance was significant (P > 0.05) in the replication sample. Our analysis shows that the effect-size inflation due to publication bias is modest (<10%) with fewer than 500 participants for half of the phenotypes using the model from Marek et al. and all phenotypes but the flanker using the ridge model. The blue squares show conditional relationships assessed to derive metrics in e,f and g with reference to b. The top and bottom squares indicate positive and negative results in the discovery sample, respectively. The left and right squares indicate negative and positive results in the replication sample. The blue squares indicate how these conditions were applied to derive the metrics.
See this image and copyright information in PMC

Comment in

  • Reply to: Multivariate BWAS can be replicable with moderate sample sizes.
    Tervo-Clemmens B, Marek S, Chauvin RJ, Van AN, Kay BP, Laumann TO, Thompson WK, Nichols TE, Yeo BTT, Barch DM, Luna B, Fair DA, Dosenbach NUF. Tervo-Clemmens B, et al. Nature. 2023 Mar;615(7951):E8-E12. doi: 10.1038/s41586-023-05746-w. Nature. 2023. PMID: 36890374 Free PMC article. No abstract available.

Comment on

  • Reproducible brain-wide association studies require thousands of individuals.
    Marek S, Tervo-Clemmens B, Calabro FJ, Montez DF, Kay BP, Hatoum AS, Donohue MR, Foran W, Miller RL, Hendrickson TJ, Malone SM, Kandala S, Feczko E, Miranda-Dominguez O, Graham AM, Earl EA, Perrone AJ, Cordova M, Doyle O, Moore LA, Conan GM, Uriarte J, Snider K, Lynch BJ, Wilgenbusch JC, Pengo T, Tam A, Chen J, Newbold DJ, Zheng A, Seider NA, Van AN, Metoki A, Chauvin RJ, Laumann TO, Greene DJ, Petersen SE, Garavan H, Thompson WK, Nichols TE, Yeo BTT, Barch DM, Luna B, Fair DA, Dosenbach NUF. Marek S, et al. Nature. 2022 Mar;603(7902):654-660. doi: 10.1038/s41586-022-04492-9. Epub 2022 Mar 16. Nature. 2022. PMID: 35296861 Free PMC article.

References

    1. Button KS, et al. Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 2013;14:365–376. doi: 10.1038/nrn3475. - DOI - PubMed
    1. Woo C-WW, Chang LJ, Lindquist MA, Wager TD. Building better biomarkers: Brain models in translational neuroimaging. Nat. Neurosci. 2017;20:365–377. doi: 10.1038/nn.4478. - DOI - PMC - PubMed
    1. Bzdok D, Varoquaux G, Steyerberg EW. Prediction, not association, paves the road to precision medicine. JAMA Psychiatry. 2021;78:127–128. doi: 10.1001/jamapsychiatry.2020.2549. - DOI - PubMed
    1. Marek S, et al. Reproducible brain-wide association studies require thousands of individuals. Nature. 2022;603:654–660. doi: 10.1038/s41586-022-04492-9. - DOI - PMC - PubMed
    1. Poldrack RA, Huckins G, Varoquaux G. Establishment of best practices for evidence for prediction: a review. JAMA Psychiatry. 2020;77:534–540. doi: 10.1001/jamapsychiatry.2019.3671. - DOI - PMC - PubMed