Power and reproducibility in the external validation of brain-phenotype predictions

Abstract

Identifying reproducible and generalizable brain-phenotype associations is a central goal of neuroimaging. Consistent with this goal, prediction frameworks evaluate brain-phenotype models in unseen data. Most prediction studies train and evaluate a model in the same dataset. However, external validation, or the evaluation of a model in an external dataset, provides a better assessment of robustness and generalizability. Despite the promise of external validation and calls for its usage, the statistical power of such studies has yet to be investigated. In this work, we ran over 60 million simulations across several datasets, phenotypes, and sample sizes to better understand how the sizes of the training and external datasets affect statistical power. We found that prior external validation studies used sample sizes prone to low power, which may lead to false negatives and effect size inflation. Furthermore, increases in the external sample size led to increased simulated power directly following theoretical power curves, whereas changes in the training dataset size offset the simulated power curves. Finally, we compared the performance of a model within a dataset to the external performance. The within-dataset performance was typically within r=0.2 of the cross-dataset performance, which could help decide how to power future external validation studies. Overall, our results illustrate the importance of considering the sample sizes of both the training and external datasets when performing external validation.

Figure 1.

Within-dataset held-out prediction performance in HBN for age, attention problems, and matrix reasoning. The performance was evaluated in a randomly selected held-out sample of size n=200. The error bars show the 2.5^th and 97.5^th percentiles among 100 repeats of resampling at each training sample size. The dotted line reflects the correlation value required for a significance level of p<0.05. Similar results were observed for the ABCD, HCPD, and PNC datasets; see Figures S2–3. AP: attention problems, MR: matrix reasoning.

Figure 2.

Power and false positive rates for cross-dataset predictions, training in HBN and testing in ABCD (top row), HCPD (middle row), or PNC (bottom row) for prediction of age (left column), attention problems (middle column), or matrix reasoning (right column). The blue lines represent theoretical power assuming a known ground truth performance. The panel with N/A means that data were not included in this study. Similar results were observed for the ABCD, HCPD, and PNC datasets; see Figure S4. AP: attention problems, MR: matrix reasoning.

Figure 3.

Median effect size inflation for cross-dataset predictions, training in HBN and testing in ABCD (top row), HCPD (middle row), or PNC (bottom row) for prediction of age (left column), attention (middle column), or matrix reasoning (right column). Panels with N/A mean that data were not available. Similar results were observed for the ABCD, HCPD, and PNC datasets; see Figure S5. AP: attention problems, MR: matrix reasoning.

Figure 4.

Boxplots of the difference between internal and external performance for each subsample of the training data. For each training data size, 100 random subsamples were taken. The model was evaluated for internal performance in a held-out sample of size n=200. For external performance, the model formed in the training subsample was applied to the full external dataset. Panels with N/A mean that data were not available. Similar results were observed for the ABCD, HCPD, and PNC datasets; see Figure S6. AP: attention problems, MR: matrix reasoning.