Comparing the stability and reproducibility of brain-behavior relationships found using canonical correlation analysis and partial least squares within the ABCD sample

Abstract

Canonical correlation analysis (CCA) and partial least squares correlation (PLS) detect linear associations between two data matrices by computing latent variables (LVs) having maximal correlation (CCA) or covariance (PLS). This study compared the similarity and generalizability of CCA- and PLS-derived brain-behavior relationships. Data were accessed from the baseline Adolescent Brain Cognitive Development (ABCD) dataset (N > 9,000, 9-11 years). The brain matrix consisted of cortical thickness estimates from the Desikan-Killiany atlas. Two phenotypic scales were examined separately as the behavioral matrix; the Child Behavioral Checklist (CBCL) subscale scores and NIH Toolbox performance scores. Resampling methods were used to assess significance and generalizability of LVs. LV₁ for the CBCL brain relationships was found to be significant, yet not consistently stable or reproducible, across CCA and PLS models (singular value: CCA = .13, PLS = .39, p < .001). LV₁ for the NIH brain relationships showed similar relationships between CCA and PLS and was found to be stable and reproducible (singular value: CCA = .21, PLS = .43, p < .001). The current study suggests that stability and reproducibility of brain-behavior relationships identified by CCA and PLS are influenced by the statistical characteristics of the phenotypic measure used when applied to a large population-based pediatric sample.

Keywords: Brain-behavior relationships; Cortical thickness; Multivariate modeling; Population-based samples.

Figure 1.

Overview of the analytical pipeline used in the current study. (A) The main CCA and PLS analysis: the decomposition of the cross-product matrices to produce singular values and loadings (reweighted singular vectors in the case of CCA). (B) The sum of squares permutation analysis to assess the statistical significance of each LV. (C) The split-half analysis used to assess the similarity between respective loadings from each split-half. In this analysis, the brain and behavioral data are both split into two different matrices (resulting in X₁, X₂, Y₁, Y₂) for each of the 10,000 iterations. Each respective split pair (e.g., X₁ and Y₁) then undergoes the CCA or PLS analysis shown in panel A. The resulting loadings from the respective split pairs are correlated (e.g., X₁ and X₂) to provide a distribution of Pearson correlation coefficients between each respective brain and behavior loading across the 10,000 iterations. (D) The train-test resampling analysis that assesses how well the singular values from the training sample can predict the singular values of the test sample. For each of the 10,000 iterations, the dataset was split into an 80% train set (X₈₀, Y₈₀) and 20% test set (X₂₀, Y₂₀). The train set underwent the CCA or PLS analysis shown in panel A. The respective loadings from the SVD of the training set were used to solve for the singular values (S) of the test set. The resulting predicted singular values of each of the 10,000 train/test analyses were plotted as a distribution. Importantly, each analysis is independent from one another (i.e., the results of one analysis are not used in another) and not sequential (i.e., the permutation test in panel B did not necessarily need to happen prior to the split-half or train-test analysis). The bootstrap confidence interval estimation, not shown here, was used to assess the stability of the parameter estimates of variable weights in the loadings. SVD = singular value decomposition.

Figure 2.

Unthresholded behavior and brain loadings from the PLS and CCA analysis performed in the CBCL brain analysis (N = 9,191). The highest PLS-derived behavior loadings were aggressive behavior, thought problems, and stress problems. The highest PLS-derived brain loadings were right pars triangularis, right inferior parietal cortex, and left posterior cingulate cortex. The highest CCA-derived behavior loadings were social problems, anxious/depressive symptoms, and stress problems. The highest CCA-derived brain loadings were the right superior temporal gyrus, left fusiform gyrus, and right lingual gyrus. Panel B shows the latent scores between XU and YV for LV₁. Prior to calculating the latent scores, the brain and behavioral loadings have been standardized by the singular values. Panel C depicts the train-test distributions of the predicted singular values of the test sample for each iteration. Asterisks indicate the LVs that showed a distribution with a Z-score greater than 1.96. LV₁ from CCA was found to be reliable (i.e., LV₁ of the training sample can reliably predict the singular values of LV₁ from the test sample). The lack of any other significant distributions of predicted singular values suggest that the 80% train set does not reliably and consistently predict the singular values from the 20% test set. OCD = obsessive compulsive symptoms; withdep = withdrawn/depression symptoms; sct = sluggish-cognitive-tempo; anxdep = anxious/depressive symptoms; rulebreak = rule-breaking behavior.

Figure 3.

Unthresholded behavior and brain loadings of LV₁ from the PLS and CCA analysis performed between NIH Cognitive Toolbox scores and cortical thickness. The largest PLS- and CCA-derived loadings were found for the list sorting task and the picture vocabulary task. The highest CCA-derived brain loadings were the left pars opercularis, superior frontal gyrus, and parahippocampal gyrus. The highest PLS-derived brain loadings were the left pars opercularis, parahippocampal gyrus, and medial orbitofrontal gyrus. Panel B shows the latent scores between XU and YV for LV₁. Prior to calculating the latent scores, the brain and behavioral loadings have been standardized by the singular values. Overall, there is a similar relationship between the brain and behavioral latent scores when comparing CCA and PLS. Panel C shows the results of the train-test resampling analysis. Asterisks indicate the LVs that showed a distribution with a Z-score greater than 1.96. LV₁ (Z-score: PLS = 4.8, CCA = 7.8) and LV₃ (Z-score: PLS = 2.6, CCA = 2.2) for both PLS and CCA, and LV₂ (Z-score: CCA = 2.02) for CCA, were found to be reliable (i.e., singular values of these LVs of the training sample can reliably predict the singular values from the test sample). Flanker = Flanker task; pattern = pattern comparison processing speed task; cardsort = dimensional change card sort task; reading = oral reading recognition task; picture = picture vocabulary task; list = list sorting working memory task; picvocab = picture vocabulary task.

Figure 4.

This figure depicts the distributions of the resampled loadings from the split-half analysis for the CBCL brain and NIH brain analyses. The x-axis from the split-half distributions are the Pearson correlation coefficients between respective loadings from each split-half analysis (e.g., U₁ and U₂ from the analysis comparing X₁ and Y₁ and separately, X₂ and Y₂). In the CBCL brain analysis, the distribution of Pearson correlation coefficients centered around 0 for the majority of LVs, indicating minimal correspondence between respective loadings from the split-halves. This suggests that characteristics of participants are highly influential in the loadings derived from CCA or PLS models in the CBCL brain analysis. In the NIH brain analysis, the distribution of Pearson correlation coefficients are centered around r = 0.6–0.8, indicating high correspondence between respective split-halves and that loadings from CCA and PLS models remain similar regardless of which participants are included in each iteration. Asterisks indicate the LVs which showed a distribution with a Z-score greater than 1.96.