Using predictive validity to compare associations between brain damage and behavior

Abstract

Lesion-behavior mapping (LBM) provides a statistical map of the association between voxel-wise brain damage and individual differences in behavior. To understand whether two behaviors are mediated by damage to distinct regions, researchers often compare LBM weight outputs by either the Overlap method or the Correlation method. However, these methods lack statistical criteria to determine whether two LBM are distinct versus the same and are disconnected from a major goal of LBMs: predicting behavior from brain damage. Without such criteria, researchers may draw conclusions from numeric differences between LBMs that are irrelevant to predicting behavior. We developed and validated a predictive validity comparison method (PVC) that establishes a statistical criterion for comparing two LBMs using predictive accuracy: two LBMs are distinct if and only if they provide unique predictive power for the behaviors being assessed. We applied PVC to two lesion-behavior stroke data sets, demonstrating its utility for determining when behaviors arise from the same versus different lesion patterns. Using region-of-interest-based simulations derived from proportion damage from a large data set (n = 131), PVC accurately detected when behaviors were mediated by different regions (high sensitivity) versus the same region (high specificity). Both the Overlap method and Correlation method performed poorly on the simulated data. By objectively determining whether two behavioral deficits can be explained by single versus distinct patterns of brain damage, PVC provides a critical advance in establishing the brain bases of behavior. We have developed and released a GUI-driven web app to encourage widespread adoption.

Keywords: lesion-behavior mapping; multivariate analysis; neuropsychology; stroke.

FIGURE 1

Model building phase in the predictive validity comparison (PVC) method. (a) Normalized, segmented lesion volumes for all participants (S₁ to S _n; purple denotes lesioned voxels). (b) Center and scale (z‐scored) behavioral scores (B_1, B₂) prior to fitting for S₁ to S _n (B₁ ➔ Z ₁, blue; B₂ ➔ Z ₂, orange). (c) Under the null hypothesis (H₀), a single LBM is fit using a single behavior score, Z ₀ (z‐scored average of Z ₁ and Z ₂, left) and subject's lesion volume (middle), producing a coefficient for each voxel (scaled from 0 to 1 for display purposes, transparent white to opaque red; right). (d) Under the alternative hypothesis (H_A), distinct LBMs are fit using participants' lesion volumes paired with each scaled behavior Z ₁ (LBM₁) and Z₂ (LBM₂).

FIGURE 2

Model comparison in the predictive validity comparison (PVC) method. (a) Participants' lesion volumes are multiplied by each fitted lesion‐behavior map (LBM) to generate predicted values. (b) Under the null hypothesis (H₀) there is a single LBM, producing a single prediction (${\widehat{Z}}_0$) for the two actual scaled behaviors ($Z_1,Z_2$). (c) Predictions are compared with the actual scaled behaviors using AIC (solid line indicates equality between predicted and actual values). The total AIC (inset solid bar) for H₀ calculated. (d) Under the alternative hypothesis (H_A), there are distinct LBMs (LBM₁, LBM₂), producing distinct predictions (${\widehat{Z}}_1$, ${\widehat{Z}}_2$) for the two actual scaled behaviors. (e) Predictions are compared with actual behaviors and total AIC for H_A calculated (inset). (g) AIC difference (H₀ − H_A) determines the winning hypothesis: positive values favor H_A; negative values favor H₀. The gray dashed line at +10:1 is the cutoff for claiming decisive support for H_A.

FIGURE 3

Comparing lesion‐behavior maps (LBMs) built from the MRRI data. (a) Scatter plot showing the strong linear relationship (r = 0.89) between the Western Aphasia Battery Aphasia Quotient (AQ) and the Philadelphia naming task (% accuracy) for 130 participants. (b) Fitted voxel weights (max‐scaled across maps; transparent white to opaque red) from LBMs fitted under the null hypothesis (LBM₀) and alternative hypothesis (LBM₁ and LBM₂). The axial slice shown (z = 107) had the most nonzero voxel weights for both LBM₁ and LBM₂. LH: Left Hemisphere. (c)/(d) The PVC method compares the actual behavior to predictions generated under the null hypothesis (H₀) and the alternative hypotheses (H_A). Solid diagonal line indicates perfect prediction. (e) The AIC difference was decisive for H₀ (cutoff at −10, gray dashed line). (f) The Overlap method highlights slices that differ between the LBMs. Voxel color indicates the sign and magnitude of the difference (opaque blue to transparent white to opaque red; max‐scaled). The dice coefficient of 0.77 suggests a moderate to high overlap between the maps. (g) The Correlation method suggested a moderate relationship (r = 0.64) between the weights in LBM₁ and LBM₂.

FIGURE 4

Comparing lesion‐behavior maps (LBMs) built from the Schnur Laboratory data. (a) Scatter plot showing no linear relationship between z‐scored words per min (speech rate) and z‐scored relative percentage of pronouns to nouns produced for 52 participants. (b) Fitted voxel weights (max‐scaled across maps; transparent white to opaque red) from LBMs fitted under the null hypothesis (LBM₀) and alternative hypothesis (LBM₁ and LBM₂). The axial slices shown (z = 84, 113) had the most nonzero voxel weights for LBM₁ and LBM₂, respectively. LH: Left Hemisphere. Yellow arrows note areas of distinction across the slices. (c)/(d) The PVC method compares the actual behavior to predictions generated under the null hypothesis (H₀) and the alternative hypotheses (H_A). Solid diagonal line indicates perfect prediction. (e) The AIC difference (H₀ − H_A = 1143) used by the PVC method was decisive for H_A (cutoff at +10, gray dashed line). (f) The Overlap method highlights slices that differ between the LBMs. Voxel color indicates the sign and magnitude of the difference (opaque blue to transparent white to opaque red; max‐scaled). The Dice coefficient of 0.52 suggests a moderate overlap between the maps. (g) The Correlation method assesses the relationship between the weights in LBM₁ and LBM₂, suggesting a weak but significant positive correlation (r = 0.11, p = 10⁻¹⁶).

FIGURE 5

Effect of varying SCCAN sparseness parameter and directionality of voxel weights on lesion‐behavior map (LBM) comparison results using real data. More sparse models have fewer voxels in the resulting LBM; number of nonzero voxels in the fitted LBMs are plotted on the horizontal axis (sparsity) and voxel‐weight directionality is shown in separate lines (pink: directional weights allowed; black: non‐negative weights only). The dashed gray line provides the threshold for determining same versus different LBMs. (a) For the MRRI dataset (H₀), changing the sparsity of the result had quantitative impacts on the AIC difference used by the predictive validity comparison (PVC) method (Left), but no change in the conclusion (all simulations decisively supported the null hypothesis). For the Dice coefficient used by the Overlap method (second column) and the correlation‐based methods (third and fourth columns), the sparseness of the model modulated the quantitative metrics, but there was little impact of voxel‐weight directionality on the results. Across much of the range, all methods supported the same conclusion (maps are the same), except for sparse maps using the ROI correlation method. (b) For the Schnur Lab dataset (H_A), the PVC method again showed quantitative differences across parameter manipulations, but no decisional change (all simulations favored the alternative hypothesis). The Dice coefficient and the ROI‐level correlation were inconsistent and strongly dependent on the choice of sparseness and voxel directionality; the voxel‐wise correlation showed consistent support for H₀.

FIGURE 6

Simulation results. (a) Averaged accuracy from the different‐region simulations provide the sensitivity (% correct for judging that two behaviors have different neural bases) for each method (PVC: Blue; Voxel‐wise correlation: pale orange; ROI‐level correlation: orange; Overlap method/Dice coefficient: green), binned by the between‐region damage correlation. Each point represents the average sensitivity across a varying number of simulations. (b) Relative sensitivity versus PVC for each method (same colors as panel a). Only bins with >50% sensitivity are displayed. The dashed line at 0 represents the sensitivity of the PVC method (blue line in panel a). (c) The number of simulations within each bin (total simulation count = 765). This distribution was determined by the lesion distribution in the MRRI dataset.