Statistical properties of functional connectivity MRI enrichment analysis in school-age autism research

Abstract

Mass univariate testing on functional connectivity MRI (fcMRI) data is limited by difficulties achieving experiment-wide significance. Recent work addressing this problem has used enrichment analysis, which aggregates univariate screening statistics for a set of variables into a single enrichment statistic. There have been promising results using this method to explore fcMRI-behavior associations. However, there has not yet been a rigorous examination of the statistical properties of enrichment analysis when applied to fcMRI data. Establishing power for fcMRI enrichment analysis will be important for future neuropsychiatric and cognitive neuroscience study designs that plan to include this method. Here, we use realistic simulation methods, which mimic the covariance structure of fcMRI data, to examine the false positive rate and statistical power of one technique for enrichment analysis, over-representation analysis. We find it can attain high power even for moderate effects and sample sizes, and it strongly outperforms univariate analysis. The false positive rate associated with permutation testing is robust.

Keywords: Asd; BWAS; Brain Network; Enrichment; Functional connectivity; Resting state fcMRI.

Fig. 1

(a) Cortical ROIs, with network assignments for 13 functional networks derived from Seitzman 2020 (Seitzman et al. 2020), are displayed projected onto the surface of the brain (b) Cerebellar ROIs, with network assignments derived from the pre-press version of Seitzman 2020, are displayed onto a cerebellum flat map (Diedrichsen and Zotow, 2015).

Fig. 2

Illustration of fcMRI enrichment over-representation analysis. Mass univariate linear regression is performed on input fcMRI and behavior data. The resultant screening statistics are compared to a predefined threshold to binarize each as a hit or miss. The enrichment statistic is then the number of hits per network-pair. Not shown, these enrichment statistics are then assigned significance through either permutation testing or through comparison to a precalculated enrichment threshold.

Fig. 3

Covariance structure is not drastically affected by using raw connectivity values as opposed to controlling for behaviors first. For 1000 random ROI-pairs plotting correlations between observed connectivity against correlations between residuals when regressing connectivity on behavior in actual data.

Fig. 4

Statistical significance can be unattainable for smaller network-pairs. The negative logarithm of false positive rates (determined using previously computed enrichment statistic thresholds) for network-pairs with fewer than 50 constituent ROI-pairs are plotted against the size of the network-pair, colored by the screening statistic threshold used. The target false positive rate ($p={10}^{-4}$) is indicated by the black dashed line.

Fig. 5

An illustration of how enrichment statistic cutoffs were determined. Panel 1 shows the enrichment statistic CDFs for 1000 generated datasets ($N=105)$, comprised of a million permutations each, for the DMN-SML network-pair (size $S=336$) and BOT-2 upper limb subscore, and screening statistic threshold $\alpha =0.5$ (i.e., screening statistics in the top 50 % of the distribution count as hits, corresponding to a positive screening statistic). Panel (b) shows the selection of the first enrichment statistic associated with $p\le 0.0001$ for each CDF. Panel (c) shows the distribution of these values, with the official enrichment statistic cutoff selected as the 75th quantile, with the 25th quantile selected as an alternate cutoff to explore the effect of quantile choice. A range of cutoffs between 230 and 235 for a network-pair of this size is reasonable. (Key: DMN = Default Mode, SML = Somatomotor Lateral).

Fig. 6

Simulated power to detect an embedded connectivity/behavior effect (characterized by mean ($\mu$) and standard deviation ($\sigma$) of the effect across ROI-pairs) for various sample sizes and network-pairs. The top row has sample size $N=25$, the middle row $N=50$, and the bottom row $N=150$. The columns, from left to right, use the DAN-MTL network-pair (of size $R=56$ ROI-pairs), the CO-SAL network-pair ($R=234$), the CO-FP network-pair ($R=936$), and the DAN-DMN network-pair ($R=938$). All plots here use a screening statistic threshold of $\mathrm{\alpha }=0.90$. (Key: DMN = Default Mode, CO = Cingulo Opercular, DAN = Dorsal Attention, MTL = Medial Temporal Lobe, SAL = salience, FP = Fronto-Parietal).

Fig. 7

As screening statistic threshold increases, power increases for most embedded effects, though power may decrease for effects with sufficiently small deviation across ROI-pairs (σ). The contour corresponding to 80 % power is shown for four network-pairs and a sample size of $N=50$, for three screening statistic thresholds, from $\alpha =0.50$ (any positive association is counted as a hit) to $\alpha =0.95$ (top 5 % of associations are counted). That is, a connectivity/behavior effect characterized by (µ, σ) to the right of a contour will be detected by ORA enrichment analysis with corresponding screening statistic threshold with a power greater than 80 %. (Key: DMN = Default Mode, CO = Cingulo Opercular, DAN = Dorsal Attention, MTL = Medial Temporal Lobe, SAL = salience, FP = Fronto-Parietal).

Fig. 8

Power is not drastically affected by deriving the ORA enrichment statistic cutoffs for statistical significance from the 25th vs the 75th percentile of the previously found distributions of significance cutoffs. For four network pairs, a range of embedded effects, and sample size N = 50, the power of ORA enrichment analysis using the cutoffs derived from the 75th percentile of the previously found distributions are plotted against the power in detecting the same effect using the cutoffs derived using the 25th percentile. The line of parity is also shown in red. (Key: DMN = Default Mode, CO = Cingulo Opercular, DAN = Dorsal Attention, MTL = Medial Temporal Lobe, SAL = salience, FP = Fronto-Parietal).

Fig. 9

Enrichment analysis attains high power even for effects with low coefficient of determination. For three network-pairs (with network-pair size $R=742$, 1742, and 3551 respectively), expected $R^2$ is displayed as a function of µ and σ. Additionally, overlaid on top of this, the contour corresponding with 80 % power is displayed for various sample sizes. That is, a connectivity/behavior effect characterized by (µ, σ) to the right of the yellow contour will have a corresponding ORA enrichment power greater than 80 % for a sample size of $N=50$, but an effect characterized by a (µ, σ) to the left of the red contour will have an ORA enrichment power less than 80 % for a sample size of $N=25$. (Key: DMN = Default Mode, SMD = Somatomotor Dorsal, CO = Cingulo Opercular, DAN = Dorsal Attention).

Fig. 10

ORA enrichment analysis is well-powered for milder effects than univariate analysis. Comparisons between the power of univariate analysis (for $p=0.001$) on a single ROI-pair in detecting an embedded signal in simulated data and the power of enrichment analysis (for $\alpha =0.95$) in detecting an equivalent signal in three different network-pairs of various sizes ($S=3551$, 1742, and 742 for DMN-SMD, DMN-CO, and SMD-DAN respectively), for three sample sizes (in the left column $N=25$, in the middle $N=50$, and in the right $N=100$). The statistical power is plotted against the mean effect size ($\mathrm{\mu }$), for univariate analysis and for enrichment analysis on each network-pair, for two effect size standard deviations, $\mathrm{\sigma }=0.1$ on the top row and $\mathrm{\sigma }=0.5$ on the bottom. (Key: DMN = Default Mode, SMD = Somatomotor Dorsal, CO = Cingulo Opercular, DAN = Dorsal Attention).