Flexible multi-step hypothesis testing of human ECoG data using cluster-based permutation tests with GLMEs

Abstract

Background: Time series analysis is critical for understanding brain signals and their relationship to behavior and cognition. Cluster-based permutation tests (CBPT) are commonly used to analyze a variety of electrophysiological signals including EEG, MEG, ECoG, and sEEG data without a priori assumptions about specific temporal effects. However, two major limitations of CBPT include the inability to directly analyze experiments with multiple fixed effects and the inability to account for random effects (e.g. variability across subjects). Here, we propose a flexible multi-step hypothesis testing strategy using CBPT with Linear Mixed Effects Models (LMEs) and Generalized Linear Mixed Effects Models (GLMEs) that can be applied to a wide range of experimental designs and data types.

Methods: We first evaluate the statistical robustness of LMEs and GLMEs using simulated data distributions. Second, we apply a multi-step hypothesis testing strategy to analyze ERPs and broadband power signals extracted from human ECoG recordings collected during a simple image viewing experiment with image category and novelty as fixed effects. Third, we assess the statistical power differences between analyzing signals with CBPT using LMEs compared to CBPT using separate t-tests run on each fixed effect through simulations that emulate broadband power signals. Finally, we apply CBPT using GLMEs to high-gamma burst data to demonstrate the extension of the proposed method to the analysis of nonlinear data.

Results: First, we found that LMEs and GLMEs are robust statistical models. In simple simulations LMEs produced highly congruent results with other appropriately applied linear statistical models, but LMEs outperformed many linear statistical models in the analysis of "suboptimal" data and maintained power better than analyzing individual fixed effects with separate t-tests. GLMEs also performed similarly to other nonlinear statistical models. Second, in real world human ECoG data, LMEs performed at least as well as separate t-tests when applied to predefined time windows or when used in conjunction with CBPT. Additionally, fixed effects time courses extracted with CBPT using LMEs from group-level models of pseudo-populations replicated latency effects found in individual category-selective channels. Third, analysis of simulated broadband power signals demonstrated that CBPT using LMEs was superior to CBPT using separate t-tests in identifying time windows with significant fixed effects especially for small effect sizes. Lastly, the analysis of high-gamma burst data using CBPT with GLMEs produced results consistent with CBPT using LMEs applied to broadband power data.

Conclusions: We propose a general approach for statistical analysis of electrophysiological data using CBPT in conjunction with LMEs and GLMEs. We demonstrate that this method is robust for experiments with multiple fixed effects and applicable to the analysis of linear and nonlinear data. Our methodology maximizes the statistical power available in a dataset across multiple experimental variables while accounting for hierarchical random effects and controlling FWER across fixed effects. This approach substantially improves power leading to better reproducibility. Additionally, CBPT using LMEs and GLMEs can be used to analyze individual channels or pseudo-population data for the comparison of functional or anatomical groups of data.

Keywords: Broadband Power; Burst Analysis; Cluster-based statistics; Event Related Potentials (ERPs); Mixed Effects Models; generalized linear models (GLMs); linear models.

Figure 1:. Replication of Previous Results using a Predefined Time Window.

A) Proportion of channels whose ERP signals’ significance were labeled the same or different by LMEs and separate t-tests. B) Proportion of channels whose broadband (BB) power signals’ significance was labeled the same or different by LMEs and separate t-tests. C) Congruence of results across data types and statistical models. Congruence was lower across data types. D) Percent of channels labeled as significant by data type and statistical model. E) Averaged ERP responses for significant channels (blue) vs. non-significant channels (orange) showed biphasic responses as well as sustained activity after image onset. F) Averaged broadband power responses for significant channels (blue) vs. non-significant channels (orange) showed good onset timing but missed sustained activity after image offset. G) Averaged ERP responses for category-selective channels (n = 70) showed most category-selectivity occurred within the predefined time window. H) Averaged ERP responses for novelty-selective channels (n = 26) showed sustained novelty-selectivity after image offset. I) Averaged ERP responses for channels (n = 6) selective for both category and novelty showed more complicated responses with sustained activity after image offset. *Shaded regions indicate the predefined time window from 100-400 ms after image onset. Images disappeared at 400 ms.

Figure 2:. ERP and Broadband Power Analysis using CBPT with LMEs.

A) Average magnitude of category beta weights for face-, house-, and non-selective ERP channels. B) Category beta weight peak latencies (mean +/− s.e.m) of face- and house-selective ERP channels (ks-test, p = 0.649). C) Magnitude of the category beta weights for face and house ERP group-level models. D) Adjusted R² values for various group-level models: fixed effects-only (FE only), random effect for patient IDs (RE Patient), random effect for channel IDs (RE Channel), and hierarchical random effects (RE Hier). Asterisks indicate when a more complicated model was a better fit than the less complicated one (LLR test, p < 0.01/3). The best model for the face and house group-level ERP was RE Channel. E-H) Same as A-C except for the broadband (BB) power signals. Category beta weight peak latencies were significantly different across face- and hose-selective channels (*, ks-test, p = 8.08e-6). I-L) Average responses for face-selective ERP group-level data (H), house-selective ERP group-level data (I), face-selective broadband power group-level data (J), and house-selective broadband power group-level data (K). Shaded regions indicate significant cluster times from the RE Channel model.

Figure 3:. Influence of Random Effects on Beta Weights Estimates and Significance.

Time Course of beta weights from various group-level models and their significance; top row is the beta weight time courses and bottom row is the associated significance time course: A) ERP face-selective group-level models, B) ERP house-selective group-level models, C) broadband power (BB) face-selective group-level models, and D) broadband power house-selective group-level models. *Note lines are offset slightly for visualization. Black arrows (↕) indicate different results between models.

Figure 4:. CBPT Analysis of Simulated Broadband Power Signals.

A) Proportion of simulations for which LMs and separate t-tests missed or found an extra cluster compared to LMEs (*, χ² proportion test compared to 5%, p = 0.00139). B) Adjusted-R² for various partial and full LM models as well as LME models. C) The proportion of simulations with a potential cluster identified for the novelty effect as a function of novelty beta weight size. We fit this data with logistic regression models for each statistical model for positive (solid lines) and negative (dashed lines) beta weights separately. D) Percent difference in cluster sizes detected by LMs and separate t-tests compared to those detected by LMEs (*, Wilcoxon rank sum-test, p = 4.34e-16). E) The proportion of simulations that benefited from more complicated models compared to less complicated models (LLR test, p < 0.05). F) Detected novelty cluster sizes as function of novelty beta weight magnitude for each statistical model as well as for positive (solid lines) and negative (dashed lines) beta weights.

Figure 5:. Analysis High-Gamma Burst Data using CBPT with GLMEs.

A) Proportion of channels with high-gamma burst rates selective for image category, novelty, or both over time. B) Average category-selective beta weight magnitudes for individual face-, house-selective, and non-selective channels. C) Peak latency (mean +/− s.e.m) of category beta weights for face- and house-selective channels (*, ks-test, p = 0.00852). D) Average burst rate for face-selective group-level data. Shaded regions indicate significant time points after CBPT; the cluster was significant for both category and novelty. E) Average burst rate for house-selective group-level data. Shaded regions indicate significant time points after CBPT; the first cluster was significant for novelty and the second cluster was significant for image category. F) Category beta weight time course for face- and house-selective group-level data. A peak (*) was detected for the face-selective group-level data but not the house-selective group-level data.