A systematic framework for functional connectivity measures

Abstract

Various methods have been proposed to characterize the functional connectivity between nodes in a network measured with different modalities (electrophysiology, functional magnetic resonance imaging etc.). Since different measures of functional connectivity yield different results for the same dataset, it is important to assess when and how they can be used. In this work, we provide a systematic framework for evaluating the performance of a large range of functional connectivity measures-based upon a comprehensive portfolio of models generating measurable responses. Specifically, we benchmarked 42 methods using 10,000 simulated datasets from 5 different types of generative models with different connectivity structures. Since all functional connectivity methods require the setting of some parameters (window size and number, model order etc.), we first optimized these parameters using performance criteria based upon (threshold free) ROC analysis. We then evaluated the performance of the methods on data simulated with different types of models. Finally, we assessed the performance of the methods against different levels of signal-to-noise ratios and network configurations. A MATLAB toolbox is provided to perform such analyses using other methods and simulated datasets.

Keywords: Granger causality; evaluation framework; fMRI; functional connectivity; neural mass models.

Figure 1

We benchmarked 42 methods against more than 10⁴ simulated datasets from 5 types of models with different connectivity structures. Connection strengths, noise levels and time delays were varied to test the robustness of the methods. The same procedure was used to identify the proper range of parameters specifically used in the different methods. The performance of a given method was evaluated by comparing the computed graph to the ground truth structure.

Figure 2

(A) Examples of signals generated by the 5 types of models in the same 5 nodes structure (right part of the middle panel). The y-axis units represent the node number and the x-axis units are time-series in seconds with different sampling rates and time scales. (B) Left: “ground truth” connectivity matrix, which is another representation of the connectivity structure shown in (A). The color code corresponds to the strength of the connections between nodes (set to 1 for all connections in this example). Connection matrix (CM1) was obtained with PDC method. ROC 1 is the receiver operating characteristic (ROC) curve, which displays true positive rate as a function of false positive rate, for the PDC method, measured for all connection strength values (strength threshold). The AUC is the area under the curve. If AUC is 1 (as for CM2 and ROC2), it means that the method is able to find the “ground truth,” if given the correct threshold. (C) For the undirected methods, such as BCorrU, the computed connection matrices are symmetric, thus we took the symmetrized designed matrix as a ground truth for calculating their ROC and AUC.

Figure 3

Categorization of the 42 connectivity analysis methods used here. The methods yielding undirected structures are highlighted in pink and the methods yielding directed structures are highlighted in green.

Figure 4

Effect of the size of the analysis windows on the performance of the methods. (A) AUC arrays of 42 methods for NMM signals as a function of window size [from 50 (0.4 s) to 800 (6 s)] for different connection strengths (CSs from 0.7 to 1). (B) AUC arrays of 42 methods for fMRI signals as a function of window size [from 50 (100 s) to 600 (1200 s)] for different connection strengths (CSs from 0.6 to 0.9). The color scale shows the corresponding AUC values.

Figure 5

Effect of the number of windows: (A) for NMM signals and (B) for fMRI signals. The upper panels show the AUC arrays of 42 methods as a function of the number of windows for 150 (1.2 s for NMM and 300 s for fMRI) and 350 time point windows (2.8 s for NMM and 700 s for fMRI). The bottom panels show the minimum numbers of windows as a function of window size for 7 methods out of the 42. Panels (C,D) show the minimum length of analyzed signal (in time points) to find the correct structure as the function of window size for NMM and fMRI signals, respectively. The methods (color code) are the same as in the panel above.

Figure 6

Effect of frequency bands for NMM signals. (A) Time frequency analysis of NMM signals using a wavelet transform. (B) Repartition of the 15 frequency bands. (C) Average AUC arrays of 21 methods (frequency domain) over 154 structures as the function of the frequency bands defined in (B) for different connection strengths.

Figure 7

Effect of frequency bands for fMRI signals. (A) Time frequency analysis of fMRI signals using a wavelet transform. (B) Repartition of the 15 frequency bands. (C) Average AUC arrays of 21 frequency methods over 154 structures as a function of the frequency bands defined in (B) for different connection strengths.

Figure 8

Effect of the maximum lags and model orders. From left to right, we considered datasets with different signal delays [from 8 (64 ms) to 12 (96 ms)]. Then, we calculated the average AUC arrays of 38 methods over 50 structures as a function of different computed delays (maximum lags for model-free and transfer entropy families and model orders for Granger and families).

Figure 9

Performance of the 42 methods for NMM signals for different connection strengths. (A) Average AUC arrays of 42 methods over different structures as a function of connection strength. (B) Distribution of AUC values for all 154 structures given by the 42 methods when the connection strength is set to 0.6. The central marks are the median, the edges of the box are the 25 and 75th percentiles, the whiskers extend to the most extreme data points not considered outliers, and outliers are plotted individually (plus signs).

Figure 10

Average AUC arrays of the 42 methods over different structures as a function of connection strength for (A) fMRI, (B) linear (C) Rössler and (D) Hénon.

Figure 11

Robustness of the methods against system noise. (A) AUC array of the 42 methods as a function of the ratio between the amplitude of system noise and the amplitude of inputs for both NMM (left) and fMRI signals (right). (B) Examples of signals generated with different amplitudes of system noises (100, 900, and 2100 times more than the amplitude of the input signals).

Figure 12

Robustness of the 42 methods against the observation noise for NMM signals. The figure shows average AUC arrays of the 42 methods as a function of the SNR (signal-to-noise ratio). (A) Each channel receives a different noise (B) All channels receive the same noise.

Figure 13

Robustness of the 42 methods against the observation noise for fMRI signals. The figure shows average AUC arrays of the 42 methods as a function of SNR. (A) Each channel receives a different noise (B) All channels receive the same noise.

Figure 14

(A) The optimized parameters and relative performance are summarized for 6 chosen methods, each from 6 different families. Note that 1: Numbers of windows are obtained by taking a 2.8 s window for NMM and 700 s for fMRI. 2. The first line is the length of time series while the second line shows the window size. Both numbers of windows and length of time series are based on the AUC> 0.999. 3. Results are obtained for a signal delay of 8 (64 ms). (B) The minimal SNR for 8 types of noise by 6 chosen methods on both NMM and fMRI according to AUC> 0.95.

Figure 15

Computational time for the different method families. (A) The computational time (in seconds) is a function of the number of nodes for correlation and Granger families. Green points are the tested values and red bars show the distributions from the different trials. The fitted curves are shown in blue and the mathematical equations for all method families are shown in (B). (C) Computation time in second as a function of numbers of nodes and method families.