Exploring the reproducibility of functional connectivity alterations in Parkinson's disease

Abstract

Since anatomic MRI is presently not able to directly discern neuronal loss in Parkinson's Disease (PD), studying the associated functional connectivity (FC) changes seems a promising approach toward developing non-invasive and non-radioactive neuroimaging markers for this disease. While several groups have reported such FC changes in PD, there are also significant discrepancies between studies. Investigating the reproducibility of PD-related FC changes on independent datasets is therefore of crucial importance. We acquired resting-state fMRI scans for 43 subjects (27 patients and 16 normal controls, with 2 replicate scans per subject) and compared the observed FC changes with those obtained in two independent datasets, one made available by the PPMI consortium (91 patients, 18 controls) and a second one by the group of Tao Wu (20 patients, 20 controls). Unfortunately, PD-related functional connectivity changes turned out to be non-reproducible across datasets. This could be due to disease heterogeneity, but also to technical differences. To distinguish between the two, we devised a method to directly check for disease heterogeneity using random splits of a single dataset. Since we still observe non-reproducibility in a large fraction of random splits of the same dataset, we conclude that functional heterogeneity may be a dominating factor behind the lack of reproducibility of FC alterations in different rs-fMRI studies of PD. While global PD-related functional connectivity changes were non-reproducible across datasets, we identified a few individual brain region pairs with marginally consistent FC changes across all three datasets. However, training classifiers on each one of the three datasets to discriminate PD scans from controls produced only low accuracies on the remaining two test datasets. Moreover, classifiers trained and tested on random splits of the same dataset (which are technically homogeneous) also had low test accuracies, directly substantiating disease heterogeneity.

Fig 1. Overview.

(A) Main steps of the analysis. (B) Using random splits of a dataset with replicate scans to check for disease (group) heterogeneity: (right) by placing different subjects (with all their replicate scans) in the two splits (“split subjects”) and respectively (left) by splitting the replicates of the same subjects in the two splits (“split replicates”). (a,b,c,… correspond to subjects, while, for instance, a’ and a” are replicate scans for subject a).

Fig 2. Scatter-plots of ROI-pair t-values for the three dataset pairs indicate non-reproducibility of global PD-related FC changes.

Fig 3. Inconsistent reproducibility of PD-related FC changes in random heterogeneous dataset splits and consistent reproducibility in random homogeneous dataset splits.

(A) Complementary cumulative distribution function (CCDF = 1-CDF) of the reproducibility p-values for N_s = 2510 random heterogeneous splits and N_s = 325 random homogeneous splits. (B) CCDF of the reproducibility measure R. (C) A scatter-plot of ROI-pair t-values for a random heterogeneous split. (D) A scatter-plot of ROI-pair t-values for a random homogeneous split.

Fig 4. Reproducibility when changing various technical factors or preprocessing options.

Fig 5. Marginally significant FC changes w.r.t. all 3 datasets.

The ROIs were mapped onto the brain surface using BrainNet Viewer [42] (http://www.nitrc.org/projects/bnv/).

Fig 6. Average accuracies for classifiers trained on dataset 1 and tested on dataset 2 for all dataset pairs using standard preprocessing (‘standard’), global signal regression (GS) and respectively bandpass filtering (0.01–0.1Hz).

Here, SVM (linear Support Vector Machine) and GNB (Gaussian Naïve Bayes) classifiers used N = 5000 features—see Fig B in S1 File for classifier accuracies for varying N. As an example, NEUROCON-PPMI denotes classifiers trained on NEUROCON and tested on PPMI data.

Fig 7. Aggregated average accuracies for classifiers trained on each of the 3 datasets using standard preprocessing (‘standard’), global signal regression (GS) and respectively bandpass filtering (0.01–0.1Hz).

Classifiers were trained with N = 10,50,100,500,5000 features. As an example, NEUROCON(10) refers to the aggregated accuracy (Aacc(NEUROCON-PPMI) + Aacc(NEUROCON-TaoWu))/2 for classifiers trained on NEUROCON and tested on PPMI and respectively TaoWu data using N = 10 features. SVM—linear SVM classifier, GNB—Gaussian Naïve Bayes classifier.

Fig 8. Average accuracies for classifiers trained and tested on split data from the same dataset using standard preprocessing (‘standard’), global signal regression (GS) and respectively bandpass filtering (0.01–0.1Hz).

An SVM classifier with N = 5000 features was used.