Neuroimaging-based classification of PTSD using data-driven computational approaches: A multisite big data study from the ENIGMA-PGC PTSD consortium

Abstract

Background: Recent advances in data-driven computational approaches have been helpful in devising tools to objectively diagnose psychiatric disorders. However, current machine learning studies limited to small homogeneous samples, different methodologies, and different imaging collection protocols, limit the ability to directly compare and generalize their results. Here we aimed to classify individuals with PTSD versus controls and assess the generalizability using a large heterogeneous brain datasets from the ENIGMA-PGC PTSD Working group.

Methods: We analyzed brain MRI data from 3,477 structural-MRI; 2,495 resting state-fMRI; and 1,952 diffusion-MRI. First, we identified the brain features that best distinguish individuals with PTSD from controls using traditional machine learning methods. Second, we assessed the utility of the denoising variational autoencoder (DVAE) and evaluated its classification performance. Third, we assessed the generalizability and reproducibility of both models using leave-one-site-out cross-validation procedure for each modality.

Results: We found lower performance in classifying PTSD vs. controls with data from over 20 sites (60 % test AUC for s-MRI, 59 % for rs-fMRI and 56 % for d-MRI), as compared to other studies run on single-site data. The performance increased when classifying PTSD from HC without trauma history in each modality (75 % AUC). The classification performance remained intact when applying the DVAE framework, which reduced the number of features. Finally, we found that the DVAE framework achieved better generalization to unseen datasets compared with the traditional machine learning frameworks, albeit performance was slightly above chance.

Conclusion: These results have the potential to provide a baseline classification performance for PTSD when using large scale neuroimaging datasets. Our findings show that the control group used can heavily affect classification performance. The DVAE framework provided better generalizability for the multi-site data. This may be more significant in clinical practice since the neuroimaging-based diagnostic DVAE classification models are much less site-specific, rendering them more generalizable.

Keywords: Classification; Deep learning; Machine learning; Multimodal MRI; Posttraumatic stress disorder.

Fig. 1.

Brain features from structural MRI (s-MRI), resting state fMRI (rs-fMRI), and DTI (d-MRI) used in this study. T1-weighted images were processed using the FreeSurfer pipeline, the final s-MRI features included 96 ROI cortical thicknesses (CT) and volumes for both left and right hemispheres. Rs-fMRI images were preprocessed using ENIGMA HALFpipe workflow, the final rs-fMRI features included 10,878 ROI-to-ROI functional connectivity measures. DTI data were preprocessed following ENIGMA-DTI protocols, 42 Tract-Based Spatial Statistics (TBSS) derived features of mean FA were included in the analysis.

Fig. 2.

Overall analysis procedure. The analysis followed the structure of the three main aims and 2 supplementary aims of the paper. Goals 1 and 2 used data that was pooled across sites/scanners whereas goal 3 used site information to facilitate performance assessment. Goal 1, investigated both Support vector machine (SVM) and random forest (RF) for classification of s-MRI data, which was repeated for rs-fMRI, and then for D-MRI. Goal 2 investigated DVAE in conjunction with SVM and in conjunction with RF (DVAE+SVM/RF), first using s-MRI features and then using rs-fMRI features. Goal 3 investigated performance of SVM or RF on single site data that was tested separately on s-MRI, rs-fMRI, and D-MRI brain features. Goal 3 also investigated performance of LOSOCV tested separately on s-MRI (SVM, DVAE+SVM), rs-fMRI (SVM, DVAE+SVM), and D-MRI (SVM) brain features. S-MRI: structural MRI; RS-fMRI: resting state fMRI; D-MRI: diffusion MRI; SVM: support vector machine; RF: random forest; DVAE: Denoising variational autoencoder.

Fig. 3.

Denoising Variational Autoencoder analysis pipeline: The model was trained using rs-fMRI or s-MRI data from controls only. The samples were then split into a training+validation (70 %) and independent-test (30 %) data. Then 20 % of the training data was set aside for validation and hyperparameter tuning. Once the training+validation was completed, the model’s performance was evaluated on the independent-test data, which provides an unbiased estimate of how the model generalizes to unseen data. The resulting VAE model learned to encode healthy patterns from the input brain features into its latent representation. Later, the brain features from patients with PTSD (PTSD test set) were input into the same VAE model, and the latent variables were extracted as new features for classification analysis.

Fig. 4.

The overall classification performance (measured by cross validation AUC [CV AUC], and test AUC) between PTSD and all controls, between PTSD and HC, and between PTSD and TEHC, for s-MRI, rs-fMRI, and D-MRI. Error bar represents standard deviation of the 10 fold cross validation results.

Fig. 5.

Compare classification performance between PTSD and Controls using all features (labeled as RF or SVM) and DVAE-based latent variables (labeled as DVAE+RF or DVAE+SVM) in s-MRI (A) and rs-fMRI (B). Compared with the performance (CV AUC) using SVM of all features, the performance of DVAE+SVM (CV AUC) significantly improved for both s-MRI and rs-fMRI.

Fig. 6.

The reconstruction loss function of the Denoising Variational Autoencoder model for s-MRI (A), rs-fMRI (B), and D-MRI (C), blue line: loss for the training set (from control data), orange line: loss for the validation set (from control data), green line: loss for the validation data (from PTSD data).

Fig. 7.

s-MRI, rs-fMRI, and D-MRI single site performance for classification of PTSD from controls using SVM. The classification performance was measured by cross validation (CV) AUC, the dot indicates the average of the AUC of each fold in cross validation for each site, the line indicates the standard deviation of each fold in cross validation for each site. The boxplots were made by utilizing the boxplot() function from the seaborn library in Python. The box encompasses the interquartile interval, or the middle 50 % of the dataset. The upper and lower whiskers represent data points located in the top and bottom 25 % of the dataset. Data that fall outside this range are considered outliers and are plotted individually.

Fig. 8.

The comparison of Leave One Site Out Cross Validation (LOSOCV) performance with aggregated pooling method on the independent-test data using SVM for classification between PTSD and Controls across s-MRI (T1), rs-fMRI (RS), LOSOCV: In each iteration, one independent test site was completely left out from the training partition. Then the training set was further randomly partitioned into 10 subfolds for cross validation. Predictive performance was evaluated on the data from the hold-out site (presented light gray in the figure). Aggregated pooling method: data from all sites was included in the training process. We randomly split all imaging data into two subsets: 70 % of the data was used for training+validation (cross validation), and the remaining 30 % was used as independent test data. Random under-sampling was applied to the imbalanced groups, with the under-sampling transform applied to the training dataset on each split of a repeated 10-fold cross validation. Predictive performance was evaluated on the data from the independent test set (presented dark gray in the figure).