Generating synthetic task-based brain fingerprints for population neuroscience using deep learning

Abstract

Task-based functional magnetic resonance imaging (fMRI) reveals individual differences in neural correlates of cognition but faces scalability challenges due to cognitive demands, protocol variability, and limited task coverage in large datasets. Here, we propose DeepTaskGen, a deep-learning approach that synthesizes non-acquired task-based contrast maps from resting-state (rs-) fMRI. We validate this approach using the Human Connectome Project lifespan data, then generate 47 contrast maps from 7 different cognitive tasks for over 20,000 individuals from UK Biobank. DeepTaskGen outperforms several benchmarks in generating synthetic task-contrast maps, achieving superior reconstruction performance while retaining inter-individual variation essential for biomarker development. We further show comparable or superior predictive performance of synthetic maps relative to actual maps and rs-connectomes across diverse demographic, cognitive, and clinical variables. This approach facilitates the study of individual differences and the generation of task-related biomarkers by enabling the generation of arbitrary functional cognitive tasks from readily available rs-fMRI data.

Fig. 1. Input, DeepTaskGen architecture, and model evaluation on training and independent samples.

a Computation of connectomes: We utilized voxel-to-ROI rs-fMRI connectomes as input. A connectome was constructed for each subject by calculating the full correlation between the averaged time series from 50 ICA-based ROIs and the time series of individual voxels. b DeepTaskGen architecture: Task-contrast maps for various tasks were predicted from rs-fMRI connectomes using our proposed DeepTaskGen architecture. DeepTaskGen is a volumetric U-Net model with attention mechanism that processes the input resting-state connectome through a series of convolutional blocks, each comprising a 3D convolution layer, batch normalization, and a non-linear activation function. By utilizing max pooling, the model compresses images while preserving task-relevant patterns, and then, it up-samples the images to align with the output task contrast maps. The numbers below each block represent the output shape of each block and the number of feature maps (above). The details of the architecture are presented in Supplementary Table 26. c Training sample: We trained and evaluated DeepTaskGen on the HCP Young Adult dataset (n = 958). The figure above shows the reconstruction performance computed by taking Pearson’s correlation between predicted and actual contrast maps for representative contrasts from seven distinct tasks. The figure below displays the diagonality index (the difference between the on-diagonal and the mean off-diagonal elements in a correlation matrix, normalized by the mean on-diagonal values) on a symmetrical log scale (symlog, threshold = 0.10), evaluating models’ discriminability performance. We compared DeepTaskGen with methods like group-averaged contrast maps, retest scans, and a linear model (each depicted in distinct colors). d Transfer sample: We further fine-tuned the trained DeepTaskGen model on the HCP Development dataset (n = 637) using either task contrasts (e.g., GAMBLING REWARD), and predicted the other contrast (e.g., EMOTION FACES-SHAPES). The fine-tuned model was compared to the non-fine-tuned DeepTaskGen and linear models (shown in distinct colors). Reconstruction performance and discriminability were again used to assess the models’ performance for each task contrast. In boxplots, the box ranges from the first quartile to the third quartile, with a line inside indicating the median. The “whiskers” extend to the most extreme values within 1.5 times the interquartile range, which are not considered outliers. Any points outside this range are plotted individually as outliers.

Fig. 2. Reconstruction and discriminability performances of volumetric- and surface-based methods and the relative performance gain of deep-learning-based methods over the linear method.

Reconstruction performance and diagonality index (discriminability) of volumetric- and surface-based methods and baselines for the 7 distinct task contrasts from HCP-YA. The same set of subjects was used for both volumetric- and surface-based methods. BrainSurfCNN: Final model presented in Ngo et al. . a Raw performance scores of several volumetric- and surface-based baselines and generative methods. b Relative reconstruction performance and diagonality index scores of DeepTaskGen and BrainSurfCNN compared to the linear method. Relative scores are computed against the corresponding baseline linear regression model’s performance, indicating the models’ performance gain or loss compared to the baseline linear model (i.e., Relative Performance = (Model Performance–Baseline Performance)/(Baseline Performance) * 100). Positive values indicate a performance gain over the linear regression model, while negative values indicate a loss. The raw and relative performance figures for all 47 task contrasts were given in Supplementary Figs. 3–10. In boxplots, the box ranges from the first quartile to the third quartile, with a line inside indicating the median. The “whiskers” extend to the most extreme values within 1.5 times the interquartile range, which are not considered outliers. Any points outside this range are plotted individually as outliers.

Fig. 3. Visualization of single-subject and group-level contrast maps, as well as similarity (or dissimilarity) among subjects.

a Unthresholded and thresholded task activations for GAMBLING REWARD contrasts are presented for sample atypical and typical subjects. In each task contrast map, the left column represents an atypical subject, while the right column represents a typical subject, defined by their similarity to the corresponding group average task activations. For each method, unthresholded task activations are displayed at the top, and thresholded activations (top 25% most activated voxels) are shown at the bottom. Dice AUC scores for unthresholded maps and Dice scores for thresholded maps are provided below the corresponding images. The circled areas show activation patterns replicated better in DeepTaskGen compared to linear regression and group-average task contrasts. Sample atypical and typical subject images for six additional task contrasts are given in Supplementary Figs. 12–18. b Group-level contrast maps for seven representative tasks were generated for predicted maps on three datasets (HCP Young Adult, HCP Development, and UK Biobank) and compared to actual task contrast maps from HCP-YA. Activations and deactivations are displayed using scaled z-scores, with colors indicating the magnitude of the effect (shown in the color gradient bar). c To visualize within- and between-task variance among subjects, we employed Uniform Manifold Approximation and Projection (UMAP), a non-linear dimensionality reduction technique. UMAP projects high-dimensional task contrast maps into two dimensions for clear visualization, allowing us to assess if the projected datasets contain individual- and task-specific information necessary for downstream analyses. Each dot represents a subject’s task contrast map for the seven tasks, colored accordingly. Similar subjects are positioned closer together, indicating similarity, while distant dots represent dissimilarity. Accordingly, wider spread within tasks and greater distance between tasks represent higher within- and between-task variability. Note that we fitted a separate UMAP model to each dataset, as indicated in each column and row.

Fig. 4. Prediction of subjects’ age, sex, fluid intelligence, and dominant hand grip strength using task contrast maps and resting-state connectome on UK Biobank.

We predicted subjects’ demographic and cognitive measures in the UK Biobank dataset using three modalities: resting-state connectome, actual contrast map from the EMOTION task, and seven synthetic task contrast maps. All predictions were made using L2 regularized regression (i.e., ridge regression) within a 5-fold cross-validation framework with permutation testing $\left(P=1000\right)$. Blue represents resting-state connectome and actual task contrast maps, while red represents predicted task contrast maps. Note that only the EMOTION task has both actual and predicted contrast maps on UKB, indicated in blue and red, while the remaining six tasks are indicated in red as UKB does not provide them. Significant predictions based on permutation testing are highlighted. Colored horizontal lines indicate mean prediction performance across CV folds. Balanced accuracy was used for sex classification, while Pearson’s correlation assessed the other variables. Sample sizes for all analyses are indicated in each figure. An asterisk (*) indicates a significant difference in prediction performance between annotated maps and resting-state connectome data, while two asterisks (**) represent a significant difference between the annotated maps and actual task-contrast maps. Detailed test statistics are provided in Supplementary Tables 16, and 18–21.

Fig. 5. Prediction of subjects’ clinical measures using task contrast maps and resting-state connectome on UK Biobank.

We predicted subjects’ clinical measures in the UK Biobank dataset using three modalities: synthetic task contrast maps, actual task contrast maps, and resting-state connectome data. resting-state connectome, actual contrast map from the EMOTION task, and seven synthetic task contrast maps. All predictions were made using L2 regularized regression (i.e., ridge regression) within a 5-fold cross-validation framework. Permutation testing $\left(P=1000\right)$ was used to assess the significance of out-of-sample performance against a null distribution. Actual and synthetic brain measures are depicted in blue and red colors, respectively. Note that only the EMOTION task has both actual and predicted contrast maps on UKB, indicated in blue and red, while the remaining six tasks are indicated in red as UKB does not provide them. Significant predictions based on permutation testing are highlighted. Colored horizontal lines indicate mean prediction performance. Balanced accuracy was used to measure hypertension classification performance, while Pearson’s correlation was employed to assess other variables. Sample sizes for all analyses are indicated in each figure. Test statistics were only performed between predictions surviving permutation testing. The detailed test statistics are given in Supplementary Tables 16, 17, and 22–24. The results for the additional clinical measures are depicted in Supplementary Fig. 19.