Text-related functionality and dynamics of visual human pre-frontal activations revealed through neural network convergence

Abstract

Human prefrontal areas show enhanced activations when individuals are presented with images, under diverse task conditions. However, the functional role of these increased activations remains a deeply debated question. Here we addressed this question by comparing, dynamically, the relational structure of prefrontal activations and both visual and textual-trained deep neural networks (DNNs) during a visual memorization task. We analyzed intra-cranial recordings, conducted for clinical purposes, while patients viewed and memorized images of familiar faces and places. Our results reveal that relational structures in the frontal cortex elicited during visual memorization were predicted by text and not visual DNNs. Importantly, the temporal dynamics of these correlations showed striking differences, with a rapid decline over time for the visual component, but persistent dynamics including a significant image offset response for the text component. The results point to a dynamic text-related function of prefrontal cortex during visual memorization in the human brain.

Fig. 1. Experimental design and general methods.

a Experimental design: participants viewed 28 images of familiar faces or places while their brain activity was recorded via intracranial EEG. b Contact locations: yellow - Fronto-parietal visual contacts, blue— Face-selective ventral contacts, white—visual non-selective contacts, gray- non-visual contacts. The original images used in the experiment are not copyrighted, images in the figure were replaced by licensed images with a similar appearance. Credits: Barack Obama image is from Whitehouse.gov under a Creative Commons license CC BY 3.0, Eiffel Tower image is from Freepik (https://freepick.com).

Fig. 2. Representational dissimilarity matrices.

A Representational Dissimilarity Matrices (RDMs) of the 28 stimuli. RDMs of the embeddings of the images based on visual DNNs (VGG, CLIP-image) and of their textual description based on textual DNNs (CLIP-text, GPT). RDMs of the neural response to the images in face-selective ventral and fronto-parietal visual contacts, based on multivariate responses across contacts. B The correlations between the RDMs of the visual and textual DNNs. The thinner the ellipsoids, the greater the correlation.

Fig. 3. Correlations (Top) and Partial correlations (bottom) of image and text DNNs with Face-selective ventral and Frontal-parietal visual contacts.

Zero order correlations are presented in top panels, and partial correlations with each of the DNNs, when the other three DNNs are held constant, are presented in bottom panels. A Face-selective occipito-temporal contacts (n = 43), Error bars indicate leave one participant out procedure s.e.m. B Fronto-parietal visual contacts (n = 63), Error bars indicate leave one participant out procedure s.e.m. All p values were derived from a pair-images permutation test (10,000 permutations). Reported p values are FDR corrected in each ROI separately. Note the clear shift in bias from visual-based to text-based preference from occipito-temporal to frontal-parietal ROIs. ${**p}_{FDR}<0.01$, ${***p}_{FDR}<0.001$.

Fig. 4. Correlations between VGG or CLIP-Text with Face selective ventral and Frontal-parietal visual contacts at different time windows.

Top bar plots (green bars) show the correlations with VGG in face-selective ventral contacts (left), and fronto-parietal visual contacts (right). Bottom bar plots (red bars) show the correlations with CLIP-Text in face-selective ventral contacts (left), and fronto-parietal visual contacts (right). Note the strikingly different dynamics in fronto-parietal contacts between the visual component that declined rapidly and the text component with persisted followed by an offset response. Error bars indicate leave one participant out procedure s.e.m ${*p}_{FDR}<0.05$, ${**p}_{FDR}<0.01$, ${***p}_{FDR}<0.001$.

Fig. 5. CLIP-Text different layers' correlations with Fronto-parietal visual contacts when embeddings were based on the original text or on shuffled text.

The correlations with the fronto-parietal visual contacts (n = 63) in the first time-window (0.1–0.4 s) with CLIP-Text different layers based on the original text or shuffled text. A The correlations with CLIP-Text representations based on the original text, Error bars indicate leave one participant out procedure s.e.m. B The average correlation with CLIP-Text different layers based on six different variations of shuffled original text. Error bars indicate s.e.m. All p values were derived from a pair-image permutation test (10,000 permutations). ${**p}_{FDR}<0.01$, ${***p}_{FDR}<0.001$.

Fig. 6. Correlations (Top) and Partial correlations (bottom) of human visual and semantic similarity ratings with Face-selective ventral and Frontal-parietal visual contacts.

Zero-order correlations are presented in top panels, and partial correlations between each similarity task RDM (averaged across participants; n = 20 in each task), when the other task RDM is held constant, are presented in bottom panels. A Face-selective occipito-temporal contacts (n = 43), Error bars indicate leave one participant out procedure s.e.m. B Fronto-parietal visual contacts (n = 63), Error bars indicate leave one participant out procedure s.e.m. All p values were derived from a pair-image permutation test (10000 permutations). Reported p values are FDR corrected in each ROI separately. ${**p}_{FDR}<0.01$, ${***p}_{FDR}<0.001$. The original images used in the experiment are not copyrighted, images in the figure were replaced by licensed images with a similar appearance. Credits: Barack Obama image is from Whitehouse.gov under a Creative Commons license CC BY 3.0, Eiffel Tower image is from Freepik (https://freepick.com).