Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2025 Jan 2;15(1):370.
doi: 10.1038/s41598-024-84530-w.

Brain-model neural similarity reveals abstractive summarization performance

Zhejun Zhang #  1 , Shaoting Guo #  1 , Wenqing Zhou  1 , Yingying Luo  1 , Yingqi Zhu  1 , Lin Zhang  1   2 , Lei Li  3
Affiliations

Brain-model neural similarity reveals abstractive summarization performance

Zhejun Zhang et al. Sci Rep. .

Abstract

Deep language models (DLMs) have exhibited remarkable language understanding and generation capabilities, prompting researchers to explore the similarities between their internal mechanisms and human language cognitive processing. This study investigated the representational similarity (RS) between the abstractive summarization (ABS) models and the human brain and its correlation to the performance of ABS tasks. Specifically, representational similarity analysis (RSA) was used to measure the similarity between the representational patterns (RPs) of the BART, PEGASUS, and T5 models' hidden layers and the human brain's language RPs under different spatiotemporal conditions. Layer-wise ablation manipulation, including attention ablation and noise addition was employed to examine the hidden layers' effect on model performance. The results demonstrate that as the depth of hidden layers increases, the models' text encoding becomes increasingly similar to the human brain's language RPs. Manipulating deeper layers leads to more substantial decline in summarization performance compared to shallower layers, highlighting the crucial role of deeper layers in integrating essential information. Notably, the study confirms the hypothesis that the hidden layers exhibiting higher similarity to human brain activity play a more critical role in model performance, with their correlations reaching statistical significance even after controlling for perplexity. These findings deepen our understanding of the cognitive mechanisms underlying language representations in DLMs and their neural correlates, potentially providing insights for optimizing and improving language models by aligning them with the human brain's language-processing mechanisms.

Keywords: Abstractive summarization; Deep language models; Electroencephalography; Neural correlates; Representational similarity analysis.

PubMed Disclaimer

Conflict of interest statement

Declarations. Competing interests: The authors declare no competing interests. Ethical approval declarations: The study was conducted according to the guidelines of the Declaration of Helsinki and approved by the Ethics Committee of the Beijing University of Posts and Telecommunications (Ethic approval code: 202302003).

Figures

Figure 1
Figure 1
Schematic diagram of the research framework.
Figure 2
Figure 2
Procedure of the EEG experiment.
Figure 3
Figure 3
Schematic of representational similarity analysis.
Figure 4
Figure 4
Visualization of RSMs.
Figure 5
Figure 5
RSA scores between each hidden layer of the DLMs and the human brain.
Figure 6
Figure 6
Impact of layer-wise attention ablation and noise addition on ABS task performance.
Figure 7
Figure 7
Correlation between RSA scores and Rouge1 declines in BART, PEGASUS, and T5 models.
See this image and copyright information in PMC

Similar articles

See all similar articles

References

    1. Chang, Y. et al. A survey on evaluation of large language models. ACM Trans. Intell. Syst. Technol.15, 1–45. 10.1145/3641289 (2024).
    1. Zhao, W. X. et al. A survey of large language models, 10.48550/arXiv.2303.18223 (2023). arXiv: 2303.18223.
    1. Antonello, R., Vaidya, A. & Huth, A. Scaling laws for language encoding models in fmri, in Oh, A. et al. (eds.) Advances in Neural Information Processing Systems, vol. 36 21895–21907 (2023). - PMC - PubMed
    1. Arana, S., Pesnot Lerousseau, J. & Hagoort, P. Deep learning models to study sentence comprehension in the human brain. Lang. Cognit. Neurosci.. 10.1080/23273798.2023.2198245 (2023).
    1. Goldstein, A. et al. Alignment of brain embeddings and artificial contextual embeddings in natural language points to common geometric patterns. Nat. Commun.. 10.1038/s41467-024-46631-y (2024). - PMC - PubMed

Publication types

LinkOut - more resources