Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Comparative Study
. 2023 Mar 28;13(1):5035.
doi: 10.1038/s41598-023-32248-6.

Overlap in meaning is a stronger predictor of semantic activation in GPT-3 than in humans

Jan Digutsch  1   2 , Michal Kosinski  3
Affiliations
Comparative Study

Overlap in meaning is a stronger predictor of semantic activation in GPT-3 than in humans

Jan Digutsch et al. Sci Rep. .

Abstract

Modern large language models generate texts that are virtually indistinguishable from those written by humans and achieve near-human performance in comprehension and reasoning tests. Yet, their complexity makes it difficult to explain and predict their functioning. We examined a state-of-the-art language model (GPT-3) using lexical decision tasks widely used to study the structure of semantic memory in humans. The results of four analyses showed that GPT-3's patterns of semantic activation are broadly similar to those observed in humans, showing significantly higher semantic activation in related (e.g., "lime-lemon") word pairs than in other-related (e.g., "sour-lemon") or unrelated (e.g., "tourist-lemon") word pairs. However, there are also significant differences between GPT-3 and humans. GPT-3's semantic activation is better predicted by similarity in words' meaning (i.e., semantic similarity) rather than their co-occurrence in the language (i.e., associative similarity). This suggests that GPT-3's semantic network is organized around word meaning rather than their co-occurrence in text.

PubMed Disclaimer

Conflict of interest statement

The authors declare no competing interests.

Figures

Figure 1
Figure 1
Semantic activation across priming conditions.
Figure 2
Figure 2
Semantic activation and prime-target association type. Word pairs in brackets are examples. Error bars represent 95% confidence intervals.
Figure 3
Figure 3
Semantic activation for associatively (red) and semantically (blue) related words. Dotted lines represent semantic activation for the associatively (red) and semantically (blue) related words in humans.
See this image and copyright information in PMC

References

    1. Brown, T. B. et al. (2020). Language models are few-shot learners. arXivhttp://arxiv.org/abs/2005.14165 (2020).
    1. Van Noorden R. How language-generation AIs could transform science. Nature. 2022;605(7908):21–21. doi: 10.1038/d41586-022-01191-3. - DOI - PubMed
    1. DeepL. (n.d.). DeepL SE. https://www.DeepL.com/translator
    1. Bender, E. M., Gebru, T., McMillan-Major, A. & Shmitchell, S. On the dangers of stochastic parrots: Can language models be too big?. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency 610–623. 10.1145/3442188.3445922 (2021).
    1. Lake BM, Murphy GL. Word meaning in minds and machines. Psychol. Rev. 2021 doi: 10.1037/rev0000297. - DOI - PubMed

Publication types