Uncovering the semantics of concepts using GPT-4

Abstract

The ability of recent Large Language Models (LLMs) such as GPT-3.5 and GPT-4 to generate human-like texts suggests that social scientists could use these LLMs to construct measures of semantic similarity that match human judgment. In this article, we provide an empirical test of this intuition. We use GPT-4 to construct a measure of typicality-the similarity of a text document to a concept. We evaluate its performance against other model-based typicality measures in terms of the correlation with human typicality ratings. We conduct this comparative analysis in two domains: the typicality of books in literary genres (using an existing dataset of book descriptions) and the typicality of tweets authored by US Congress members in the Democratic and Republican parties (using a novel dataset). The typicality measure produced with GPT-4 meets or exceeds the performance of the previous state-of-the art typicality measure we introduced in a recent paper [G. Le Mens, B. Kovács, M. T. Hannan, G. Pros Rius, Sociol. Sci. 2023, 82-117 (2023)]. It accomplishes this without any training with the research data (it is zero-shot learning). This is a breakthrough because the previous state-of-the-art measure required fine-tuning an LLM on hundreds of thousands of text documents to achieve its performance.

Keywords: LLM; categories; chatGPT; deep learning; typicality.

Fig. 1.

Using GPT-4 to measure the genre typicality of a book based on its description: The aggregate typicality measure produced with GPT-4 is highly correlated with the average of human typicality ratings. Leftmost panel: All books in the test data for the Mystery genre. Center Left panel: Separate plots for Mystery and Non-Mystery books. Center Right panel: All books in the test data for the Romance genre. Rightmost panel: Separate plots for Romance and Non-Romance books.

Fig. 2.

Using GPT-4 to measure the typicality of a tweet in a political party: The aggregate typicality measure produced with GPT-4 is highly correlated with the average of human typicality ratings. Leftmost panel: All tweets in the test data for typicality in the Democratic Party. Center left panel: Separate plots for tweets by Democratic and Republican Congress members. Center right panel: All tweets in the test data for typicality in the Republican Party. Rightmost panel: Separate plots for tweets by Democratic and Republican Congress members.