Examining Chat GPT with nonwords and machine psycholinguistic techniques

Abstract

Strings of letters or sounds that lack meaning (i.e., nonwords) have been used in cognitive psychology and psycholinguistics to provide foundational knowledge of human processing and representation, and insights into language-related performance. The present set of studies used the machine psycholinguistic approach (i.e., using nonword stimuli and tasks similar to those used with humans) to gain insight into the performance of Chat GPT in comparison to human performance. In Study 1, Chat GPT was able to provide correct definitions to many extinct words (i.e., real English words that are no longer used). In Study 2 the nonwords were real words in Spanish, and Chat GPT was prompted to provide a word that sounded similar to the nonword. Responses tended to be Spanish words unless the prompt specified that the similar sounding word should be an English word. In Study 3 Chat GPT provided subjective ratings of wordlikeness (and buyability) that correlated with ratings provided by humans, and with the phonotactic probabilities of the nonwords. In Study 4, Chat GPT was prompted to generate a new English word for a novel concept. The results of these studies highlight certain strengths and weaknesses in human and machine performance. Future work should focus on developing AI that complements or extends rather than duplicates or competes with human abilities. The machine psycholinguistic approach may help to discover additional strengths and weaknesses of human and artificial intelligences.