LLM-generated messages can persuade humans on policy issues

Abstract

The emergence of large language models (LLMs) has made it possible for generative artificial intelligence (AI) to tackle many higher-order cognitive tasks, with critical implications for industry, government, and labor markets. Here, we investigate whether existing, openly-available LLMs can be used to create messages capable of influencing humans' political attitudes. Across three pre-registered experiments (total N = 4829), participants who read persuasive messages generated by LLMs showed significantly more attitude change across a range of policies - including polarized policies, like an assault weapons ban, a carbon tax, and a paid parental-leave program - relative to control condition participants who read a neutral message. Overall, LLM-generated messages were similarly effective in influencing policy attitudes as messages crafted by lay humans. Participants' reported perceptions of the authors of the persuasive messages suggest these effects occurred through somewhat distinct causal pathways. While the persuasiveness of LLM-generated messages was associated with perceptions that the author used more facts, evidence, logical reasoning, and a dispassionate voice, the persuasiveness of human-generated messages was associated with perceptions of the author as unique and original. These results demonstrate that recent developments in AI make it possible to create politically persuasive messages quickly, cheaply, and at massive scale.

Fig. 1. Participants’ change in policy support by condition across studies.

Y-axes represent the average difference between participants’ post-treatment and pre-treatment policy support (both scaled from 0 to 100, 100 = highest level of support). Higher scores indicate participants became more supportive of the policy. Data are analyzed using regression, two-tailed, and results are presented as mean values with 95% confidence intervals. N_{Study 1} = 1203 participants, N_{Study 2} = 2016 participants, N_{Study 3} = 1610 participants.

Fig. 2. Participants’ perceptions of the author by condition in studies 1 and 2.

Data are analyzed using regression, two-tailed, and results are presented as mean values with 95% confidence intervals. Perceptions of informed, logical, and angry were asked in both studies, whereas unique and vivid story-telling was only assessed in Study 2. N_{Studies 1 and 2} = 3219 participants.