The Spot the Troll Quiz game increases accuracy in discerning between real and inauthentic social media accounts

Abstract

The proliferation of political mis/disinformation on social media has led many scholars to embrace "inoculation" techniques, where individuals are trained to identify the signs of low-veracity information prior to exposure. Coordinated information operations frequently spread mis/disinformation through inauthentic or "troll" accounts that appear to be trustworthy members to the targeted polity, as in Russia's attempts to influence the 2016 US presidential election. We experimentally tested the efficacy of inoculation against inauthentic online actors, using the Spot the Troll Quiz, a free, online educational tool that teaches how to spot markers of inauthenticity. Inoculation works in this setting. Across an online US nationally representative sample (N = 2,847), which also oversampled older adults, we find that taking the Spot the Troll Quiz (vs. playing a simple game) significantly increases participants' accuracy in identifying trolls among a set of Twitter accounts that are novel to participants. This inoculation also reduces participants' self-efficacy in identifying inauthentic accounts and reduced the perceived reliability of fake news headlines, although it had no effect on affective polarization. And while accuracy in the novel troll-spotting task is negatively associated with age and Republican party identification, the Quiz is equally effective on older adults and Republicans as it was on younger adults and Democrats. In the field, a convenience set of Twitter users who posted their Spot the Troll Quiz results in the fall of 2020 (N = 505) reduced their rate of retweeting in the period after the Quiz, with no impact on original tweeting.

Keywords: Twitter; disinformation; inoculation; misinformation; social media; trolls.

Fig. 1.

Image of the top section of a profile (one of eight) that Quiz takers are asked to read and then guess whether the profile is real or a troll account. Users could scroll down to see several more tweets not visible in Fig. 1.

Fig. 2.

The message Quiz takers get after (correctly) identifying this profile as a troll. The subsequent “signs” page provides Quiz takers with detailed visual information regarding what in this profile served as evidence it was a troll account.

Fig. 3.

Plot of standardized OLS regression estimates from the models testing the stated hypotheses. The top row displays the condition effect of Quiz vs. control on the dependent variables, and the bottom row displays the fixed effect partisan difference for Republicans vs. Democrats in the models testing for condition effects. Bars at 95% CI. *P < 0.05, **P < 0.01, and ***P < 0.001.

Fig. 4.

Total tweets, original tweets, and retweets per day, relative to 6+ days before Quiz. Coefficients and 95% CI from three separate panel regressions including day and account fixed effects.