Online searches to evaluate misinformation can increase its perceived veracity

Abstract

Considerable scholarly attention has been paid to understanding belief in online misinformation^1,2, with a particular focus on social networks. However, the dominant role of search engines in the information environment remains underexplored, even though the use of online search to evaluate the veracity of information is a central component of media literacy interventions^3-5. Although conventional wisdom suggests that searching online when evaluating misinformation would reduce belief in it, there is little empirical evidence to evaluate this claim. Here, across five experiments, we present consistent evidence that online search to evaluate the truthfulness of false news articles actually increases the probability of believing them. To shed light on this relationship, we combine survey data with digital trace data collected using a custom browser extension. We find that the search effect is concentrated among individuals for whom search engines return lower-quality information. Our results indicate that those who search online to evaluate misinformation risk falling into data voids, or informational spaces in which there is corroborating evidence from low-quality sources. We also find consistent evidence that searching online to evaluate news increases belief in true news from low-quality sources, but inconsistent evidence that it increases belief in true news from mainstream sources. Our findings highlight the need for media literacy programmes to ground their recommendations in empirically tested strategies and for search engines to invest in solutions to the challenges identified here.

Fig. 1. The effect of searching online to evaluate misinformation on belief in misinformation across studies 1 to 4.

a,b, The average treatment effect of SOTEN and 95% confidence intervals during studies 1 (n = 2,275 total evaluations), 2 (n = 2,020), 3 (n = 1,964) and 4 (n = 772). All effects were estimated using OLS with article fixed effects and standard errors clustered at the individual and article level. a, The effect of SOTEN on rating misinformation as true for studies 1 (P = 0.037), 2 (P < 0.0001), 3 (P = 0.0018) and 4 (P = 0.0451). b, The effect of SOTEN on a seven-point ordinal scale of veracity for studies 1 (P = 0.154), 2 (P = 0.0004), 3 (P = 0.0038) and 4 (P = 0.0054). Source Data

Fig. 2. How Google search results impact belief in misinformation (study 5).

a, The proportion of individuals who, when searching online about a false/misleading (FM) or true article, are exposed to different levels of unreliable news sites in Google search results. b, The average treatment effects and 95% confidence intervals for linear regression models measuring the effect of searching online during study 5 (n = 1,485) as a unit of the standard deviation of the dependent variable. Searching online increased the probability that a respondent rated a false/misleading article as true (P = 0.0143). c,d, The same average treatment effects and 95% confidence intervals, but the treatment group was subset by the quality of news returned in their search engine results. c, The probability that an individual rates misinformation as true is higher in the treatment group compared with the control group among respondents whose exposure consisted of at least 10% unreliable news sites (n = 1,027, P = 0.004). The probability that an individual rates a false/misleading article as true is not different in the treatment group compared to the control group among respondents who were exposed to only very reliable news (n = 940, P = 0.927). d, The probability that an individual rates a false/misleading article as true in the treatment group compared with the control group among respondents who were exposed to the lowest quartile of news quality (n = 1,006, P = 0.0241) and the second-lowest quartile of news quality (n = 1,005, P = 0.0116). The probability that an individual rates a false/misleading article as true is not different in the treatment group compared to the control group among respondents who were exposed to the second-highest quartile of news quality (n = 1,006, P = 0.801) and the highest quartile of news quality (n = 1,008, P = 0.420). All effects were estimated using OLS with article fixed effects and standard errors clustered at the individual and article level. Source Data

Fig. 3. Analysis of the individuals who were exposed to unreliable news sites when evaluating misinformation online (study 5).

a, The effect of demographic variables on the probability of exposure to unreliable news sources when searching online about false/misleading news articles and the 95% confidence intervals during study 5 (n = 501). b, The proportion of Google searches by individuals (n = 930) that return varying numbers of unreliable news sites, when searching online about a false/misleading article. We present these proportions for individuals who used the headline of the article or the link of the article and those who used another query. c, The effect of demographic variables on the probability of using the headline/lede or unique URL when searching online about false/misleading news articles and the 95% confidence intervals during study 5 (n = 930). Those with lower levels of digital literacy are more likely to use the headline or the unique URL of the false article as their search query when SOTEN, conditioning on ideological congruence and demographics. All effects were estimated using OLS with article fixed effects and standard errors were clustered at the individual and article level. Source Data

Fig. 4. The effect of SOTEN on belief in false/misleading and true news.

a, The effect of searching online on whether individuals rate true news as true and false/misleading news as true and the 95% confidence intervals during studies 1 (n = 6,269, n = 2,275), 2 (n = 6,046, n = 2,020), 3 (n = 5,098, n = 1,964), 4 (n = 1,420, n = 772) and 5 (n = 3,141, n = 1,485). b, The effect of searching online on whether individuals rate true news as true from low-quality sources, true news as true from mainstream sources and false/misleading news as true, and the 95% confidence intervals during studies 1 (n = 2,782, n = 3,487, n = 2,275), 2 (n = 2,596, n = 3,450, n = 2,020), 3 (n = 2,490, n = 3,418, n = 1,964), 4 (n = 516, n = 904, n = 772) and 5 (n = 1,350, n = 1,791, n = 1,485). c, The effect of searching online on whether individuals rate true news as true from low-quality sources, true news as true from mainstream sources and false/misleading news as true, and the 95% confidence intervals for between-respondent experiments (studies 1 and 5) (n = 4,132, n = 5,278, n = 4,756) and within-respondent experiments (studies 2–4) (n = 5,602, n = 7,702, n = 3,760). All effects were estimated using OLS with article fixed effects and standard errors were clustered at the individual and article level. Source Data

Extended Data Fig. 1. The online search effect using different online search instructions (categorical veracity measure).

This figure displays the average treatment effect of SOTEN on rating a false/misleading article as true and 95 percent confidence intervals using different online search instructions in Study 6. It shows that the effect of searching online increases the probability of rating a false/misleading article as true regardless of the instructions given to respondents. When comparing the control group (N = 1,113) to treatment group 1 (N = 1,075; the same instructions used in Studies 1–5), treatment group 2 (N = 1,034; limited instructions), and treatment group 3 (N = 1,036; no instructions), searching online increased the likelihood of rating false/misleading news as true by 0.09 (P = 0.0027), 0.05 (P = 0.0389), and 0.05 (P = 0.0021) respectively. The effects of online search were similar for true news from mainstream sources and true news from low-quality sources. All effects are estimated using ordinary least squares with article fixed effects and standard errors clustered at the individual and article level. Source Data

Extended Data Fig. 2. Search effect across self-reported political ideology.

This figure presents the effect of searching online on rating a false/misleading article as true and 95 percent confidence intervals subset by political ideology (Liberal, Moderate, and Conservative) in during Studies 1 (N = 780, N = 712, N = 783), 2 (N = 664, N = 670, N = 686), 3 (N = 700, N = 594, N = 670), 4 (N = 270, N = 226, N = 276), 5 (N = 757, N = 465, N = 249). Generally, the effect sizes are quite similar across political ideological groups. All effects are estimated using ordinary least squares with article fixed effects and standard errors clustered at the individual and article level. Source Data

Extended Data Fig. 3. Who is most susceptible to unreliable information when searching for more information? (Study 5).

Panels a through b present the effect of searching online on rating a false/misleading article as true and 95 percent confidence intervals during Study 5 as a unit of the standard deviation of the dependent variable. Marginal effects are subset by the quality of news returned in their search engine results (top 50% and bottom 50% of average source quality of news returned). Panel a presents the effect of being encouraged to search online among those ideologically congruent with the ideological perspective of the item of misinformation they are evaluating (N = 562, N = 320), while panel b presents the search effect of being encouraged to search online among those ideologically incongruent with the ideological perspective of the item of misinformation they are evaluating (N = 790, N = 428). All effects are estimated using ordinary least squares with article fixed effects and standard errors clustered at the individual and article level. Source Data

Extended Data Fig. 4. How does news returned in Google search results affect belief in misinformation? (Study 5) - excluding original article in search results analysis.

Panel a presents the proportion of individuals who, when searching online about a false/misleading or true article, are exposed to different levels of unreliable news sites in Google search results. Panel b presents the average treatment effects and 95 percent confidence intervals for linear regression models measuring the effect of searching online during Study 5 (N = 1,485) as a unit of the standard deviation of the dependent variable. Searching online increased the probability a respondent rated a false/misleading article as true. Subsetting the treatment group by the quality of news returned in their search engine results, Panel c and d present these same average treatment effects and 95 percent confidence intervals. Panel c shows that the probability an individual rates misinformation as true is higher than the control group among respondents who are exposed to at least one unreliable news site (N = 986). The probability an individual a false/misleading article as true is not different than the control group among respondents who are exposed to only very reliable news (N = 958). Panel d shows that the probability an individual rates a false/misleading article as true than the control group among respondents who are exposed to the lowest quartile of news quality (N = 1,006) and second lowest quartile of news quality (N = 1,005). The probability an individual rates a false/misleading article as true is not different than the control group among respondents who are exposed to the second highest quartile of news quality (N = 1,005) and the highest quartile of news quality (N = 1,006). All effects are estimated using ordinary least squares with article fixed effects and standard errors clustered at the individual and article level. Source Data

Extended Data Fig. 5. Who is exposed to unreliable news sites when evaluating misinformation online? (Study 5) - Excluding original article in search results analysis.

Panel a presents the effect of demographic variables on the probability of exposure to unreliable news sources when searching online about false/misleading news articles and 95 percent confidence intervals during Study 5 (N = 498). Panel b presents the proportion of online searches individuals engage in (N = 930) when searching online about a false/misleading article returns different levels of unreliable news sites by the Google search engine. We present these proportions for those who use the headline of the article or the link of the article and those who use another query. Panel c presents the effect of demographic variables on the probability of using the headline/lede or unique URL when searching online about false/misleading news articles and 95 percent confidence intervals during Study 5 (N = 930). All effects are estimated using ordinary least squares with article fixed effects and standard errors clustered at the individual and article level. Source Data

Extended Data Fig. 6. Fig. 4 with 7-point ordinal scale.

Panel a presents the effect of rating true news as true and false/misleading news as true and 95 percent confidence intervals using a seven-point ordinal scale during Studies 1 (N = 6,269, N = 2,275), 2 (N = 6,046, N = 2,020), 3 (N = 5,098, N = 1,964), 4 (N = 1,420, N = 772), and 5 (N = 3,141, N = 1,485). Panel b the effect of rating true news as true from low-quality sources, true news as true from mainstream sources, and false/misleading news as true and 95 percent confidence intervals during Studies 1 (N = 2,782, N = 3,487, N = 2,275), 2 (N = 2,596, N = 3,450, N = 2,020), 3 (N = 2,490, N = 3,418, N = 1,964), 4 (N = 516, N = 904, N = 772), and 5 (N = 1,350, N = 1,791, N = 1,485). Panel c presents the effect of rating true news as true from low-quality sources, true news as true from mainstream sources, and false/misleading news as true and 95 percent confidence intervals for between-respondent experiments (Studies 1 and 5) (N = 4,132, N = 5,278, N = 4,756) and within-respondent experiments (Studies 2–4) (N = 5,602, N = 7,702, N = 3,760). Source Data

Extended Data Fig. 7. Effect of SOTEN about false/misleading news articles when unprompted.

This figure presents the effect of SOTEN unprompted on rating a false/misleading article as true and 95 percent confidence intervals using a categorical measure in panel a and a 7-point ordinal scale in panel b for Studies 1 (N = 1,145), 2 (N = 1,010), 3 (N = 982), 4 (N = 386, and all four studies pooled together (N = 3,523). When pooled together we observe that searching online increases rating a false/misleading news article by 0.086 (P = 0.0102, Cohen’s D = 0.18, N = 3,523) and increases perceived veracity on a seven-point ordinal scale by 0.278 (P = 0.0463, Cohen’s D = 0.16, N = 3,523). Source Data

Extended Data Fig. 8. The effect of searching online on belief in misinformation across Study 1 through 4 using the preregistered models.

Panels a and b present the average treatment effect of SOTEN on rating false/misleading articles as true and 95 percent confidence intervals during Studies 1 (N = 2,275), 2 (N = 2,020), 3 (N = 1,964), and 4 (N = 772) using our preregistered model, which only clustered standard errors at the respondent level. All effects are estimated using ordinary least squares with article fixed effects and standard errors clustered at the individual level. Panel a presents the effect of SOTEN on rating misinformation as true and 95 percent confidence intervals for Study 1 (P < 0.0001), 2 (P < 0.0001), 3 (P < 0.0001), and 4 (P < 0.0001). Panel b presents the effect of SOTEN on a 7-point ordinal scale of veracity and 95 percent confidence intervals for Study 1 (P < 0.0001), 2 (P < 0.0001), 3 (P < 0.0001), and 4 (P < 0.0001). Source Data

Extended Data Fig. 9. Predicting seven-point ordinal scale with categorical rating for evaluation of false/misleading articles.

We predict the rating of an article on a 7-point scale using our categorical measure using a simple linear regression (ordinary least squares). This figure presents the effect of rating a false/misleading article as true on rating a false/misleading article as true on the 7-point ordinal scale and 95 percent confidence intervals. Pre-treatment, rating a false/misleading article as true increases the 7-point ordinal scale by 2.51 (Study 1; N = 1,145; P < 0.0001), 2.62 (Study 2; N = 1,010; P < 0.0001), 2.78 (Study 3; N = 982; P < 0.0001), 2.52 (Study 4; N = 386; P < 0.0001), and 2.74 (Study 5; N = 877; P < 0.0001). Post-treatment, rating a false/misleading article as true increases the 7-point ordinal scale by 2.86 (Study 1; N = 1,130; P < 0.0001), 2.52 (Study 2; N = 1,010; P < 0.0001), 2.76 (Study 3; N = 982; P < 0.0001), 2.61 (Study 4; N = 386; P < 0.0001), and 3.02 (Study 5; N = 608; P < 0.0001). Source Data

Extended Data Fig. 10. Does the effect of SOTEN increase with fact-checker agreement across Studies 1 through 5?.

Panels a and b present the change (with 95 percent confidence intervals) in the effect of searching online on rating a false/misleading article as true when fact-checker agreement increases from 0 to 1 during Studies 1 (N = 2,275), 2 (N = 2,020), 3 (N = 1,964), 4 (N = 772) and 5 (N = 1,485) using a categorical scale (Panel a) and a 7-point ordinal scale (Panel b). Panels c and d present the change (with 95 percent confidence intervals) in the effect of searching online on rating a false/misleading article as true when fact-checker agreement increases from 0 to 1 during between-respondent experiments (Studies 1 and 5; N = 3,760) and within-respondent experiments (Studies 2, 3, and 4; N = 4,756) and 5 (N = 1,485) using a categorical scale (Panel c) and a 7-point ordinal scale (Panel d). Source Data