Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2020 Nov 11;5(1):57.
doi: 10.1186/s41235-020-00256-z.

Does the "surprisingly popular" method yield accurate crowdsourced predictions?

Abraham M Rutchick  1 , Bryan J Ross  2 , Dustin P Calvillo  3 , Catherine C Mesick  2
Affiliations

Does the "surprisingly popular" method yield accurate crowdsourced predictions?

Abraham M Rutchick et al. Cogn Res Princ Implic. .

Abstract

The "surprisingly popular" method (SP) of aggregating individual judgments has shown promise in overcoming a weakness of other crowdsourcing methods-situations in which the majority is incorrect. This method relies on participants' estimates of other participants' judgments; when an option is chosen more often than the average metacognitive judgments of that option, it is "surprisingly popular" and is selected by the method. Although SP has been shown to improve group decision making about factual propositions (e.g., state capitals), its application to future outcomes has been limited. In three preregistered studies, we compared SP to other methods of aggregating individual predictions about future events. Study 1 examined predictions of football games, Study 2 examined predictions of the 2018 US midterm elections, and Study 3 examined predictions of basketball games. When applied to judgments made by objectively assessed experts, SP performed slightly better than other aggregation methods. Although there is still more to learn about the conditions under which SP is effective, it shows promise as a means of crowdsourcing predictions of future outcomes.

Keywords: Crowdsourcing; Forecasting; Prediction; Surprisingly popular method; Wisdom of crowds.

PubMed Disclaimer

Conflict of interest statement

The authors declare that they have no competing interests.

Figures

Fig. 1
Fig. 1
Relationships between pairs of prediction methods for Study 1 (total sample). Note sp = surprisingly popular; conf = confidence-weighted; mode = democratic; 538 = Fivethirtyeight.com. Correct predictions are labeled as “c,” and incorrect predictions are labeled as “i,” with the top-left square indicating correct predictions from both methods, the bottom-right square indicating incorrect predictions from both methods, the bottom-left square indicating correct predictions from the left-labeled method but not the top-labeled method, and the top-right square indicating correct predictions from the top-labeled method but not the left-labeled method
Fig. 2
Fig. 2
Relationships between pairs of prediction methods for Study 1 (objectively assessed experts). Note sp = surprisingly popular; conf = confidence-weighted; mode = democratic; 538 = fivethirtyeight.com
Fig. 3
Fig. 3
Relationships between pairs of prediction methods for Study 2 (total sample). Note sp = surprisingly popular; conf = confidence-weighted; mode = democratic; 538 = fivethirtyeight.com
Fig. 4
Fig. 4
Relationships between pairs of prediction methods for Study 2 (self-assessed experts). Note sp = surprisingly popular; conf = confidence-weighted; mode = democratic; 538 = fivethirtyeight.com
Fig. 5
Fig. 5
Relationships between pairs of prediction methods for Study 2 (objectively assessed experts). Note sp = surprisingly popular; conf = confidence-weighted; mode = democratic; 538 = fivethirtyeight.com
Figure 6
Figure 6
Relationships between pairs of prediction methods for Study 3 (total sample). Note sp = surprisingly popular; conf = confidence-weighted; mode = democratic; 538 = fivethirtyeight.com.
See this image and copyright information in PMC

References

    1. Gaissmaier W, Marewski JN. Forecasting elections with mere recognition from small, lousy samples: A comparison of collective recognition, wisdom of crowds, and representative polls. Judgment and Decision Making. 2011;6:73–88.
    1. Kruger J, Dunning D. Unskilled and unaware of it: How difficulties in recognizing one's own incompetence lead to inflated self-assessments. Journal of Personality and Social Psychology. 1999;77:1121–1134. doi: 10.1037/0022-3514.77.6.1121. - DOI - PubMed
    1. Lee MD, Danileiko I, Vi J. Testing the ability of the surprisingly popular method to predict NFL games. Judgment and Decision Making. 2018;13:322–333.
    1. Lee MD, Zhang S, Shi J. The wisdom of the crowd playing the price is right. Memory and Cognition. 2011;39:914–923. doi: 10.3758/s13421-010-0059-7. - DOI - PMC - PubMed
    1. MacIntyre TE, Igou ER, Campbell MJ, Moran AP, Matthews J. Metacognition and action: a new pathway to understanding social and cognitive aspects of expertise in sport. Frontiers in Psychology. 2014;5:1155. doi: 10.3389/fpsyg.2014.01155. - DOI - PMC - PubMed

Publication types

LinkOut - more resources