Predicting the replicability of social and behavioural science claims in COVID-19 preprints

Alexandru Marcoci^#^{1

2}, David P Wilkinson^#^{3

4}, Ans Vercammen^{3

5

6}, Bonnie C Wintle³, Anna Lou Abatayo⁷, Ernest Baskin⁸, Henk Berkman⁹, Erin M Buchanan¹⁰, Sara Capitán¹¹, Tabaré Capitán¹², Ginny Chan¹³, Kent Jason G Cheng¹⁴, Tom Coupé¹⁵, Sarah Dryhurst^{16

17

18}, Jianhua Duan¹⁹, John E Edlund²⁰, Timothy M Errington²¹, Anna Fedor²², Fiona Fidler³, James G Field²³, Nicholas Fox²¹, Hannah Fraser³, Alexandra L J Freeman¹⁷, Anca Hanea^{3

24}, Felix Holzmeister²⁵, Sanghyun Hong¹⁵, Raquel Huggins¹⁰, Nick Huntington-Klein²⁶, Magnus Johannesson²⁷, Angela M Jones²⁸, Hansika Kapoor^{29

30}, John Kerr^{17

31}, Melissa Kline Struhl³², Marta Kołczyńska³³, Yang Liu³⁴, Zachary Loomas²¹, Brianna Luis²¹, Esteban Méndez³⁵, Olivia Miske²¹, Fallon Mody^{3

36}, Carolin Nast³⁷, Brian A Nosek^{21

38}, E Simon Parsons²¹, Thomas Pfeiffer³⁹, W Robert Reed¹⁵, Jon Roozenbeek¹⁶, Alexa R Schlyfestone¹⁰, Claudia R Schneider^{16

17

40}, Andrew Soh⁴¹, Zhongchen Song⁴², Anirudh Tagat⁴³, Melba Tutor⁴⁴, Andrew H Tyner²¹, Karolina Urbanska⁴⁵, Sander van der Linden¹⁶

Affiliations

¹ Centre for the Study of Existential Risk, University of Cambridge, Cambridge, UK. alexandru.marcoci@gmail.com.
² School of Politics and International Relations, University of Nottingham, Nottingham, UK. alexandru.marcoci@gmail.com.
³ MetaMelb Research Initiative, University of Melbourne, Melbourne, Victoria, Australia.
⁴ QAECO, University of Melbourne, Melbourne, Victoria, Australia.
⁵ School of Communication and Arts, The University of Queensland, Brisbane, Queensland, Australia.
⁶ School of Population Health, Curtin University, Bentley, Western Australia, Australia.
⁷ Environmental Economics and Natural Resources Group, Wageningen University and Research, Wageningen, the Netherlands.
⁸ Department of Food, Pharma and Healthcare, Saint Joseph's University, Philadelphia, PA, USA.
⁹ Business School, University of Auckland, Auckland, New Zealand.
¹⁰ Analytics, Harrisburg University of Science and Technology, Harrisburg, PA, USA.
¹¹ Department of Ecology, Swedish University of Agricultural Sciences, Uppsala, Sweden.
¹² Department of Economics, Swedish University of Agricultural Sciences, Uppsala, Sweden.
¹³ Rhizom Psychological Services LLC, Atlanta, GA, USA.
¹⁴ Center for Healthy Aging, The Pennsylvania State University, University Park, PA, USA.
¹⁵ UCMeta, University of Canterbury, Christchurch, New Zealand.
¹⁶ Department of Psychology, University of Cambridge, Cambridge, UK.
¹⁷ Winton Centre for Risk and Evidence Communication, Department of Pure Mathematics and Mathematical Statistics, University of Cambridge, Cambridge, UK.
¹⁸ UCL Institute for Risk and Disaster Reduction, University College London, London, UK.
¹⁹ Statistics New Zealand, Christchurch, New Zealand.
²⁰ Rochester Institute of Technology, Rochester, NY, USA.
²¹ Center for Open Science, Charlottesville, VA, USA.
²² Independent researcher, Budapest, Hungary.
²³ Department of Management, John Chambers School of Business and Economics, West Virginia University, Morgantown, WV, USA.
²⁴ Centre of Excellence for Biosecurity Risk Analysis, University of Melbourne, Melbourne, Victoria, Australia.
²⁵ Department of Economics, University of Innsbruck, Innsbruck, Austria.
²⁶ Seattle University, Seattle, WA, USA.
²⁷ Department of Economics, Stockholm School of Economics, Stockholm, Sweden.
²⁸ School of Criminal Justice and Criminology, Texas State University, San Marcos, TX, USA.
²⁹ Department of Psychology, Monk Prayogshala, Mumbai, India.
³⁰ Neag School of Education, University of Connecticut, Storrs, USA.
³¹ Department of Public Health, University of Otago, Wellington, New Zealand.
³² Massachusetts Institute of Technology, Cambridge, MA, USA.
³³ Institute of Political Studies, Polish Academy of Sciences, Warszawa, Poland.
³⁴ Department of Computer Science and Engineering, University of California, Santa Cruz, Santa Cruz, CA, USA.
³⁵ Central Bank of Costa Rica, San José, Costa Rica.
³⁶ History and Philosophy of Science, University of Melbourne, Melbourne, Victoria, Australia.
³⁷ University of Stavanger, School of Business and Law, Stavanger, Norway.
³⁸ Department of Psychology, University of Virginia, Charlottesville, VA, USA.
³⁹ NZ IAS, Massey University, Auckland, New Zealand.
⁴⁰ School of Psychology, Speech and Hearing, University of Canterbury, Christchurch, New Zealand.
⁴¹ Department of Philosophy, University of Hawaii at Manoa, Honolulu, HI, USA.
⁴² New Zealand Institute of Economic Research (NZIER), Wellington, New Zealand.
⁴³ Department of Economics, Monk Prayogshala, Mumbai, India.
⁴⁴ Independent researcher, Quezon City, Philippines.
⁴⁵ Independent researcher, Sheffield, UK.

^# Contributed equally.

PMID: 39706868
PMCID: PMC11860236
DOI: 10.1038/s41562-024-01961-1

Predicting the replicability of social and behavioural science claims in COVID-19 preprints

Alexandru Marcoci et al. Nat Hum Behav. 2025 Feb.

. 2025 Feb;9(2):287-304.

doi: 10.1038/s41562-024-01961-1. Epub 2024 Dec 20.

Authors

Affiliations

¹ Centre for the Study of Existential Risk, University of Cambridge, Cambridge, UK. alexandru.marcoci@gmail.com.
² School of Politics and International Relations, University of Nottingham, Nottingham, UK. alexandru.marcoci@gmail.com.
³ MetaMelb Research Initiative, University of Melbourne, Melbourne, Victoria, Australia.
⁴ QAECO, University of Melbourne, Melbourne, Victoria, Australia.
⁵ School of Communication and Arts, The University of Queensland, Brisbane, Queensland, Australia.
⁶ School of Population Health, Curtin University, Bentley, Western Australia, Australia.
⁷ Environmental Economics and Natural Resources Group, Wageningen University and Research, Wageningen, the Netherlands.
⁸ Department of Food, Pharma and Healthcare, Saint Joseph's University, Philadelphia, PA, USA.
⁹ Business School, University of Auckland, Auckland, New Zealand.
¹⁰ Analytics, Harrisburg University of Science and Technology, Harrisburg, PA, USA.
¹¹ Department of Ecology, Swedish University of Agricultural Sciences, Uppsala, Sweden.
¹² Department of Economics, Swedish University of Agricultural Sciences, Uppsala, Sweden.
¹³ Rhizom Psychological Services LLC, Atlanta, GA, USA.
¹⁴ Center for Healthy Aging, The Pennsylvania State University, University Park, PA, USA.
¹⁵ UCMeta, University of Canterbury, Christchurch, New Zealand.
¹⁶ Department of Psychology, University of Cambridge, Cambridge, UK.
¹⁷ Winton Centre for Risk and Evidence Communication, Department of Pure Mathematics and Mathematical Statistics, University of Cambridge, Cambridge, UK.
¹⁸ UCL Institute for Risk and Disaster Reduction, University College London, London, UK.
¹⁹ Statistics New Zealand, Christchurch, New Zealand.
²⁰ Rochester Institute of Technology, Rochester, NY, USA.
²¹ Center for Open Science, Charlottesville, VA, USA.
²² Independent researcher, Budapest, Hungary.
²³ Department of Management, John Chambers School of Business and Economics, West Virginia University, Morgantown, WV, USA.
²⁴ Centre of Excellence for Biosecurity Risk Analysis, University of Melbourne, Melbourne, Victoria, Australia.
²⁵ Department of Economics, University of Innsbruck, Innsbruck, Austria.
²⁶ Seattle University, Seattle, WA, USA.
²⁷ Department of Economics, Stockholm School of Economics, Stockholm, Sweden.
²⁸ School of Criminal Justice and Criminology, Texas State University, San Marcos, TX, USA.
²⁹ Department of Psychology, Monk Prayogshala, Mumbai, India.
³⁰ Neag School of Education, University of Connecticut, Storrs, USA.
³¹ Department of Public Health, University of Otago, Wellington, New Zealand.
³² Massachusetts Institute of Technology, Cambridge, MA, USA.
³³ Institute of Political Studies, Polish Academy of Sciences, Warszawa, Poland.
³⁴ Department of Computer Science and Engineering, University of California, Santa Cruz, Santa Cruz, CA, USA.
³⁵ Central Bank of Costa Rica, San José, Costa Rica.
³⁶ History and Philosophy of Science, University of Melbourne, Melbourne, Victoria, Australia.
³⁷ University of Stavanger, School of Business and Law, Stavanger, Norway.
³⁸ Department of Psychology, University of Virginia, Charlottesville, VA, USA.
³⁹ NZ IAS, Massey University, Auckland, New Zealand.
⁴⁰ School of Psychology, Speech and Hearing, University of Canterbury, Christchurch, New Zealand.
⁴¹ Department of Philosophy, University of Hawaii at Manoa, Honolulu, HI, USA.
⁴² New Zealand Institute of Economic Research (NZIER), Wellington, New Zealand.
⁴³ Department of Economics, Monk Prayogshala, Mumbai, India.
⁴⁴ Independent researcher, Quezon City, Philippines.
⁴⁵ Independent researcher, Sheffield, UK.

^# Contributed equally.

PMID: 39706868
PMCID: PMC11860236
DOI: 10.1038/s41562-024-01961-1

Abstract

Replications are important for assessing the reliability of published findings. However, they are costly, and it is infeasible to replicate everything. Accurate, fast, lower-cost alternatives such as eliciting predictions could accelerate assessment for rapid policy implementation in a crisis and help guide a more efficient allocation of scarce replication resources. We elicited judgements from participants on 100 claims from preprints about an emerging area of research (COVID-19 pandemic) using an interactive structured elicitation protocol, and we conducted 29 new high-powered replications. After interacting with their peers, participant groups with lower task expertise ('beginners') updated their estimates and confidence in their judgements significantly more than groups with greater task expertise ('experienced'). For experienced individuals, the average accuracy was 0.57 (95% CI: [0.53, 0.61]) after interaction, and they correctly classified 61% of claims; beginners' average accuracy was 0.58 (95% CI: [0.54, 0.62]), correctly classifying 69% of claims. The difference in accuracy between groups was not statistically significant and their judgements on the full set of claims were correlated (r(98) = 0.48, P < 0.001). These results suggest that both beginners and more-experienced participants using a structured process have some ability to make better-than-chance predictions about the reliability of 'fast science' under conditions of high uncertainty. However, given the importance of such assessments for making evidence-based critical decisions in a crisis, more research is required to understand who the right experts in forecasting replicability are and how their judgements ought to be elicited.

PubMed Disclaimer

Conflict of interest statement

Competing interests: A.M. is a UKRI Policy Fellow seconded to the Department for Science, Innovation and Technology. The views and conclusions contained herein are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Department for Science, Innovation and Technology or the UK Government. B.A.N., T.M.E., O.M., Z.L., A.H.T., B.L., N.F., E.S.P., M.K.S. and A.L.A. are or were employees of the nonprofit Center for Open Science that has a mission to increase openness, integrity and reproducibility of research. The remaining authors declare no competing interests.

Figures

**Fig. 1. The IDEA protocol.**
The IDEA protocol as implemented on the repliCATS platform.

**Fig. 2. Overview of the repliCATS platform.**
Overview of the repliCATS platform as displayed to participants in Round 2. The full platform view is shown in the centre, summarizing Round 1 responses from 7 participants for one of the evaluated research claims. Enlarged platform components show examples of the Round 2 elicitation questions (collapsed, a), the research claim’s statistical summary information (b), an example restatement of the claim from one of the participants in response to Q1 on the platform (c) and an example of Round 1 participant reasoning paired with their quantitative replicability judgement in response to Q3 on the platform (d).

**Fig. 3. Participants’ best estimates.**
Smoothed distribution of participants’ best estimates for each of the 29 known-outcome research claims with ≥0.8 power with an α = 0.05, organized by type of replication (new or secondary data) and success (did or did not replicate). Experienced participants are shown in yellow and beginners in blue.

**Fig. 4. Predictive accuracy results.**
Average error-based and classification accuracy results and 95% confidence intervals for both individuals and groups of beginners and experienced participants. Estimates and 95% confidence intervals (mean ± s.e. ×1.96) drawn from linear models described in Table 3 refit with no reference class. Statistical test results: a, Round one (Beginners: estimated effect size $\hat{β}$ = 0.563, 95%CI = [0.531, 0.595]; Experienced: $\hat{β}$ = 0.568, 95%CI = [0.535, 0.602]; Difference: t(603.01) = −0.274, P = 0.784, $\hat{β} = - 0.005$ , 95%CI = [−0.043, 0.032], n = 606); Round two (Beginners: $\hat{β}$ = 0.577, 95%CI = [0.536, 0.617]; Experienced: $\hat{β}$ = 0.569, 95%CI = [0.527, 0.611]; Difference: t(591.00) = 0.431, P = 0.667, $\hat{β}$ = 0.008, 95%CI = [−0.028, 0.044], n = 594). b, Round one (Beginners: $\hat{β}$ = 0.675, 95%CI = [0.618, 0.732]; Experienced: $\hat{β}$ = 0.642, 95%CI = [0.594, 0.689]; Difference: t(336.00) = 0.886, P = 0.376, $\hat{β}$ = 0.033, 95%CI = [−0.041, 0.107], n = 338); Round two (Beginners: $\hat{β}$ = 0.694, 95%CI = [0.614, 0.775]; Experienced: $\hat{β} = 0.613$ , 95%CI = [0.538, 0.688]; Difference: t(326.93) = 2.131, P = 0.034, $\hat{β}$ = 0.081, 95%CI = [0.007, 0.156], n = 329). c, Round one (Beginners: $\hat{β} = 0.535$ = 0.535, 95%CI = [0.482, 0.589]; Experienced: $\hat{β}$ = 0.569, 95%CI = [0.515, 0.622]; Difference: t(114.00) = −1.145, P = 0.255, $\hat{β} = - 0.033$ , 95%CI = [−0.09, 0.024], n = 116); Round two (Beginners: $\hat{β}$ = 0.544, 95%CI = [0.493, 0.594]; Experienced: $\hat{β}$ = 0.564, 95%CI = [0.513, 0.614]; Difference: t(113.00) = −0.580, P = 0.563, $\hat{β}$ = −0.020, 95%CI = [−0.087, 0.047], n = 116).

**Fig. 5. Structured group judgements vs final market prices.**
Pearson correlations between Round 2 structured group judgements (collected by the repliCATS team) and final market price for both beginners and experienced participants. Correlations are calculated with a sample size of 100, and the regression line and 95% confidence intervals are calculated using major axis regression.

**Fig. 6. Participants’ best estimates and interval widths.**
Average best estimates and interval widths for both beginners and experienced participants. Estimates and 95% confidence intervals (mean ± s.e. ×1.96) drawn from linear models described in Table 3 refit with no reference class. Statistical test results: a, Round one [Beginners: $\hat{β}$ = 0.632, 95%CI = [0.614, 0.65]; Experienced: $\hat{β}$ = 0.594, 95%CI = [0.576, 0.612]; Difference: t(1981.3591) = 4.596, P < 0.0001, $\hat{β}$ = 0.038, s.e. = 0.008, 95%CI = [0.022, 0.054], n = 2080]; Round two [Beginners: $\hat{β}$ = 0.642, 95%CI = [0.622, 0.662]; Experienced: $\hat{β}$ = 0.587, 95%CI = [0.567, 0.607]; Difference: t(1980.0922) = 8.008, P < 0.0001, $\hat{β}$ = 0.055, s.e. = 0.007, 95%CI = [0.041, 0.068], n = 2080]. b, Round one [Beginners: $\hat{β}$ = 0.309, 95%CI = [0.298, 0.319]; Experienced: $\hat{β}$ = 0.317, 95%CI = [0.306, 0.328]; Difference: t(1988.3245) = −1.044, P = 0.297, $\hat{β}$ = −0.008, s.e. = 0.008, 95%CI = [−0.023, 0.007], n = 2080]; Round two [Beginners: $\hat{β}$ = 0.289, 95%CI = [0.279, 0.298]; Experienced: $\hat{β}$ = 0.318, 95%CI = [0.308, 0.327]; Difference: t(1985.2034) = −4.882, P < 0.0001, $\hat{β}$ = −0.029, s.e. = 0.006, 95%CI = [−0.041, −0.017], n = 2080].

See this image and copyright information in PMC

References

1. Begley, C. G. & Ellis, L. M. Drug development: raise standards for preclinical cancer research. Nature483, 531–533 (2012). - PubMed
1. Errington, T. M. et al. Investigating the replicability of preclinical cancer biology. Elife10, e71601 (2021). - PMC - PubMed
1. Klein, R. A. et al. Investigating variation in replicability. Soc. Psychol.45, 142–152 (2014).
1. Open Science Collaboration. Estimating the reproducibility of psychological science. Science349, aac4716 (2015). - PubMed
1. Liang, H. & Fu, K. W. Testing propositions derived from Twitter studies: generalization and replication in computational social science. PLoS ONE10, e0134270 (2015). - PMC - PubMed

MeSH terms

Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions

Grants and funding

LinkOut - more resources

Full Text Sources
- Nature Publishing Group
- PubMed Central
Medical
- MedlinePlus Health Information

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

Predicting the replicability of social and behavioural science claims in COVID-19 preprints

Affiliations

Predicting the replicability of social and behavioural science claims in COVID-19 preprints

Authors

Affiliations

Abstract

Conflict of interest statement

Figures

References

MeSH terms

Grants and funding

LinkOut - more resources

Full Text Sources

Medical