Teaching open and reproducible scholarship: a critical review of the evidence base for current pedagogical methods and their outcomes

Madeleine Pownall¹, Flávio Azevedo², Laura M König³, Hannah R Slack⁴, Thomas Rhys Evans^{5

6}, Zoe Flack⁷, Sandra Grinschgl⁸, Mahmoud M Elsherif⁹, Katie A Gilligan-Lee¹⁰, Catia M F de Oliveira¹¹, Biljana Gjoneska¹², Tamara Kalandadze¹³, Katherine Button¹⁴, Sarah Ashcroft-Jones¹⁵, Jenny Terry¹⁶, Nihan Albayrak-Aydemir^{17

18}, Filip Děchtěrenko¹⁹, Shilaan Alzahawi²⁰, Bradley J Baker²¹, Merle-Marie Pittelkow²², Lydia Riedl²³, Kathleen Schmidt²⁴, Charlotte R Pennington²⁵, John J Shaw²⁶, Timo Lüke²⁷, Matthew C Makel²⁸, Helena Hartmann²⁹, Mirela Zaneva¹⁵, Daniel Walker³⁰, Steven Verheyen³¹, Daniel Cox³², Jennifer Mattschey¹⁷, Tom Gallagher-Mitchell³³, Peter Branney³⁰, Yanna Weisberg³⁴, Kamil Izydorczak³⁵, Ali H Al-Hoorie³⁶, Ann-Marie Creaven³⁷, Suzanne L K Stewart³⁸, Kai Krautter³⁹, Karen Matvienko-Sikar⁴⁰, Samuel J Westwood⁴¹, Patrícia Arriaga⁴², Meng Liu⁴³, Myriam A Baum³⁹, Tobias Wingen⁴⁴, Robert M Ross⁴⁵, Aoife O'Mahony⁴⁶, Agata Bochynska⁴⁷, Michelle Jamieson⁴⁸, Myrthe Vel Tromp⁴⁹, Siu Kit Yeung⁵⁰, Martin R Vasilev⁵¹, Amélie Gourdon-Kanhukamwe^{52

53}, Leticia Micheli⁵⁴, Markus Konkol⁵⁵, David Moreau⁵⁶, James E Bartlett⁵⁷, Kait Clark⁵⁸, Gwen Brekelmans⁵⁹, Theofilos Gkinopoulos⁶⁰, Samantha L Tyler⁶¹, Jan Philipp Röer⁶², Zlatomira G Ilchovska⁹, Christopher R Madan⁴, Olly Robertson^{63

64}, Bethan J Iley⁶⁵, Samuel Guay⁶⁶, Martina Sladekova^{16

7}, Shanu Sadhwani⁷; FORRT

Affiliations

¹ School of Psychology, University of Leeds, Leeds LS2 9JT, UK.
² Department of Psychology, University of Cambridge, CB2 3EB, UK.
³ Faculty of Life Sciences: Food, Nutrition and Health, University of Bayreuth, 95447 Bayreuth, Germany.
⁴ School of Psychology, University of Nottingham, Nottingham NG7 2RD, UK.
⁵ School of Human Sciences, University of Greenwich, London SE10 9LS, UK.
⁶ Centre for Workforce Development, Institute for Lifecourse Development, University of Greenwich, London SE10 9LS, UK.
⁷ School of Humanities and Social Science, University of Brighton, BN2 0JY, UK.
⁸ Institute of Psychology, University of Graz, 8010 Graz, Austria.
⁹ School of Psychology, University of Birmingham, B15 2TT, UK.
¹⁰ School of Psychology, University College Dublin, Dublin 4, DO4 V1WD, Ireland.
¹¹ Department of Psychology, University of York, York YO10 5DD, USA.
¹² Macedonian Academy of Sciences and Arts, North Macedonia, XCWR+GJM, 1000.
¹³ Faculty of Teacher Education and Languages, Department of Education, ICT and Learning, Ostfold University College, 1757 Halden, Norway.
¹⁴ Department of Psychology, University of Bath, Bath BA2 7AY, UK.
¹⁵ Department of Experimental Psychology, University of Oxford, Oxford OX1 4BH18, UK.
¹⁶ School of Psychology, University of Sussex, Brighton BN1 9RH, UK.
¹⁷ School of Psychology and Counselling, the Open University, Milton Keynes MK7 6AA, UK.
¹⁸ Department of Psychological and Behavioural Science, London School of Economics and Political Science, UK.
¹⁹ Department of Mathematics, College of Polytechnics Jihlava, 1556/16, 586 01, Czech Republic.
²⁰ Graduate School of Business, Stanford University, CA 94305, USA.
²¹ Department of Sport and Recreation Management, Temple University, PA 19122, USA.
²² Department of Psychology, University of Groningen, 9712 CP, Groningen, the Netherlands.
²³ Department of Psychiatry and Psychotherapy, Philipps-University Marburg, D-35039 Marburg, Germany.
²⁴ Department of Psychology, Ashland University, OH 44805, USA.
²⁵ School of Psychology, Aston University, Birmingham B4 7ET, UK.
²⁶ Division of Psychology, De Montfort University, Leicester LE1 9BH, UK.
²⁷ Institute for Educational Research and Teacher Education, University of Graz, Graz, 8010 Graz, Austria.
²⁸ School of Education, Johns Hopkins University, MD 21218, USA.
²⁹ Department for Cognition, Emotion, and Methods in Psychology, University of Vienna, Vienna 1010, Austria.
³⁰ Department of Psychology, Faculty of Management, Law and Social Sciences, University of Bradford, Bradford BD7 1DP, UK.
³¹ Department of Psychology, Education and Child Studies, Erasmus University Rotterdam, Rotterdam 3000, The Netherlands.
³² Division of Neuroscience and Experimental Psychology, University of Manchester, Manchester M13 9PL, UK.
³³ Department of Psychology, Liverpool Hope University, Liverpool L16 9JD, UK.
³⁴ Department of Psychology, Linfield University, Linfield, 503-883-2200, USA.
³⁵ Faculty of Psychology in Wrocław, SWPS University of Social Sciences and Humanities, Wrocław 03-81536, Al Jubail 35819, Poland.
³⁶ Jubail English Language and Preparatory Year Institute, Royal Commission for Jubail and Yanbu, Saudi Arabia.
³⁷ Department of Psychology, University of Limerick, V94 T9PX, Ireland.
³⁸ School of Psychology, University of Chester, Chester CH1 4BJ, UK.
³⁹ Department of Psychology, Saarland University, 66123 Saarbrücken, Germany.
⁴⁰ School of Public Health, University College Cork, Cork, T12 K8AF, Ireland.
⁴¹ Department of Psychology, School of Social Science, University of Westminster, London W1B 2HW, UK.
⁴² Iscte-Universty Institute of Lisbon, CIS-IUL, 1649-026 Lisboa, Portugal.
⁴³ Faculty of Education, University of Cambridge, Cambridge CB2 1TN, UK.
⁴⁴ Institute of General Practice and Family Medicine, University Hospital Bonn, University of Bonn, 53127 Bonn, Germany.
⁴⁵ Department of Philosophy, Macquarie University, NSW 2109, Australia.
⁴⁶ School of Psychology, Cardiff University, Cardiff CF10 3AT, UK.
⁴⁷ University Library, University of Oslo, 0313 Oslo, Norway.
⁴⁸ School of Social and Political Sciences, University of Glasgow, Glasgow G12 8QQ, UK.
⁴⁹ Department of Psychology, Leiden University, 2311 EZ Leiden, The Netherlands.
⁵⁰ Department of Psychology, the Chinese University of Hong Kong, Hong Kong, SAR 100871, People's Republic of China.
⁵¹ Department of Psychology, Bournemouth University, Poole BH12 5BB, UK.
⁵² Department of Neuroimaging, IoPPN, King's College London, UK.
⁵³ IGDORE, London, UK.
⁵⁴ Department of Psychology III, University of Würzburg, 97070 Würzburg, Germany.
⁵⁵ Faculty for Geo-Information Science and Earth Observation, University of Twente, 7522 NB, The Netherlands.
⁵⁶ School of Psychology, University of Auckland, Auckland 1142, New Zealand.
⁵⁷ School of Psychology and Neuroscience, University of Glasgow, Glasgow G12 8QQ, UK.
⁵⁸ Department of Social Sciences, University of the West of England, Bristol BS16 1QY, UK.
⁵⁹ Department of Biological and Experimental Psychology, Queen Mary University of London, E1 4NS, UK.
⁶⁰ South-East European Research Centre, Thessaloniki, Greece.
⁶¹ Department of Neuroscience, Psychology and Behaviour, University of Leicester, UK.
⁶² Department of Psychology, Witten/Herdecke University, Germany.
⁶³ Departments of Psychiatry and Experimental Psychology, University of Oxford, UK.
⁶⁴ School of Psychology, Keele University, Newcastle ST5 5BG, UK.
⁶⁵ School of Psychology, Queen's University Belfast, Belfast BT7 1NN, UK.
⁶⁶ Department of Psychology, University of Montreal, Canada.

PMID: 37206965
PMCID: PMC10189598
DOI: 10.1098/rsos.221255

Review

Teaching open and reproducible scholarship: a critical review of the evidence base for current pedagogical methods and their outcomes

Madeleine Pownall et al. R Soc Open Sci. 2023.

. 2023 May 17;10(5):221255.

doi: 10.1098/rsos.221255. eCollection 2023 May.

Authors

Affiliations

¹ School of Psychology, University of Leeds, Leeds LS2 9JT, UK.
² Department of Psychology, University of Cambridge, CB2 3EB, UK.
³ Faculty of Life Sciences: Food, Nutrition and Health, University of Bayreuth, 95447 Bayreuth, Germany.
⁴ School of Psychology, University of Nottingham, Nottingham NG7 2RD, UK.
⁵ School of Human Sciences, University of Greenwich, London SE10 9LS, UK.
⁶ Centre for Workforce Development, Institute for Lifecourse Development, University of Greenwich, London SE10 9LS, UK.
⁷ School of Humanities and Social Science, University of Brighton, BN2 0JY, UK.
⁸ Institute of Psychology, University of Graz, 8010 Graz, Austria.
⁹ School of Psychology, University of Birmingham, B15 2TT, UK.
¹⁰ School of Psychology, University College Dublin, Dublin 4, DO4 V1WD, Ireland.
¹¹ Department of Psychology, University of York, York YO10 5DD, USA.
¹² Macedonian Academy of Sciences and Arts, North Macedonia, XCWR+GJM, 1000.
¹³ Faculty of Teacher Education and Languages, Department of Education, ICT and Learning, Ostfold University College, 1757 Halden, Norway.
¹⁴ Department of Psychology, University of Bath, Bath BA2 7AY, UK.
¹⁵ Department of Experimental Psychology, University of Oxford, Oxford OX1 4BH18, UK.
¹⁶ School of Psychology, University of Sussex, Brighton BN1 9RH, UK.
¹⁷ School of Psychology and Counselling, the Open University, Milton Keynes MK7 6AA, UK.
¹⁸ Department of Psychological and Behavioural Science, London School of Economics and Political Science, UK.
¹⁹ Department of Mathematics, College of Polytechnics Jihlava, 1556/16, 586 01, Czech Republic.
²⁰ Graduate School of Business, Stanford University, CA 94305, USA.
²¹ Department of Sport and Recreation Management, Temple University, PA 19122, USA.
²² Department of Psychology, University of Groningen, 9712 CP, Groningen, the Netherlands.
²³ Department of Psychiatry and Psychotherapy, Philipps-University Marburg, D-35039 Marburg, Germany.
²⁴ Department of Psychology, Ashland University, OH 44805, USA.
²⁵ School of Psychology, Aston University, Birmingham B4 7ET, UK.
²⁶ Division of Psychology, De Montfort University, Leicester LE1 9BH, UK.
²⁷ Institute for Educational Research and Teacher Education, University of Graz, Graz, 8010 Graz, Austria.
²⁸ School of Education, Johns Hopkins University, MD 21218, USA.
²⁹ Department for Cognition, Emotion, and Methods in Psychology, University of Vienna, Vienna 1010, Austria.
³⁰ Department of Psychology, Faculty of Management, Law and Social Sciences, University of Bradford, Bradford BD7 1DP, UK.
³¹ Department of Psychology, Education and Child Studies, Erasmus University Rotterdam, Rotterdam 3000, The Netherlands.
³² Division of Neuroscience and Experimental Psychology, University of Manchester, Manchester M13 9PL, UK.
³³ Department of Psychology, Liverpool Hope University, Liverpool L16 9JD, UK.
³⁴ Department of Psychology, Linfield University, Linfield, 503-883-2200, USA.
³⁵ Faculty of Psychology in Wrocław, SWPS University of Social Sciences and Humanities, Wrocław 03-81536, Al Jubail 35819, Poland.
³⁶ Jubail English Language and Preparatory Year Institute, Royal Commission for Jubail and Yanbu, Saudi Arabia.
³⁷ Department of Psychology, University of Limerick, V94 T9PX, Ireland.
³⁸ School of Psychology, University of Chester, Chester CH1 4BJ, UK.
³⁹ Department of Psychology, Saarland University, 66123 Saarbrücken, Germany.
⁴⁰ School of Public Health, University College Cork, Cork, T12 K8AF, Ireland.
⁴¹ Department of Psychology, School of Social Science, University of Westminster, London W1B 2HW, UK.
⁴² Iscte-Universty Institute of Lisbon, CIS-IUL, 1649-026 Lisboa, Portugal.
⁴³ Faculty of Education, University of Cambridge, Cambridge CB2 1TN, UK.
⁴⁴ Institute of General Practice and Family Medicine, University Hospital Bonn, University of Bonn, 53127 Bonn, Germany.
⁴⁵ Department of Philosophy, Macquarie University, NSW 2109, Australia.
⁴⁶ School of Psychology, Cardiff University, Cardiff CF10 3AT, UK.
⁴⁷ University Library, University of Oslo, 0313 Oslo, Norway.
⁴⁸ School of Social and Political Sciences, University of Glasgow, Glasgow G12 8QQ, UK.
⁴⁹ Department of Psychology, Leiden University, 2311 EZ Leiden, The Netherlands.
⁵⁰ Department of Psychology, the Chinese University of Hong Kong, Hong Kong, SAR 100871, People's Republic of China.
⁵¹ Department of Psychology, Bournemouth University, Poole BH12 5BB, UK.
⁵² Department of Neuroimaging, IoPPN, King's College London, UK.
⁵³ IGDORE, London, UK.
⁵⁴ Department of Psychology III, University of Würzburg, 97070 Würzburg, Germany.
⁵⁵ Faculty for Geo-Information Science and Earth Observation, University of Twente, 7522 NB, The Netherlands.
⁵⁶ School of Psychology, University of Auckland, Auckland 1142, New Zealand.
⁵⁷ School of Psychology and Neuroscience, University of Glasgow, Glasgow G12 8QQ, UK.
⁵⁸ Department of Social Sciences, University of the West of England, Bristol BS16 1QY, UK.
⁵⁹ Department of Biological and Experimental Psychology, Queen Mary University of London, E1 4NS, UK.
⁶⁰ South-East European Research Centre, Thessaloniki, Greece.
⁶¹ Department of Neuroscience, Psychology and Behaviour, University of Leicester, UK.
⁶² Department of Psychology, Witten/Herdecke University, Germany.
⁶³ Departments of Psychiatry and Experimental Psychology, University of Oxford, UK.
⁶⁴ School of Psychology, Keele University, Newcastle ST5 5BG, UK.
⁶⁵ School of Psychology, Queen's University Belfast, Belfast BT7 1NN, UK.
⁶⁶ Department of Psychology, University of Montreal, Canada.

PMID: 37206965
PMCID: PMC10189598
DOI: 10.1098/rsos.221255

Abstract

In recent years, the scientific community has called for improvements in the credibility, robustness and reproducibility of research, characterized by increased interest and promotion of open and transparent research practices. While progress has been positive, there is a lack of consideration about how this approach can be embedded into undergraduate and postgraduate research training. Specifically, a critical overview of the literature which investigates how integrating open and reproducible science may influence student outcomes is needed. In this paper, we provide the first critical review of literature surrounding the integration of open and reproducible scholarship into teaching and learning and its associated outcomes in students. Our review highlighted how embedding open and reproducible scholarship appears to be associated with (i) students' scientific literacies (i.e. students' understanding of open research, consumption of science and the development of transferable skills); (ii) student engagement (i.e. motivation and engagement with learning, collaboration and engagement in open research) and (iii) students' attitudes towards science (i.e. trust in science and confidence in research findings). However, our review also identified a need for more robust and rigorous methods within pedagogical research, including more interventional and experimental evaluations of teaching practice. We discuss implications for teaching and learning scholarship.

Keywords: higher education; open research; open scholarship; open science; pedagogy; teaching.

PubMed Disclaimer

Conflict of interest statement

We declare we have no competing interests.

References

1. Azevedo F, Liu M, Pennington CR, Pownall M, Evans TR, Parsons S, Elsherif M, Micheli L, Westwood SJ and FORRT. 2022. Towards a culture of open scholarship: the role of pedagogical communities. BMC Res. Notes 15, 75. (10.1186/s13104-022-05944-1) - DOI - PMC - PubMed
1. Azevedo F, Middleton S, Phan JM, Kapp S, Gourdon-Kanhukamwe A, Iley B, Elsherif M, Shaw JJ. 2022. Navigating academia as neurodivergent researchers: promoting neurodiversity within Open Scholarship. APS Observer 35. See https://www.psychologicalscience.org/observer/gs-navigating-academia-as-....
1. Azevedo F, et al. 2019. Introducing a Framework for Open and Reproducible Research Training (FORRT). PsyArXiv. (10.31219/osf.io/bnh7p) - DOI
1. Parsons S, et al. 2022. A community-sourced glossary of open scholarship terms. Nat. Hum. Behav. 6, 312-318. (10.1038/s41562-02101269-4) - DOI - PubMed
1. Manalili MA, et al. 2023. From puzzle to progress: how engaging with neurodiversity can improve cognitive science. Cogn. Sci. 47, e13255. (10.1111/cogs.13255) - DOI - PMC - PubMed

Publication types

Actions

Associated data

figshare/10.6084/m9.figshare.c.6641489

LinkOut - more resources

Full Text Sources

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

Teaching open and reproducible scholarship: a critical review of the evidence base for current pedagogical methods and their outcomes

Affiliations

Teaching open and reproducible scholarship: a critical review of the evidence base for current pedagogical methods and their outcomes

Authors

Affiliations

Abstract

Conflict of interest statement

References

Publication types

Associated data

LinkOut - more resources

Full Text Sources