doi: 10.1098/rsos.241038.
eCollection 2025 Mar.
The reliability of replications: a study in computational reproductions
Nate Breznau
1
, Eike Mark Rinke
2
, Alexander Wuttke
3
, Muna Adem
4
, Jule Adriaans
5
, Esra Akdeniz
6
, Amalia Alvarez-Benjumea
7
, Henrik K Andersen
8
, Daniel Auer
9
, Flavio Azevedo
10
, Oke Bahnsen
11
, Ling Bai
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
26
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
23
70
71
72
73
74
75
76
77
78
79
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
65
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
, Dave Balzer
12
, Paul C Bauer
13
14
, Gerrit Bauer
15
, Markus Baumann
16
, Sharon Baute
17
, Verena Benoit
18
, Julian Bernauer
19
, Carl Berning
20
, Anna Berthold
18
, Felix S Bethke
21
, Thomas Biegert
22
, Katharina Blinzler
23
, Johannes N Blumenberg
24
, Licia Bobzien
25
, Andrea Bohman
26
, Thijs Bol
27
, Amie Bostic
28
, Zuzanna Brzozowska
29
, Katharina Burgdorf
30
, Kaspar Burger
31
32
33
, Kathrin Busch
34
, Juan-Carlos Castillo
35
, Nathan Chan
36
, Pablo Christmann
37
, Roxanne Connelly
38
, Christian S Czymara
39
, Elena Damian
40
, Eline A de Rooij
41
, Alejandro Ecker
42
, Achim Edelmann
43
, Christina Eder
23
, Maureen A Eger
26
44
, Simon Ellerbrock
19
, Anna Forke
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
26
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
23
70
71
72
73
74
75
76
77
78
79
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
65
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
, Andrea Forster
45
, Danilo Freire
46
, Chris Gaasendam
47
, Konstantin Gavras
11
, Vernon Gayle
38
, Theresa Gessler
48
, Timo Gnambs
49
, Amélie Godefroidt
50
, Max Grömping
51
, Martin Groß
52
, Stefan Gruber
53
, Tobias Gummer
54
, Andreas Hadjar
55
, Verena Halbherr
56
, Jan Paul Heisig
57
, Sebastian Hellmeier
58
, Stefanie Heyne
19
, Magdalena Hirsch
59
, Mikael Hjerm
26
, Oshrat Hochman
60
, Jan H Höffler
61
, Andreas Hövermann
62
, Sophia Hunger
63
, Christian Hunkler
64
, Nora Huth-Stöckle
65
, Zsófia S Ignácz
39
, Sabine Israel
34
, Laura Jacobs
66
, Jannes Jacobsen
67
, Bastian Jaeger
68
, Sebastian Jungkunz
69
, Nils Jungmann
23
, Jennifer Kanjana
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
26
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
23
70
71
72
73
74
75
76
77
78
79
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
65
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
, Mathias Kauff
70
, Salman Khan
71
, Sayak Khatua
72
, Manuel Kleinert
73
, Julia Klinger
34
, Jan-Philipp Kolb
74
, Marta Kołczyńska
75
, John Kuk
76
, Katharina Kunißen
12
, Dafina Kurti Sinatra
77
, Alexander Langenkamp
39
, Robin C Lee
78
, Philipp M Lersch
79
, David Liu
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
26
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
23
70
71
72
73
74
75
76
77
78
79
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
65
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
, Lea-Maria Löbel
79
, Philipp Lutscher
80
, Matthias Mader
81
, Joan E Madia
82
, Natalia Malancu
83
, Luis Maldonado
84
, Helge Marahrens
85
, Nicole Martin
86
, Paul Martinez
87
, Jochen Mayerl
8
, Oscar J Mayorga
88
, Robert McDonnell
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
26
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
23
70
71
72
73
74
75
76
77
78
79
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
65
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
, Patricia McManus
89
, Kyle McWagner
90
, Cecil Meeusen
91
, Daniel Meierrieks
59
, Jonathan Mellon
92
, Friedolin Merhout
93
, Samuel Merk
94
, Daniel Meyer
95
, Leticia Micheli
96
, Jonathan Mijs
97
, Cristóbal Moya
98
, Marcel Neunhoeffer
99
, Daniel Nüst
100
, Olav Nygård
101
, Fabian Ochsenfeld
102
, Gunnar Otte
12
, Anna Pechenkina
103
, Mark Pickup
104
, Christopher Prosser
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
26
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
23
70
71
72
73
74
75
76
77
78
79
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
65
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
, Louis Raes
105
, Kevin Ralston
38
, Miguel Ramos
106
, Frank Reichert
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
26
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
23
70
71
72
73
74
75
76
77
78
79
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
65
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
, Arne Roets
107
, Jonathan Rogers
108
, Guido Ropers
34
, Robin Samuel
109
, Gregor Sand
110
, Constanza Sanhueza Petrarca
111
, Ariela Schachter
112
, Merlin Schaeffer
93
, David Schieferdecker
113
, Elmar Schlueter
73
, Katja Schmidt
114
115
, Regine Schmidt
18
, Alexander Schmidt-Catran
39
, Claudia Schmiedeberg
15
, Jürgen Schneider
116
, Martijn Schoonvelde
117
, Julia Schulte-Cloos
118
, Sandy Schumann
119
, Reinhard Schunck
65
, Julian Seuring
120
, Henning Silber
121
, Willem Sleegers
68
, Nico Sonntag
12
, Alexander Staudt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
26
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
23
70
71
72
73
74
75
76
77
78
79
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
65
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
, Nadia Steiber
122
, Nils D Steiner
20
, Sebastian Sternberg
34
, Dieter Stiers
123
, Dragana Stojmenovska
27
, Nora Storz
124
, Erich Striessnig
122
, Anne-Kathrin Stroppe
23
, Jordan W Suchow
125
, Janna Teltemann
126
, Andrey Tibajev
127
, Brian Tung
128
, Giacomo Vagni
129
, Jasper Van Assche
130
131
, Meta van der Linden
10
, Jolanda van der Noll
132
, Arno Van Hootegem
133
, Stefan Vogtenhuber
134
, Bogdan Voicu
135
136
, Fieke Wagemans
137
, Nadja Wehl
138
, Hannah Werner
139
, Brenton M Wiernik
34
, Fabian Winter
140
, Christof Wolf
141
, Cary Wu
142
, Yuki Yamada
18
, Björn Zakula
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
26
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
23
70
71
72
73
74
75
76
77
78
79
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
65
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
, Nan Zhang
143
, Conrad Ziller
144
, Stefan Zins
145
, Tomasz Żółtak
146
, Hung H V Nguyen
147
Affiliations
Affiliations
- 1 Organization and Program Planning, German Institute for Adult Education-Leibniz Center of Lifelong Learning, Bonn 53175, Germany.
- 2 School of Politics and International Studies, University of Leeds, Leeds LS2 9JT, UK.
- 3 Geschwister Scholl Institute, LMU Munich, Munich 81541, Germany.
- 4 Department of Sociology, University of Maryland, College Park 47405, USA.
- 5 Faculty of Sociology, University of Bielefeld, Bielefeld 33615, Germany.
- 6 School of Medicine, Marmara University, Istanbul 34722, Turkey.
- 7 The Institute of Public Goods and Policies (IPP), Centro de Ciencias Humanas y Sociales-Consejo Superior de Investigaciones Científicas, Madrid 28003, Spain.
- 8 Institute of Sociology, Chemnitz University of Technology, Chemnitz 09126, Germany.
- 9 Social and Political Science, Collegio Carlo Alberto, Turin 10122, Italy.
- 10 Department of Interdisciplinary Social Science, Utrecht University, Utrecht 3584CH, Netherlands.
- 11 School of Social Sciences, University of Mannheim, Mannheim 68159, Germany.
- 12 Institute of Sociology, Johannes Gutenberg University Mainz, Mainz 55128, Germany.
- 13 Institute of Political Science, University of Freiburg, Freiburg im Breisgau 79085, Germany.
- 14 Institute of Statistics, LMU Munich, Munich, 79098, Germany.
- 15 Department of Sociology, LMU Munich, Munich 80801, Germany.
- 16 Office of the Executive Board, GESIS-Leibniz Institute for the Social Sciences, Mannheim 68159, Germany.
- 17 Department of Politics and Public Administration, University of Konstanz, Konstanz 78457, Germany.
- 18 Faculty of Social Sciences, Economics, and Business Administration, University of Bamberg, Bamberg 96052, Germany.
- 19 Mannheim Centre for European Social Research (MZES), University of Mannheim, Mannheim 68131, Germany.
- 20 Institute for Political Science, Johannes Gutenberg University Mainz, Mainz 55099, Germany.
- 21 Research Depart-ment IV: Intrastate Conflict, Peace Research Institute Frankfurt (PRIF), Frankfurt 60329, Germany.
- 22 Department of Social Policy, London School of Economics and Political Science, London WC2A 2AE, UK.
- 23 Survey Data Curation, GESIS-Leibniz Institute for the Social Sciences, Mannheim 50667, Germany.
- 24 Knowledge Exchange and Outreach, GESIS-Leibniz-Institute for the Social Sciences, Mannheim 67067, Germany.
- 25 Faculty of Economics and Social Sciences, University of Potsdam, Potsdam 10117, Germany.
- 26 Department of Sociology, Umeå University, Umeå 901 87, Sweden.
- 27 Department of Sociology, University of Amsterdam, Amsterdam 1012 WP, Netherlands.
- 28 Department of Sociology, The University of Texas Rio Grande Valley, Edinburg TX 78539, USA.
- 29 Vienna Institute of Demography, Austrian Academy of Sciences, Vienna, Austria & Austrian National Public Health Institute (Gesundheit Österreich GmbH, GÖG), Vienna, Austria.
- 30 School of Social Sciences, University of Bremen, Bremen 28359, Germany.
- 31 Department of Education, University of Potsdam, Potsdam, 14469, Germany.
- 32 Jacobs Center for Productive Youth Development, University of Zurich, Zürich, 8006, Switzerland.
- 33 Social Research Institute, University College London, London WC1H 0AL, UK.
- 34 Independent Researcher.
- 35 Department of Sociology Universidad de Chile, Millennium Nucleus on Digital Inequalities and Opportunities (NUDOS NCS2022_046), and Center for Social Conflict and Cohesion Studies (COES), University of Chile, Santiago 7800284, Chile.
- 36 Department of Political Science and International Relations, Loyola Marymount University, Los Angeles CA 90045, USA.
- 37 Data and Research on Society, GESIS-Leibniz-Institute for the Social Sciences, Mannheim 68159, Germany.
- 38 School of Social and Political Science, University of Edinburgh, Edinburgh EH8 9LD, Scotland.
- 39 Institute of Sociology, Goethe-University Frankfurt, Frankfurt 60323, Germany.
- 40 Epidemiology and Public Health, Sciensano, Brussels 1050, Belgium.
- 41 Political Science, Simon Fraser University, Burnaby V5A 1S6, Canada.
- 42 Institute of Political Science, Heidelberg University, Heidelberg 69115, Germany.
- 43 Médialab Sciences Po, Paris 75007, France.
- 44 Center for Advanced Study in the Behavioral Sciences, Stanford, CA 94305, USA.
- 45 Department of Sociology, Utrecht University, Utrecht 3584 CS, Netherlands.
- 46 Department of Quantitative Theory and Methods, Emory University, Atlanta 30306, USA.
- 47 Department of Work and Social Economy, Government of Flanders-Belgium.
- 48 Kulturwissenschaftliche Fakultät, Europa Universität Viadrina, Frankfurt 15230, Germany.
- 49 Educational Measurement, Leibniz Institute for Educational Trajectories, Bamberg 96047, Germany.
- 50 Centre for Research on Peace and Development, KU Leuven, Leuven 3000, Belgium.
- 51 School of Government and International Relations, Griffith University, Nathan, Queensland 4111, Australia.
- 52 Department of Sociology, University of Tübingen, Tübingen 72074, Germany.
- 53 Research Data Center and Communication, SHARE BERLIN Institute, Berlin 10115, Germany.
- 54 Data and Research on Society, GESIS-Leibniz Institute for the Social Sciences, Mannheim 68159, Germany.
- 55 Division Sociology, Social Policy and Social Work, University of Fribourg, Fribourg CH-1700, Switzerland.
- 56 Association for Doctoral Studies Baden-Wuerttemberg 70174, Germany.
- 57 Research Group 'Health and Social Inequality', WZB Berlin Social Science Center, Berlin 10785, Germany.
- 58 Transformations of Democracy Unit, WZB Berlin Social Science Center, Berlin 10785, Germany.
- 59 Research Unit Migration, Integration, Transnationalization, WZB Berlin Social Science Center, Berlin 10785, Germany.
- 60 Data and Research on Society, GESIS-Leibniz Institue for the Social Sciences, Mannheim 68159, Germany.
- 61 Facultad de Emprendimiento, Negocios y Economía, Universidad Espíritu Santo, ReplicationWiki, and EQ-Lab.
- 62 Wirtschafts- und Sozialwissenschaftliches Institut (WSI), Hans-Böckler-Foundation, Düsseldorf 40474, Germany.
- 63 SOCIUM-Research Center on Inequality and Social Policy, University of Bremen, Bremen 10785, Germany.
- 64 Berlin Institute for Integration and Migration Research (BIM), Humboldt Universität zu Berlin, Berlin 10099, Germany.
- 65 School of Human and Social Sciences, University of Wuppertal, Wuppertal 42119, Germany.
- 66 Department of Political Science, University of Antwerp, Antwerpen 2000, Belgium.
- 67 Cluster 'Data-Methods-Monitoring', German Center for Integration and Migration Research (DeZIM).
- 68 Department of Social Psychology, Tilburg University, Tilburg 5037AB, Netherlands.
- 69 Institute of Political Science and Sociology, University of Bonn, Bonn 53111, Germany.
- 70 Department of Psychology, Medical School Hamburg, Hamburg 20457, Germany.
- 71 Economics, University of Illinois, Chicago, Chicago, IL, USA.
- 72 School of Public Policy, Oregon State University, Corvallis 97330, USA.
- 73 Institute of Sociology, Justus Liebig University Giessen, Giessen 35394, Germany.
- 74 Statistisches Bundesamt, Statistisches Bundesamt Wiesbaden, Wiesbaden 67549, Germany.
- 75 Department of Research on Social and Institutional Transformations, Institute of Political Studies of the Polish Academy of Sciences, Warsaw 00-625, Poland.
- 76 Department of Political Science, Michigan State University, East Lansing 48823, USA.
- 77 Center for Evaluation, Independent Researcher (Formerly Uni Cologne).
- 78 Department of Sociology, Princeton University, Princeton, USA.
- 79 Socio-Economic Panel, German Institute for Economic Research, Berlin 10117, Germany.
- 80 Department of Political Science, University of Oslo, Oslo 0851, Norway.
- 81 Department of Philosophy, Politics and Economics, Witen/Herdecke University, Witten 58488, Germany.
- 82 Department of Primary Care and Health Sciences, University of Oxford, Oxford OX11JD, England.
- 83 Swiss Forum for Migration and Population Studies, University of Neuchatel, Neuchâtel 1205, Switzerland.
- 84 Instituto de Sociologia, Pontifical Catholic University of Chile, Santiago 7820436, Chile.
- 85 Massive Data Institute, Georgetown University, Washington D.C. 20057, USA.
- 86 Department of Politics, University of Manchester, Manchester M19JS, UK.
- 87 Department of Institutional Research, Western Governors University, Millcreek 84107, USA.
- 88 Director of Data for Freedom, Equity Research Cooperative, 19107.
- 89 Department of Sociology, Indiana University Bloomington, Bloomington, IN 47405, USA.
- 90 Department of Political Science, University of Wisconsin-Milwaukee, Milwaukee, WI 53211, USA.
- 91 Department of Sociology, Center for Sociological Research, KU Leuven, 3000.
- 92 Department of Politics, Westpoint Department of Systems Engineering, M19 2JS.
- 93 Department of Sociology and Centre for Social Data Science, University of Copenhagen, 1353.
- 94 Department of School Development, Karlsruhe University of Education, Karlsruhe 76133, Germany.
- 95 Competence Centre for Regional Development, Federal Institute for Research on Building, Urban Affairs and Spatial Development (BBSR), 03048.
- 96 Department of Social, Economic & Organisational Psychology, Leiden University, Leiden 2333AK, The Netherlands.
- 97 Department of Sociology, Boston University, Boston, MA 02215, USA.
- 98 Socio-Economic Panel, Institute for Economic Research, Berlin 10117, Germany.
- 99 School of Social Sciences, LMU Munich, Munich, Germany.
- 100 Department of Geosciences, Technische Universität Dresden, Dresden 01069, Germany.
- 101 Division of Migration, Ethnicity and Society (REMESO), Linköping University, 60174.
- 102 Administrative Headquarters, Max Planck Society, Munich 80539, Germany.
- 103 Department of Political Science, Utah State University, Logan, UT 84321, USA.
- 104 Political Science, Simon Fraser University, Canada.
- 105 Department of Economics, Tilburg University, Tilburg 5037 AB, The Netherlands.
- 106 Department of Social Policy, Sociology and Criminology, University of Birmingham, Birmingham B15 2TT, UK.
- 107 Department of Developmental, Personality, and Social Psychology, Ghent University, Sint-Pietersnieuwstraat, B-9000, Belgium.
- 108 School of Law, Empirical Research Group, University of California, Los Angeles, Los Angeles, CA, USA.
- 109 Department of Social Sciences, University of Luxembourg, 4366, Luxembourg.
- 110 SHARE Operations, SHARE Berlin Institute, Berlin 10115, Germany.
- 111 School of Politics and International Relations, Australian National University, Canberra 2132, Australia.
- 112 Department of Sociology, Washington University in St. Louis, St. Louis MO 63130, USA.
- 113 Institute for Media and Communication Studies, Freie Universität Berlin, Berlin 14195, Germany.
- 114 Department of Social Sciences, Humboldt University Berlin, Berlin, 10117, Germany.
- 115 Department of Social Sciences, Socio-Economic Panel, Berlin, 10117, Germany.
- 116 Teacher and Teaching Quality, Leibniz Institute for Research and Information in Education, Frankfurt 60323, Germany.
- 117 Chair group European Politics and Society, University of Groningen, Groningen 9712 EK, Netherlands.
- 118 Department of Political Science, University of Marburg, Marburg 35037, Germany.
- 119 Department of Security and Crime Science, University College London, London WC1E 6BT, UK.
- 120 Department Migration, Leibniz Institute for Educational Trajectories, Bamberg 96047, Germany.
- 121 Institute for Social Research, University of Michigan, Ann Arbor MI 48109, USA.
- 122 Department of Sociology, University of Vienna, Vienna 1090, Austria.
- 123 Centre for Political Science Research, KU Leuven, Leuven 3000, Belgium.
- 124 Expert Council on Intergration and Migration, Berlin 10178, Germany.
- 125 School of Business, Stevens Institute of Technology, Hoboken 07030, USA.
- 126 Institute for Social Sciences, University of Hildesheim, Hildesheim 31141, Germany.
- 127 Department of Women's and Children's Health, Uppsala University, Uppsala SE-751 05, Sweden.
- 128 Department of Sociology, Washington University in St. Louis, St. Louis 63130, USA.
- 129 Social Research Institute (UCL), University College London, London WC1E 6BT, USA.
- 130 Center for Social and Cultural Psychology (CESCUP), Université Libre de Bruxelles, Brussels, BE-1050, Belgium.
- 131 Optentia Research Unit, North-West University, Potchefstroom, 2531, South Africa.
- 132 Department of Psychology, University of Hagen, Hagen 58097, Germany.
- 133 Department of Sociology and Human Geography, University of Oslo, Oslo 851, Norway.
- 134 Education and Employment, Institute for Advanced Studies, Vienna 1080, Austria.
- 135 Research Institute for Quality of Life, Romanian Academy, Bucharest, 010071, Romania.
- 136 Department of Sociology, Lucian Blaga University of Sibiu, Sibiu, 550024, Romania.
- 137 Beleidsvisies, Burgervisies en Gedragingen (Policy Perspectives, Citizen Perspectives, and Behaviors), Netherlands Institute for Social Research, The Hague 2594, Netherlands.
- 138 Research Cluster 'The Politics of Inequality', University of Konstanz, Konstanz 78464, Germany.
- 139 Department of Political Science, University of Zurich, Zürich 8050, Switzerland.
- 140 Mechanisms of Normative Change, Max-Planck-Institute for Research on Collective Goods, Bonn 53113, Germany.
- 141 GESIS Leibniz-Institute for the Social Science & University of Mannheim, Mannheim 68159, Germany.
- 142 Department of Sociology, York University, Toronto M3J 1P3, Canada.
- 143 Faculty of Arts and Science, Kyushu University, Fukuoka 819-0395, Japan.
- 144 Department of Political Science, University of Duisburg-Essen, Duisburg-Essen 47057, Germany.
- 145 Institute for Employment Research, Federal Employment Agency, Nuremberg 90478, Germany.
- 146 Department of Computational Social Sciences, Institute of Philosophy and Sociology of the Polish Academy of Sciences, Warsaw 00-330, Poland.
- 147 Political Science, University of Bremen, Bremen 28359, Germany.
- PMID: 40109936
- PMCID: PMC11922520
- DOI: 10.1098/rsos.241038
Item in Clipboard
The reliability of replications: a study in computational reproductions
Nate Breznau et al.
R Soc Open Sci.
.
Display options
Format
doi: 10.1098/rsos.241038.
eCollection 2025 Mar.
Authors
Nate Breznau
1
, Eike Mark Rinke
2
, Alexander Wuttke
3
, Muna Adem
4
, Jule Adriaans
5
, Esra Akdeniz
6
, Amalia Alvarez-Benjumea
7
, Henrik K Andersen
8
, Daniel Auer
9
, Flavio Azevedo
10
, Oke Bahnsen
11
, Ling Bai
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
26
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
23
70
71
72
73
74
75
76
77
78
79
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
65
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
, Dave Balzer
12
, Paul C Bauer
13
14
, Gerrit Bauer
15
, Markus Baumann
16
, Sharon Baute
17
, Verena Benoit
18
, Julian Bernauer
19
, Carl Berning
20
, Anna Berthold
18
, Felix S Bethke
21
, Thomas Biegert
22
, Katharina Blinzler
23
, Johannes N Blumenberg
24
, Licia Bobzien
25
, Andrea Bohman
26
, Thijs Bol
27
, Amie Bostic
28
, Zuzanna Brzozowska
29
, Katharina Burgdorf
30
, Kaspar Burger
31
32
33
, Kathrin Busch
34
, Juan-Carlos Castillo
35
, Nathan Chan
36
, Pablo Christmann
37
, Roxanne Connelly
38
, Christian S Czymara
39
, Elena Damian
40
, Eline A de Rooij
41
, Alejandro Ecker
42
, Achim Edelmann
43
, Christina Eder
23
, Maureen A Eger
26
44
, Simon Ellerbrock
19
, Anna Forke
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
26
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
23
70
71
72
73
74
75
76
77
78
79
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
65
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
, Andrea Forster
45
, Danilo Freire
46
, Chris Gaasendam
47
, Konstantin Gavras
11
, Vernon Gayle
38
, Theresa Gessler
48
, Timo Gnambs
49
, Amélie Godefroidt
50
, Max Grömping
51
, Martin Groß
52
, Stefan Gruber
53
, Tobias Gummer
54
, Andreas Hadjar
55
, Verena Halbherr
56
, Jan Paul Heisig
57
, Sebastian Hellmeier
58
, Stefanie Heyne
19
, Magdalena Hirsch
59
, Mikael Hjerm
26
, Oshrat Hochman
60
, Jan H Höffler
61
, Andreas Hövermann
62
, Sophia Hunger
63
, Christian Hunkler
64
, Nora Huth-Stöckle
65
, Zsófia S Ignácz
39
, Sabine Israel
34
, Laura Jacobs
66
, Jannes Jacobsen
67
, Bastian Jaeger
68
, Sebastian Jungkunz
69
, Nils Jungmann
23
, Jennifer Kanjana
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
26
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
23
70
71
72
73
74
75
76
77
78
79
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
65
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
, Mathias Kauff
70
, Salman Khan
71
, Sayak Khatua
72
, Manuel Kleinert
73
, Julia Klinger
34
, Jan-Philipp Kolb
74
, Marta Kołczyńska
75
, John Kuk
76
, Katharina Kunißen
12
, Dafina Kurti Sinatra
77
, Alexander Langenkamp
39
, Robin C Lee
78
, Philipp M Lersch
79
, David Liu
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
26
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
23
70
71
72
73
74
75
76
77
78
79
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
65
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
, Lea-Maria Löbel
79
, Philipp Lutscher
80
, Matthias Mader
81
, Joan E Madia
82
, Natalia Malancu
83
, Luis Maldonado
84
, Helge Marahrens
85
, Nicole Martin
86
, Paul Martinez
87
, Jochen Mayerl
8
, Oscar J Mayorga
88
, Robert McDonnell
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
26
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
23
70
71
72
73
74
75
76
77
78
79
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
65
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
, Patricia McManus
89
, Kyle McWagner
90
, Cecil Meeusen
91
, Daniel Meierrieks
59
, Jonathan Mellon
92
, Friedolin Merhout
93
, Samuel Merk
94
, Daniel Meyer
95
, Leticia Micheli
96
, Jonathan Mijs
97
, Cristóbal Moya
98
, Marcel Neunhoeffer
99
, Daniel Nüst
100
, Olav Nygård
101
, Fabian Ochsenfeld
102
, Gunnar Otte
12
, Anna Pechenkina
103
, Mark Pickup
104
, Christopher Prosser
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
26
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
23
70
71
72
73
74
75
76
77
78
79
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
65
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
, Louis Raes
105
, Kevin Ralston
38
, Miguel Ramos
106
, Frank Reichert
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
26
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
23
70
71
72
73
74
75
76
77
78
79
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
65
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
, Arne Roets
107
, Jonathan Rogers
108
, Guido Ropers
34
, Robin Samuel
109
, Gregor Sand
110
, Constanza Sanhueza Petrarca
111
, Ariela Schachter
112
, Merlin Schaeffer
93
, David Schieferdecker
113
, Elmar Schlueter
73
, Katja Schmidt
114
115
, Regine Schmidt
18
, Alexander Schmidt-Catran
39
, Claudia Schmiedeberg
15
, Jürgen Schneider
116
, Martijn Schoonvelde
117
, Julia Schulte-Cloos
118
, Sandy Schumann
119
, Reinhard Schunck
65
, Julian Seuring
120
, Henning Silber
121
, Willem Sleegers
68
, Nico Sonntag
12
, Alexander Staudt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
26
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
23
70
71
72
73
74
75
76
77
78
79
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
65
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
, Nadia Steiber
122
, Nils D Steiner
20
, Sebastian Sternberg
34
, Dieter Stiers
123
, Dragana Stojmenovska
27
, Nora Storz
124
, Erich Striessnig
122
, Anne-Kathrin Stroppe
23
, Jordan W Suchow
125
, Janna Teltemann
126
, Andrey Tibajev
127
, Brian Tung
128
, Giacomo Vagni
129
, Jasper Van Assche
130
131
, Meta van der Linden
10
, Jolanda van der Noll
132
, Arno Van Hootegem
133
, Stefan Vogtenhuber
134
, Bogdan Voicu
135
136
, Fieke Wagemans
137
, Nadja Wehl
138
, Hannah Werner
139
, Brenton M Wiernik
34
, Fabian Winter
140
, Christof Wolf
141
, Cary Wu
142
, Yuki Yamada
18
, Björn Zakula
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
26
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
23
70
71
72
73
74
75
76
77
78
79
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
65
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
, Nan Zhang
143
, Conrad Ziller
144
, Stefan Zins
145
, Tomasz Żółtak
146
, Hung H V Nguyen
147
Affiliations
- 1 Organization and Program Planning, German Institute for Adult Education-Leibniz Center of Lifelong Learning, Bonn 53175, Germany.
- 2 School of Politics and International Studies, University of Leeds, Leeds LS2 9JT, UK.
- 3 Geschwister Scholl Institute, LMU Munich, Munich 81541, Germany.
- 4 Department of Sociology, University of Maryland, College Park 47405, USA.
- 5 Faculty of Sociology, University of Bielefeld, Bielefeld 33615, Germany.
- 6 School of Medicine, Marmara University, Istanbul 34722, Turkey.
- 7 The Institute of Public Goods and Policies (IPP), Centro de Ciencias Humanas y Sociales-Consejo Superior de Investigaciones Científicas, Madrid 28003, Spain.
- 8 Institute of Sociology, Chemnitz University of Technology, Chemnitz 09126, Germany.
- 9 Social and Political Science, Collegio Carlo Alberto, Turin 10122, Italy.
- 10 Department of Interdisciplinary Social Science, Utrecht University, Utrecht 3584CH, Netherlands.
- 11 School of Social Sciences, University of Mannheim, Mannheim 68159, Germany.
- 12 Institute of Sociology, Johannes Gutenberg University Mainz, Mainz 55128, Germany.
- 13 Institute of Political Science, University of Freiburg, Freiburg im Breisgau 79085, Germany.
- 14 Institute of Statistics, LMU Munich, Munich, 79098, Germany.
- 15 Department of Sociology, LMU Munich, Munich 80801, Germany.
- 16 Office of the Executive Board, GESIS-Leibniz Institute for the Social Sciences, Mannheim 68159, Germany.
- 17 Department of Politics and Public Administration, University of Konstanz, Konstanz 78457, Germany.
- 18 Faculty of Social Sciences, Economics, and Business Administration, University of Bamberg, Bamberg 96052, Germany.
- 19 Mannheim Centre for European Social Research (MZES), University of Mannheim, Mannheim 68131, Germany.
- 20 Institute for Political Science, Johannes Gutenberg University Mainz, Mainz 55099, Germany.
- 21 Research Depart-ment IV: Intrastate Conflict, Peace Research Institute Frankfurt (PRIF), Frankfurt 60329, Germany.
- 22 Department of Social Policy, London School of Economics and Political Science, London WC2A 2AE, UK.
- 23 Survey Data Curation, GESIS-Leibniz Institute for the Social Sciences, Mannheim 50667, Germany.
- 24 Knowledge Exchange and Outreach, GESIS-Leibniz-Institute for the Social Sciences, Mannheim 67067, Germany.
- 25 Faculty of Economics and Social Sciences, University of Potsdam, Potsdam 10117, Germany.
- 26 Department of Sociology, Umeå University, Umeå 901 87, Sweden.
- 27 Department of Sociology, University of Amsterdam, Amsterdam 1012 WP, Netherlands.
- 28 Department of Sociology, The University of Texas Rio Grande Valley, Edinburg TX 78539, USA.
- 29 Vienna Institute of Demography, Austrian Academy of Sciences, Vienna, Austria & Austrian National Public Health Institute (Gesundheit Österreich GmbH, GÖG), Vienna, Austria.
- 30 School of Social Sciences, University of Bremen, Bremen 28359, Germany.
- 31 Department of Education, University of Potsdam, Potsdam, 14469, Germany.
- 32 Jacobs Center for Productive Youth Development, University of Zurich, Zürich, 8006, Switzerland.
- 33 Social Research Institute, University College London, London WC1H 0AL, UK.
- 34 Independent Researcher.
- 35 Department of Sociology Universidad de Chile, Millennium Nucleus on Digital Inequalities and Opportunities (NUDOS NCS2022_046), and Center for Social Conflict and Cohesion Studies (COES), University of Chile, Santiago 7800284, Chile.
- 36 Department of Political Science and International Relations, Loyola Marymount University, Los Angeles CA 90045, USA.
- 37 Data and Research on Society, GESIS-Leibniz-Institute for the Social Sciences, Mannheim 68159, Germany.
- 38 School of Social and Political Science, University of Edinburgh, Edinburgh EH8 9LD, Scotland.
- 39 Institute of Sociology, Goethe-University Frankfurt, Frankfurt 60323, Germany.
- 40 Epidemiology and Public Health, Sciensano, Brussels 1050, Belgium.
- 41 Political Science, Simon Fraser University, Burnaby V5A 1S6, Canada.
- 42 Institute of Political Science, Heidelberg University, Heidelberg 69115, Germany.
- 43 Médialab Sciences Po, Paris 75007, France.
- 44 Center for Advanced Study in the Behavioral Sciences, Stanford, CA 94305, USA.
- 45 Department of Sociology, Utrecht University, Utrecht 3584 CS, Netherlands.
- 46 Department of Quantitative Theory and Methods, Emory University, Atlanta 30306, USA.
- 47 Department of Work and Social Economy, Government of Flanders-Belgium.
- 48 Kulturwissenschaftliche Fakultät, Europa Universität Viadrina, Frankfurt 15230, Germany.
- 49 Educational Measurement, Leibniz Institute for Educational Trajectories, Bamberg 96047, Germany.
- 50 Centre for Research on Peace and Development, KU Leuven, Leuven 3000, Belgium.
- 51 School of Government and International Relations, Griffith University, Nathan, Queensland 4111, Australia.
- 52 Department of Sociology, University of Tübingen, Tübingen 72074, Germany.
- 53 Research Data Center and Communication, SHARE BERLIN Institute, Berlin 10115, Germany.
- 54 Data and Research on Society, GESIS-Leibniz Institute for the Social Sciences, Mannheim 68159, Germany.
- 55 Division Sociology, Social Policy and Social Work, University of Fribourg, Fribourg CH-1700, Switzerland.
- 56 Association for Doctoral Studies Baden-Wuerttemberg 70174, Germany.
- 57 Research Group 'Health and Social Inequality', WZB Berlin Social Science Center, Berlin 10785, Germany.
- 58 Transformations of Democracy Unit, WZB Berlin Social Science Center, Berlin 10785, Germany.
- 59 Research Unit Migration, Integration, Transnationalization, WZB Berlin Social Science Center, Berlin 10785, Germany.
- 60 Data and Research on Society, GESIS-Leibniz Institue for the Social Sciences, Mannheim 68159, Germany.
- 61 Facultad de Emprendimiento, Negocios y Economía, Universidad Espíritu Santo, ReplicationWiki, and EQ-Lab.
- 62 Wirtschafts- und Sozialwissenschaftliches Institut (WSI), Hans-Böckler-Foundation, Düsseldorf 40474, Germany.
- 63 SOCIUM-Research Center on Inequality and Social Policy, University of Bremen, Bremen 10785, Germany.
- 64 Berlin Institute for Integration and Migration Research (BIM), Humboldt Universität zu Berlin, Berlin 10099, Germany.
- 65 School of Human and Social Sciences, University of Wuppertal, Wuppertal 42119, Germany.
- 66 Department of Political Science, University of Antwerp, Antwerpen 2000, Belgium.
- 67 Cluster 'Data-Methods-Monitoring', German Center for Integration and Migration Research (DeZIM).
- 68 Department of Social Psychology, Tilburg University, Tilburg 5037AB, Netherlands.
- 69 Institute of Political Science and Sociology, University of Bonn, Bonn 53111, Germany.
- 70 Department of Psychology, Medical School Hamburg, Hamburg 20457, Germany.
- 71 Economics, University of Illinois, Chicago, Chicago, IL, USA.
- 72 School of Public Policy, Oregon State University, Corvallis 97330, USA.
- 73 Institute of Sociology, Justus Liebig University Giessen, Giessen 35394, Germany.
- 74 Statistisches Bundesamt, Statistisches Bundesamt Wiesbaden, Wiesbaden 67549, Germany.
- 75 Department of Research on Social and Institutional Transformations, Institute of Political Studies of the Polish Academy of Sciences, Warsaw 00-625, Poland.
- 76 Department of Political Science, Michigan State University, East Lansing 48823, USA.
- 77 Center for Evaluation, Independent Researcher (Formerly Uni Cologne).
- 78 Department of Sociology, Princeton University, Princeton, USA.
- 79 Socio-Economic Panel, German Institute for Economic Research, Berlin 10117, Germany.
- 80 Department of Political Science, University of Oslo, Oslo 0851, Norway.
- 81 Department of Philosophy, Politics and Economics, Witen/Herdecke University, Witten 58488, Germany.
- 82 Department of Primary Care and Health Sciences, University of Oxford, Oxford OX11JD, England.
- 83 Swiss Forum for Migration and Population Studies, University of Neuchatel, Neuchâtel 1205, Switzerland.
- 84 Instituto de Sociologia, Pontifical Catholic University of Chile, Santiago 7820436, Chile.
- 85 Massive Data Institute, Georgetown University, Washington D.C. 20057, USA.
- 86 Department of Politics, University of Manchester, Manchester M19JS, UK.
- 87 Department of Institutional Research, Western Governors University, Millcreek 84107, USA.
- 88 Director of Data for Freedom, Equity Research Cooperative, 19107.
- 89 Department of Sociology, Indiana University Bloomington, Bloomington, IN 47405, USA.
- 90 Department of Political Science, University of Wisconsin-Milwaukee, Milwaukee, WI 53211, USA.
- 91 Department of Sociology, Center for Sociological Research, KU Leuven, 3000.
- 92 Department of Politics, Westpoint Department of Systems Engineering, M19 2JS.
- 93 Department of Sociology and Centre for Social Data Science, University of Copenhagen, 1353.
- 94 Department of School Development, Karlsruhe University of Education, Karlsruhe 76133, Germany.
- 95 Competence Centre for Regional Development, Federal Institute for Research on Building, Urban Affairs and Spatial Development (BBSR), 03048.
- 96 Department of Social, Economic & Organisational Psychology, Leiden University, Leiden 2333AK, The Netherlands.
- 97 Department of Sociology, Boston University, Boston, MA 02215, USA.
- 98 Socio-Economic Panel, Institute for Economic Research, Berlin 10117, Germany.
- 99 School of Social Sciences, LMU Munich, Munich, Germany.
- 100 Department of Geosciences, Technische Universität Dresden, Dresden 01069, Germany.
- 101 Division of Migration, Ethnicity and Society (REMESO), Linköping University, 60174.
- 102 Administrative Headquarters, Max Planck Society, Munich 80539, Germany.
- 103 Department of Political Science, Utah State University, Logan, UT 84321, USA.
- 104 Political Science, Simon Fraser University, Canada.
- 105 Department of Economics, Tilburg University, Tilburg 5037 AB, The Netherlands.
- 106 Department of Social Policy, Sociology and Criminology, University of Birmingham, Birmingham B15 2TT, UK.
- 107 Department of Developmental, Personality, and Social Psychology, Ghent University, Sint-Pietersnieuwstraat, B-9000, Belgium.
- 108 School of Law, Empirical Research Group, University of California, Los Angeles, Los Angeles, CA, USA.
- 109 Department of Social Sciences, University of Luxembourg, 4366, Luxembourg.
- 110 SHARE Operations, SHARE Berlin Institute, Berlin 10115, Germany.
- 111 School of Politics and International Relations, Australian National University, Canberra 2132, Australia.
- 112 Department of Sociology, Washington University in St. Louis, St. Louis MO 63130, USA.
- 113 Institute for Media and Communication Studies, Freie Universität Berlin, Berlin 14195, Germany.
- 114 Department of Social Sciences, Humboldt University Berlin, Berlin, 10117, Germany.
- 115 Department of Social Sciences, Socio-Economic Panel, Berlin, 10117, Germany.
- 116 Teacher and Teaching Quality, Leibniz Institute for Research and Information in Education, Frankfurt 60323, Germany.
- 117 Chair group European Politics and Society, University of Groningen, Groningen 9712 EK, Netherlands.
- 118 Department of Political Science, University of Marburg, Marburg 35037, Germany.
- 119 Department of Security and Crime Science, University College London, London WC1E 6BT, UK.
- 120 Department Migration, Leibniz Institute for Educational Trajectories, Bamberg 96047, Germany.
- 121 Institute for Social Research, University of Michigan, Ann Arbor MI 48109, USA.
- 122 Department of Sociology, University of Vienna, Vienna 1090, Austria.
- 123 Centre for Political Science Research, KU Leuven, Leuven 3000, Belgium.
- 124 Expert Council on Intergration and Migration, Berlin 10178, Germany.
- 125 School of Business, Stevens Institute of Technology, Hoboken 07030, USA.
- 126 Institute for Social Sciences, University of Hildesheim, Hildesheim 31141, Germany.
- 127 Department of Women's and Children's Health, Uppsala University, Uppsala SE-751 05, Sweden.
- 128 Department of Sociology, Washington University in St. Louis, St. Louis 63130, USA.
- 129 Social Research Institute (UCL), University College London, London WC1E 6BT, USA.
- 130 Center for Social and Cultural Psychology (CESCUP), Université Libre de Bruxelles, Brussels, BE-1050, Belgium.
- 131 Optentia Research Unit, North-West University, Potchefstroom, 2531, South Africa.
- 132 Department of Psychology, University of Hagen, Hagen 58097, Germany.
- 133 Department of Sociology and Human Geography, University of Oslo, Oslo 851, Norway.
- 134 Education and Employment, Institute for Advanced Studies, Vienna 1080, Austria.
- 135 Research Institute for Quality of Life, Romanian Academy, Bucharest, 010071, Romania.
- 136 Department of Sociology, Lucian Blaga University of Sibiu, Sibiu, 550024, Romania.
- 137 Beleidsvisies, Burgervisies en Gedragingen (Policy Perspectives, Citizen Perspectives, and Behaviors), Netherlands Institute for Social Research, The Hague 2594, Netherlands.
- 138 Research Cluster 'The Politics of Inequality', University of Konstanz, Konstanz 78464, Germany.
- 139 Department of Political Science, University of Zurich, Zürich 8050, Switzerland.
- 140 Mechanisms of Normative Change, Max-Planck-Institute for Research on Collective Goods, Bonn 53113, Germany.
- 141 GESIS Leibniz-Institute for the Social Science & University of Mannheim, Mannheim 68159, Germany.
- 142 Department of Sociology, York University, Toronto M3J 1P3, Canada.
- 143 Faculty of Arts and Science, Kyushu University, Fukuoka 819-0395, Japan.
- 144 Department of Political Science, University of Duisburg-Essen, Duisburg-Essen 47057, Germany.
- 145 Institute for Employment Research, Federal Employment Agency, Nuremberg 90478, Germany.
- 146 Department of Computational Social Sciences, Institute of Philosophy and Sociology of the Polish Academy of Sciences, Warsaw 00-330, Poland.
- 147 Political Science, University of Bremen, Bremen 28359, Germany.
- PMID: 40109936
- PMCID: PMC11922520
- DOI: 10.1098/rsos.241038
Item in Clipboard
Display options
Format
Abstract
This study investigates researcher variability in computational reproduction, an activity for which it is least expected. Eighty-five independent teams attempted numerical replication of results from an original study of policy preferences and immigration. Reproduction teams were randomly grouped into a 'transparent group' receiving original study and code or 'opaque group' receiving only a method and results description and no code. The transparent group mostly verified original results (95.7% same sign and p-value cutoff), while the opaque group had less success (89.3%). Second-decimal place exact numerical reproductions were less common (76.9 and 48.1%). Qualitative investigation of the workflows revealed many causes of error, including mistakes and procedural variations. When curating mistakes, we still find that only the transparent group was reliably successful. Our findings imply a need for transparency, but also more. Institutional checks and less subjective difficulty for researchers 'doing reproduction' would help, implying a need for better training. We also urge increased awareness of complexity in the research process and in 'push button' replications.
Keywords: computational reproduction; reliability; replications; social and behavioural sciences.
© 2025 The Author(s).
Conflict of interest statement
We declare we have no competing interests.
Figures
Figure 1.
Results of a crowdsourced computational…
Figure 1.
Results of a crowdsourced computational replication from 85 Replicator Teams. Note : The…
Results of a crowdsourced computational replication from 85 Replicator Teams. Note: The study involved a computational reproduction of Brady & Finnigan [22]. Only the Transparent Group “TG” (N = 39 teams & 1,872 effects) had access to the original paper and code, while the Opaque Group “OG” (N = 46 teams & 1,874 effects) had only a description of the methods and no code (see §2 and appendix A). Left-side are effect-level results (a & c) and right-side are team-level results (b & d). Bars represent two-tailed 95% CIs. Differences between experimental groups on all outcome measures are statistically significant at the effect-level with 99% confidence (p < 0.01); at the team-level, this is only true of Exact Replication. The between (the percentage of the total effect-level variance that occurs between teams, also known as intra-class correlation coefficient [ICC]) are A. DIRECTIONAL REPRODUCTION: TG = 23.9% & OG = 29.4%; B. EXACT REPLICATION: TG = 63.1% & OG = 33.2%; C. REPLICATION ERROR: TG = 39.6% & OG = 93.3%.
Figure 2.
Rates of error per team.
Figure 2.
Rates of error per team.
Rates of error per team.
Figure 3.
Simulation of the number of…
Figure 3.
Simulation of the number of independent replications required to achieve a majority correct…
Simulation of the number of independent replications required to achieve a majority correct at different P cutoffs
References
-
- Willis C, Stodden V. 2020. Trust but Verify: How to Leverage Policies, Workflows, and Infrastructure to Ensure Computational Reproducibility in Publication. Harv. Data Sci. Rev. 2. (10.1162/99608f92.25982dcf) - DOI
-
- Auspurg K, Brüderl J. 2021. Has the Credibility of the Social Sciences Been Credibly Destroyed? Reanalyzing the ‘Many Analysts, One Data Set’ Project. Socius. Sociol. Res. Dyn. World. 7, 237802312110244. (10.1177/23780231211024421) - DOI
-
- Lundberg I, Johnson R, Stewart BM. 2021. What Is Your Estimand? Defining the Target Quantity Connects Statistical Evidence to Theory. Am. Sociol. Rev. 86, 532–565. (10.1177/00031224211004187) - DOI
Associated data
LinkOut - more resources
Full Text Sources