Credibility of preprints: an interdisciplinary survey of researchers

Abstract

Preprints increase accessibility and can speed scholarly communication if researchers view them as credible enough to read and use. Preprint services do not provide the heuristic cues of a journal's reputation, selection, and peer-review processes that, regardless of their flaws, are often used as a guide for deciding what to read. We conducted a survey of 3759 researchers across a wide range of disciplines to determine the importance of different cues for assessing the credibility of individual preprints and preprint services. We found that cues related to information about open science content and independent verification of author claims were rated as highly important for judging preprint credibility, and peer views and author information were rated as less important. As of early 2020, very few preprint services display any of the most important cues. By adding such cues, services may be able to help researchers better assess the credibility of preprints, enabling scholars to more confidently use preprints, thereby accelerating scientific communication and discovery.

Keywords: credibility; preprints; trust.

Figure 1.

Favourability towards preprints by discipline. Respondents favourability towards the use of preprints in their discipline, broken out by the six most common disciplines in our sample. Numbers to the left of the bars indicate the percentage of respondents who responded with ‘very unfavorable’, ‘somewhat unfavorable’ or ‘slightly unfavorable’, the numbers in the centre of the bars indicate the percentage who responded, ‘neither unfavorable nor favorable’, and the numbers to the right of the bars indicate the percentage who responded, ‘very favorable’, ‘somewhat favorable’ or ‘slightly favorable’.

Figure 2.

Favourability towards preprints by academic career stage. Respondents favourability towards the use of preprints in their discipline, broken out by academic career stage. Numbers to the left of the bars indicates the percentage of respondents who responded with ‘very unfavorable’, ‘somewhat unfavorable’, or ‘slightly unfavorable’, the numbers in the centre of the bars indicates the percentage who responded, ‘neither unfavorable nor favorable’, and the numbers to the right of the bars indicates the percentage who responded, ‘very favorable’, ‘somewhat favorable’, or ‘slightly favorable’.

Figure 3.

Preprint use by discipline. Whether respondents had ever ‘viewed/downloaded a preprint' (a) and whether they or a co-authors had ‘submitted a preprint’ (b), broken up by discipline. Respondents who did not answer the question or who answered ‘not sure’, are not included in the graphs.

Figure 4.

Preprint use by academic career stage. Whether respondents had ever ‘viewed/downloaded a preprint' (a) and whether they or a co-author had ‘submitted a preprint’ (b), broken up by career stage. Respondents who did not answer the question or who answered ‘not sure’ are not included in the graphs.

Figure 5.

Importance of information for credibility judgements. Response to each of the 19 survey questions asking about the importance of each type of information listed for making credibility judgements about a preprint. The number to the left of the bars indicates the percentage of respondents who responded with ‘not at all important’ or ‘slightly important’, the number in the centre of the bar indicates the percentage who responded, ‘moderately important’, and the percentage to the right of the bar indicates the percentage who responded, ‘very important’ or ‘extremely important’’.

Figure 6.

Correlations between preprint credibility questions and the extent to which participants favour the use of preprints, submit preprints and view/download them.

Figure 7.

Responses to preprint credibility questions by disciplines. Mean and standard deviation of response to preprint credibility questions by disciplines. Respondents who either skipped the question or could not be categorized into a bepress tier 1 taxonomy are not included in the table. Additionally, participants who listed their discipline as Business, Law, Education, Engineering or Arts and Humanities were also excluded because there were too few respondents in these categories. The response scale is 1—not at all important, 2—slightly important, 3—moderately important, 4—very important, 5—extremely important.

Figure 8.

Responses to preprint credibility questions by academic career stage. Mean and standard deviation of response to preprint credibility questions by career stage. Respondents who either skipped the question or listed a professional job title that could not be placed in the academic career ladder above were not included in this table. The response scale is 1—not at all important, 2—slightly important, 3—moderately important, 4—very important, 5—extremely important.

Figure 9.

Factor structure of information items. The six-factor structure resulting from the EFA. Two items, ‘authors general level of open scholarship’ and ‘preprint submitted to a journal’, did not load onto any factor particularly strongly, and were not retained in further analyses.

Figure 10.

Information cues presented by preprint services. Coding by preprint service for whether they cue the information discussed in each preprint credibility question.