A systematic review of pre-registration in autism research journals

Abstract

Pre-registration refers to the practice of researchers preparing a time-stamped document describing the plans for a study. This open research tool is used to improve transparency, so that readers can evaluate the extent to which the researcher adhered to their original plans and tested their theory appropriately. In the current study, we conducted an audit of pre-registration in autism research through a review of manuscripts published across six autism research journals between 2011 and 2022. We found that 192 publications were pre-registered, approximately 2.23% of publications in autism journals during this time frame. We also conducted a quality assessment of a sample of the pre-registrations, finding that specificity in the pre-registrations was low, particularly in the design and analysis components of the pre-registration. In addition, only 28% of sampled manuscripts adhered to their analysis plan or transparently disclosed all deviations. Autism researchers conducting confirmatory, quantitative research should consider pre-registering their work, reporting any changes in plans transparently in the published manuscript. We outline recommendations for researchers and journals to improve the transparency and robustness of the field.Lay abstractWhen researchers write down their plans for a study ahead of time and make this public, this is called pre-registration. Pre-registration allows others to see if the researchers stuck to their original plan or changed as they went along. Pre-registration is growing in popularity but we do not know how widely it is used in autism research. In this study, we looked at papers published in six major autism journals between 2011 and 2022. We found that only 2.23% of papers published in autism journals had been pre-registered. We also took a close look at a selection of the pre-registrations to check how good they were and if researchers stuck to their plans. We found that the pre-registrations generally lacked specifics, particularly about how the study was designed and the data would be analysed. We also found that only 28% of the papers closely followed the pre-registered plans or reported the changes.Based on these findings, we recommend that autism researchers consider pre-registering their work and transparently report any changes from their original plans. We have provided some recommendations for researchers and journals on how pre-registration could be better used in autism research.

Keywords: autism; autism research; meta-research; pre-registration.

Figure 1.

Flow diagram representing the selection of manuscripts.

Figure 2.

Breakdowns of manuscripts which were coded as pre-registered (a link or identification code to a pre-registration was in the manuscript). (a) Total number of manuscripts organised by type. (b) Percentage of manuscripts which included a pre-registration which were published across the journals each year. (c) Percentage of manuscripts which included a pre-registration across the time frame by journal (there were no pre-registrations in manuscripts published in Autism in Adulthood).

Figure 3.

Tile plot displaying the ratings for minimum detail for the 31 manuscripts which were assessed. Pre-registrations were rated for containing minimum detail about the study Hypothesis and Research Question (Hyp_RQ), Dependent and Independent Variables (DV_IV), Operationalisation of the DV and IV (DV_IV.Oper), planned sample size or stopping rule (Sample), Procedure and Analysis. Where the pre-registration scored <6, that manuscript was not reviewed further.

Figure 4.

Tile plot displaying the average specificity score across researcher degrees of freedom by section Hypothesis, Design, Data Collection, Data Analysis and Reporting. Increasingly light shades of blue indicate greater specificity in that section. Note comment in Discussion section regarding scores of 0 for Reporting.

Figure 5.

Tile Plot (left) and stacked bar chart (right) showing deviations (none, disclosed and undisclosed) by section Hypothesis/Research Question (H.RQ), Variables, Sample Size, Exclusion Criteria, Procedure and Analysis.