How to do Better N400 Studies: Reproducibility, Consistency and Adherence to Research Standards in the Existing Literature

Abstract

Given the complexity of ERP recording and processing pipeline, the resulting variability of methodological options, and the potential for these decisions to influence study outcomes, it is important to understand how ERP studies are conducted in practice and to what extent researchers are transparent about their data collection and analysis procedures. The review gives an overview of methodology reporting in a sample of 132 ERP papers, published between January 1980 - June 2018 in journals included in two large databases: Web of Science and PubMed. Because ERP methodology partly depends on the study design, we focused on a well-established component (the N400) in the most commonly assessed population (healthy neurotypical adults), in one of its most common modalities (visual images). The review provides insights into 73 properties of study design, data pre-processing, measurement, statistics, visualization of results, and references to supplemental information across studies within the same subfield. For each of the examined methodological decisions, the degree of consistency, clarity of reporting and deviations from the guidelines for best practice were examined. Overall, the results show that each study had a unique approach to ERP data recording, processing and analysis, and that at least some details were missing from all papers. In the review, we highlight the most common reporting omissions and deviations from established recommendations, as well as areas in which there was the least consistency. Additionally, we provide guidance for a priori selection of the N400 measurement window and electrode locations based on the results of previous studies.

Keywords: ERP methodology; Event related potentials; N400; Open science; Pictures; Reproducibility.

Fig. 1

PRISMA flow diagram

Fig. 2

Frequencies of omitting methodological details from reports. The y axis shows methodological information that was examined, while the x axis shows the percentage of papers in which this information was provided, partly provided, or not provided. All percentages are relative to No – the number of cases relevant for the variable in question (e.g., studies in which a procedure was used). Green bars show percentage of papers in which the methodological information in question was provided. Yellow bars show percentages of papers in which some information was given, but it was either partial or inconclusive. Red bars show percentages of papers from which the detail in question was omitted. Table of frequencies and more details on them can be found in Supplement 6d

Fig. 3

Variability and frequencies of different choices when making methodological decisions. The y axis shows methodological information that was examined, while the x axis shows the percentage of each option. All percentages are relative to No – the number of cases relevant for the variable in question (e.g., studies in which it was possible or appropriate to apply a procedure, or in which the relevant information was available). More details can be found in Supplement 6d. Note: * average reference is grouped into one option in this row; ** unique combinations of electrode layouts and their groupings into factors

Fig. 4

N400 window choices in all datasets, i.e., experiments on separate participant groups, in papers in which an N400 analysis window was reported (N = 133 datasets from 120 papers from which both sample size and latency window could be extracted). If a paper reported multiple analysis windows or multiple experiments on the same subjects, it was represented by a single window, whose lower and upper bounds were the most extreme measures of all windows reported in this paper. Bands show N400 latency ranges for all individual datasets. The heat bar in the bottom displays frequency of including each time point (1 ms) in the N400 latency range, weighted by the number of participants per condition for each dataset. Shades of green show differences between the lowest (white) and the maximum weighted frequency (dark green). This graph has been created by modifying the template made available by Neyeloff et al. (2012)

Fig. 5

The montage shows all electrodes that were used for measurement of the N400 in the main statistical analysis, regardless of the reference point. Only studies with 12 or fewer electrodes were used to generate this montage, because larger montages more frequently included analyses of the entire scalp with broadly distributed electrodes. If a paper included more than one experiment with different subjects, both experiments were included in the analysis separately. Shades of green show differences between the lowest (white) and the maximum frequency (dark green) of using an electrode, weighted by the number of participants per condition for each experiment