Improving practices and inferences in developmental cognitive neuroscience

Abstract

The past decade has seen growing concern about research practices in cognitive neuroscience, and psychology more broadly, that shake our confidence in many inferences in these fields. We consider how these issues affect developmental cognitive neuroscience, with the goal of progressing our field to support strong and defensible inferences from our neurobiological data. This manuscript focuses on the importance of distinguishing between confirmatory versus exploratory data analysis approaches in developmental cognitive neuroscience. Regarding confirmatory research, we discuss problems with analytic flexibility, appropriately instantiating hypotheses, and controlling the error rate given how we threshold data and correct for multiple comparisons. To counterbalance these concerns with confirmatory analyses, we present two complementary strategies. First, we discuss the advantages of working within an exploratory analysis framework, including estimating and reporting effect sizes, using parcellations, and conducting specification curve analyses. Second, we summarize defensible approaches for null hypothesis significance testing in confirmatory analyses, focusing on transparent and reproducible practices in our field. Specific recommendations are given, and templates, scripts, or other resources are hyperlinked, whenever possible.

Keywords: Exploratory; Inference; Parcellations; Preregistration; Reproducibility; Thresholding.

Fig. 1

Sample specification curve analysis of 64 unique linear regression models with individual well-being scores as the criterion. Model specifications are ordered based on the parameter estimate for the association between individual whole-brain pattern expression of a multivariate neural signature of self-referential processing and well-being scores. Each vertical column corresponds to a single model specification. The regression coefficient for each model specification is plotted in panel A and the variables included in each model are visualized in panel B. Models in which the association between multivariate expression and well-being score is statistically significant at p <  .05 are highlighted in red. Error bars represent 95 % confidence intervals. Control variables included depression scores and mean reaction time. Outliers were defined as being more than 2.5 standard deviations from the mean for each variable. pgACC = perigenual anterior cingulate cortex, vmPFC = ventromedial prefrontal cortex, VS = ventral striatum. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article).