Better statistical reporting does not lead to statistical rigour: lessons from two decades of pseudoreplication in mouse-model studies of neurological disorders

Abstract

Background: Accurately determining the sample size ("N") of a dataset is a key consideration for experimental design. Misidentification of sample size can lead to pseudoreplication, a process of artificially inflating the number of experimental replicates which systematically underestimates variability, overestimates effect sizes and invalidates statistical tests performed on the data. While many journals have adopted stringent requirements with regard to statistical reporting over the last decade, it remains unknown whether such efforts have had a meaningful impact on statistical rigour.

Methods: Here, we evaluated the prevalence of this type of statistical error among neuroscience studies involving animal models of Fragile-X Syndrome (FXS) and those using animal models of neurological disorders at large published between 2001 and 2024.

Results: We found that pseudoreplication was present in the majority of publication, increasing over time despite marked improvements in statistical reporting over the last decade. This trend generalised beyond the FXS literature to rodent studies of neurological disorders at large between 2012 and 2024, suggesting that pseudoreplication remains a widespread issue in the literature.

Limitations: The scope of this study was limited to rodent-model studies of neurological disorders which had the potential for being pseudoreplicated, by allowing repeat observations from individual animals. We did not consider reviews or articles whose experimental design could not allow for pseudoreplication, for example studies which reported only behavioural results, or studies which did not use inferential statistics.

Conclusions: These observations identify an urgent need for better standards in experimental design and increased vigilance for this type of error during peer review. While reporting standards have significantly improved over the past two decades, this alone has not been enough to curb the prevalence of pseudoreplication. We offer suggestions for how this can be remedied as well as quantifying the severity of this particular type of statistical error. Although the examined literature concerns a specific neuroscience-related area of research, the implications of pseudoreplication apply to all fields of empirical research.

Keywords: Animal models; Autism; Fragile X; Pseudoreplication; Statistics.

Fig. 1

The prevalence of pseudoreplication in the Fragile-X mouse model literature has remained fairly constant despite marked improvements in statistical reporting. (A) Proportion of articles suspected of pseudoreplication in at least one figure (orange line), and proportion reporting adequate statistical details (green line) in articles sampled between 2001 and 2024. Each time-point shows the bootstrap resampled median and 95% percentile interval of the bootstrapped distribution. (B) Average percentage of articles suspected of pseudoreplication between 2001–2012 and 2013–2024. Whisker plots show median ± 95% CI of bootstrapped distribution per group. (C) Average percentage of articles reporting adequate statistical details between 2001–2012 and 2013–2024. (D) Probability of an article being suspected of pseudoreplication when reporting adequate (dark green) or inadequate (light green) statistical details between 2001–2012 and 2013–2024. (E) Median citation rate for articles where pseudoreplication was present (dark orange) or absent (light orange) between 2001–2012 and 2013–2024. (F) Median citation rate for articles reporting adequate (dark green) or inadequate (light green) statistical details between 2001–2012 and 2013–2024

Fig. 2

The prevalence of pseudoreplication across all publications using animal models of neurological disorder has remained high despite improvements in statistical reporting. (A) Average percentage of articles suspected of pseudoreplication between 2001–2012 and 2013–2024. Whisker plots show median ± 95% CI of bootstrapped distribution per group. (B) Average percentage of articles reporting adequate statistical details between 2001–2012 and 2013–2024. (C) Probability of an article being suspected of pseudoreplication when reporting adequate (dark green) or inadequate (light green) statistical details between 2001–2012 and 2013–2024. (D) Median citation rate for articles where pseudoreplication was present (dark orange) or absent (light orange) between 2001–2012 and 2013–2024. (E) Median citation rate for articles reporting adequate (dark green) or inadequate (light green) statistical details between 2001–2012 and 2013–2024

Fig. 3

A typical example of pseudoreplication and its consequences for Student’s t-test. (A) Many experimental designs use within-animal samples to draw conclusions about the effect of a gene or environmental condition. The total variability in an effect can be split into a within-animal component (between cells, in this case) and a between-animal component. The intra-class correlation coefficient, ρ_IC, is a measure of how these sources of variation are related. (B) Schematic of variability relationships between and within animals for low (left population) and high (right population) intra-class correlation. Note that animals in the population on the left have high variance between cells (within animal), whereas animals in the population on the right have low cell variance in any given animal. (C) Pseudoreplicating by considering within-animal replicates as experimental replicates inflates the true Type-I error rate (false positive rate). X indicates the example case given in the text. The curves show how the true Type-I error rate varies with the number of within animal replicates for commonly stipulated significance levels (5%, 1%, 0.1%) and for the possible range of between-animal replicates (solid curves = 2 animals, dotted curves = infinite animals). For all curves, ρ_IC is set to 0.5. (D) The combined effect of within-animal replicates and intra-class correlation (ρ_IC) on the Type-I error rate for a significance threshold of 5% in the presence of pseudoreplication. Between animal standard deviation is shown normalised to within-animal standard deviation for comparison with corresponding values of ρ_IC