Notes on correctness of p-values when analyzing experiments using SAS and R

Abstract

It is commonly believed that if a two-way analysis of variance (ANOVA) is carried out in R, then reported p-values are correct. This article shows that this is not always the case. Results can vary from non-significant to highly significant, depending on the choice of options. The user must know exactly which options result in correct p-values, and which options do not. Furthermore, it is commonly supposed that analyses in SAS and R of simple balanced experiments using mixed-effects models result in correct p-values. However, the simulation study of the current article indicates that frequency of Type I error deviates from the nominal value. The objective of this article is to compare SAS and R with respect to correctness of results when analyzing small experiments. It is concluded that modern functions and procedures for analysis of mixed-effects models are sometimes not as reliable as traditional ANOVA based on simple computations of sums of squares.

Fig 1. Quantiles of observed p-values, testing H₀ : μ_1. = μ_2. = μ_3. versus quantiles of the uniform distribution U(0,1).

Based on 10 000 simulated datasets using model (12) with layout {a = 3, b = 2, c = 12} and parameter values ${\sigma }_{\text{e}}^2=1$, ${\sigma }_{\text{b}}^2=1$ and ${\sigma }_{\text{AB}}^2=0.1$. (A) Exact F-test. (B) R using the lmer and anova functions. (C) SAS using the mixed procedure with default settings. (D) SAS using the mixed procedure with the nobound option.