A solution to minimum sample size for regressions

Abstract

Regressions and meta-regressions are widely used to estimate patterns and effect sizes in various disciplines. However, many biological and medical analyses use relatively low sample size (N), contributing to concerns on reproducibility. What is the minimum N to identify the most plausible data pattern using regressions? Statistical power analysis is often used to answer that question, but it has its own problems and logically should follow model selection to first identify the most plausible model. Here we make null, simple linear and quadratic data with different variances and effect sizes. We then sample and use information theoretic model selection to evaluate minimum N for regression models. We also evaluate the use of coefficient of determination (R2) for this purpose; it is widely used but not recommended. With very low variance, both false positives and false negatives occurred at N < 8, but data shape was always clearly identified at N ≥ 8. With high variance, accurate inference was stable at N ≥ 25. Those outcomes were consistent at different effect sizes. Akaike Information Criterion weights (AICc wi) were essential to clearly identify patterns (e.g., simple linear vs. null); R2 or adjusted R2 values were not useful. We conclude that a minimum N = 8 is informative given very little variance, but minimum N ≥ 25 is required for more variance. Alternative models are better compared using information theory indices such as AIC but not R2 or adjusted R2. Insufficient N and R2-based model selection apparently contribute to confusion and low reproducibility in various disciplines. To avoid those problems, we recommend that research based on regressions or meta-regressions use N ≥ 25.

Fig 1. Histograms of N in research.

(a) economic meta-analyses & meta-regressions; (b) medical / epidemiological meta-analyses & meta-regressions; (c) ecological analyses of disturbance [8]; and (d) biogeographical analyses of species-area relationships [9]. Please see S1 Appendix for a description of literature search methods, data, and references for (a) and (b).

Fig 2. Data made with a null model (1^st column) and results of analyses using null (2^nd column), straight-line (3^rd column) and quadratic (4^th column) models.

Data with (a) high variance and (b) low variance were each analyzed at N = 4–50. Results are presented with maximum N = 30 for visual clarity; all results stabilized at N > 30. Circles are means; error bars are 95% confidence intervals. “Traffic signal” colors on sample size (N) axes for the null model indicate ranges where N is too small (red = stop), or sufficient (green = go) to correctly infer the pattern. Note the quadratic model outcomes at N = 4 (red circles).

Fig 3. Data made with a straight-line model (1^st column) and results of analyses using null (2^nd column), straight-line (3^rd column) and quadratic (4^th column) models.

The four combinations (a-d) of low/high variance (σ) and effect size (α) represent approximate graphical extremes. Grey lines represent transitions in leading w_i between two models. “Traffic signal” colors on sample size (N) axes for the straight-line model indicate ranges where N is too small (red = stop), about equivalent to the null (yellow = caution), or sufficient (green = go) to correctly infer the pattern. Note the null and quadratic model outcomes at low N (red circles or ellipses).

Fig 4. Data made with a quadratic model and with high variance (σ; 1^st column) and results of analyses using null (2^nd column), straight-line (3^rd column) and quadratic (4^th column) models.

All else as in Figs 2 & 3.

Fig 5. Data made with a quadratic model and with low variance (σ; 1^st column) and results of analyses using null (2^nd column), straight-line (3^rd column) and quadratic (4^th column) models.

All else as in Figs 2–4.