Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2023 Jul 14;18(7):e0288069.
doi: 10.1371/journal.pone.0288069. eCollection 2023.

Which method is optimal for estimating variance components and their variability in generalizability theory? evidence form a set of unified rules for bootstrap method

Guangming Li  1
Affiliations

Which method is optimal for estimating variance components and their variability in generalizability theory? evidence form a set of unified rules for bootstrap method

Guangming Li. PLoS One. .

Abstract

Objective: The purpose of this study is to compare the performance of the four estimation methods (traditional method, jackknife method, bootstrap method, and MCMC method), find the optimal one, and make a set of unified rules for Bootstrap.

Methods: Based on four types of simulated data (normal, dichotomous, polytomous, and skewed data), this study estimates and compares the estimated variance components and their variability of the four estimation methods when using a p×i design in generalizability theory. The estimated variance components are vc.p, vc.i and vc.pi and the variability of estimated variance components are their estimated standard errors (SE(vc.p), SE(vc.i) and SE(vc.pi)) and confidence intervals (CI(vc.p), CI(vc.i) and CI(vc.pi)).

Results: For the normal data, all the four methods can accurately estimate the variance components and their variability. For the dichotomous data, the |RPB| of SE (vc.i) of traditional method is 128.5714, the |RPB| of SE (vc.i), SE (vc.pi) and CI (vc.i) of jackknife method are 42.8571, 43.6893 and 40.5000, which are larger than 25 and not accurate. For the polytomous data, the |RPB| of SE (vc.i) and CI (vc.i) of MCMC method are 59.6612 and 45.2500, which are larger than 25 and not accurate. For the skewed data, the |RPB| of SE (vc.p), SE (vc.i) and SE (vc. pi) of traditional method and MCMC method are over 25, which are not accurate. Only the bootstrap method can estimate variance components and their variability accurately across different data distribution. Nonetheless, the divide-and-conquer strategy must be used when adopting the bootstrap method.

Conclusions: The bootstrap method is optimal among the four methods and shows the cross-distribution superiority over the other three methods. However, a set of unified rules for the divide-and-conquer strategy need to be recommended for the bootstrap method, which is optimal when boot-p for p (person), boot-pi for i (item), and boot-i for pi (person × item).

PubMed Disclaimer

Conflict of interest statement

The authors have declared that no competing interests exist.

References

    1. Clayson P. E., Carbine K. A., Baldwin S. A., Olsen J. A., & Larson M. J. (2021). Using generalizability theory and the erp reliability analysis (era) toolbox for assessing test-retest reliability of erp scores part 1: Algorithms, framework, and implementation. International Journal of Psychophysiology, 166, 174–187. doi: 10.1016/j.ijpsycho.2021.01.006 - DOI - PubMed
    1. Vispoel W. P., Xu G., & Kilinc M. (2020). Expanding G-Theory models to incorporate congeneric relationships: Illustrations using the big five inventory. Journal of Personality Assessment, 103(1), 429–442. doi: 10.1080/00223891.2020.1808474 - DOI - PubMed
    1. Li G. (2023). How many students and items are optimal for teaching level evaluation of college teachers? Evidence from generalizability theory and Lagrange multiplier. Sustainability, 15, 2.
    1. Brennan R. L. (2001). Generalizability theory. New York: Springer-Verlag.
    1. Gao X., & Brennan R. L. (2001). Variability of estimated variance components and related statistics in a performance assessment. Applied Measurement in Education, 14(2), 191–203.