Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2021 Apr;40(9):2230-2238.
doi: 10.1002/sim.8899. Epub 2021 Feb 12.

Generalized confidence interval for an agreement between raters

Dulal K Bhaumik  1   2 , Hairong Shi  2 , Domenic J Reda  2 , Bikas K Sinha  3
Affiliations

Generalized confidence interval for an agreement between raters

Dulal K Bhaumik et al. Stat Med. 2021 Apr.

Abstract

Estimation and inference are two key components toward the solution of any statistical problem; however, the inferential issues of statistical assessment of agreement among two or more raters have not been well developed as compared to the development of estimation procedures in this area. The fundamental reason for this gap is the complex expression of the concordance correlation coefficient (CCC) that is frequently used in assessing agreement among raters. Large sample-based statistical tests for CCC often fail to produce desired results for small samples. Hence, inferential procedures for small samples are urgently needed to evaluate agreement between raters. We argue that hypothesis testing of CCC has little value in practice due to the absence of a gold standard of agreement. In this article, we construct the generalized confidence interval (GCI) for CCC utilizing a bivariate normal distribution of measurements, and also develop a large sample-based confidence interval (LSCI). We establish satisfactory performance of GCI by providing the desired coverage probability (CP) via simulation. Results of GCI and LSCI are illustrated and compared with a data set of a recent study performed at U.S. Department of Veterans Affairs, Hines.

Keywords: agreement; concordance correlation coefficient; coverage probability; generalized confidence interval.

PubMed Disclaimer

References

REFERENCES

    1. Cohen J. A coefficient of Agreement for nominal scales. Educ Psychol Meas. 1960;20(1):37-46.
    1. Cohen J. Weighed kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychol Bull. 1968;70(4):213-220.
    1. Bland JM, Altman D. Statistical methods for assessing agreement between two methods of clinical measurement. The Lancet. 1986;327(8476):307-310.
    1. Lawrence I, Lin K. A concordance correlation coefficient to evaluate reproducibility. Biometrics. 1989;45(1):255-268.
    1. Lawrence I, Lin K. Assay validation using the concordance correlation coefficient. Biometrics. 1992;48(2):599-604.

Publication types