Intraclass correlation - A discussion and demonstration of basic features

Abstract

A re-analysis of intraclass correlation (ICC) theory is presented together with Monte Carlo simulations of ICC probability distributions. A partly revised and simplified theory of the single-score ICC is obtained, together with an alternative and simple recipe for its use in reliability studies. Our main, practical conclusion is that in the analysis of a reliability study it is neither necessary nor convenient to start from an initial choice of a specified statistical model. Rather, one may impartially use all three single-score ICC formulas. A near equality of the three ICC values indicates the absence of bias (systematic error), in which case the classical (one-way random) ICC may be used. A consistency ICC larger than absolute agreement ICC indicates the presence of non-negligible bias; if so, classical ICC is invalid and misleading. An F-test may be used to confirm whether biases are present. From the resulting model (without or with bias) variances and confidence intervals may then be calculated. In presence of bias, both absolute agreement ICC and consistency ICC should be reported, since they give different and complementary information about the reliability of the method. A clinical example with data from the literature is given.

Fig 1. Survey of models, mean square relations and ICC formulas.

Fig 2. Relation between ICC models and ICC formulas.

Three statistical models used in the intraclass correlation theory are indicated: Model 1 (one-way model); Model 2 (two-way random model); and Model 3 (two-way mixed model). The figure shows the relation between these models and the three well-known sample ICC formulas, i.e. ICC(1), ICC(A,1) and ICC(C,1).

Fig 3. Part of the computer printout from a SIMANOVA run using Model 1.

Corresponding ICC distributions are shown in Fig 4. See text for discussion.

Fig 4. Probability distributions of ICC(1), ICC(A,1) and ICC(C,1) obtained with a simulation based on Model 1, i.e. in the absence of bias.

Input data and results from this simulation are shown in Fig 3. With Model 1, the simulated distributions are seen to be identical apart from small differences due to finite statistics (finite N).

Fig 5. Probability distributions of ICC(1) values obtained with simulations using Model 1, showing the effect of increasing noise (error).

In all three cases n = 20, k = 3 and σ_r = 10. The standard deviation of the noise term is increased from σ_v = 5 (giving population ICC ρ₁ = 0.8) to σ_v = 7.5 (ρ₁ = 0.64) and σ_v = 10 (ρ₁ = 0.5).

Fig 6. Probability distributions of ICC(1) obtained with Model 1, showing the effect of increasing the number of subjects.

The number of subjects increases from n = 20 to n = 100 and n = 400. In all three cases k = 3 and the standard deviations of error and subject's score are 𝛔_v = 𝛔_r = 10, giving the population ICC = 0.5. As can be seen, an increasing n leads to a decrease in the width of the probability distribution.

Fig 7. Probability distributions of ICC(1) obtained with Model 1, showing the effect of increasing the number of measurements k.

In all four distributions, n = 20 and 𝛔_v = 𝛔_r = 10 (giving population ICC = 0.5). Increasing k leads to a decreasing width.

Fig 8. Graphic presentation of confidence limits.

The curves show the upper and lower 95% central range limits of the ICC(1) probability distributions as functions of the population ICC, using Model 1. The number of measurements is everywhere k = 3 while the number of subjects n range from 10 to 100. Read horizontally, the diagram provides the 95% confidence limits of the population ICC for a given sample ICC(1) value. For example, if n = 10 and the sample ICC(1) is found to be 0.70, then the confidence limits of the population ICC are graphically read to be approximately 0.38 and 0.91. The diagram is also valid for ICC(C,1), i.e. the consistency ICC obtained with Model 2 and Model 3.

Fig 9. Probability distributions of ICC(A,1) and ICC(C,1) obtained with Model 2, showing the effect of increasing bias.

In all cases n = 20, k = 3, σ_r = 10 and σ_v = 5. The bias standard deviation σ_c is increased from σ_c = 0.1 to σ_c = 5 and then to σ_c = 10. As may be seen, the ICC(C,1) distributions are insensitive to bias. The ICC(A,1) distributions are, with increasing bias, shifted towards lower values and broadened.

Fig 10. Effect of fixed bias: ICC(A,1) and ICC(C,1) distributions simulated using Model 3.

Two simulated Model 3 cases are shown, (a) and (b). In both cases, n = 20, k = 3, σ_r = 10 and σ_v = 5. In case (a), the fixed bias values are c₁ = 1, c₂ = 6 and c₃ = -1. In case (b), they are c₁ = 10, c₂ = 6 and c₃ = -10. A larger spread among the fixed bias values gives a larger shift of the ICC(A,1) distribution towards lower values. The ICC(C,1) distributions are however insensitive to bias.