Assessing community variation and randomness in public health indicators

Abstract

Background: Evidence-based health indicators are vital to needs-based programming and epidemiological planning. Agencies frequently make programming funds available to local jurisdictions based on need. The use of objective indicators to determine need is attractive but assumes that selection of communities with the highest indicators reflects something other than random variability from sampling error.

Methods: The authors compare the statistical performance of two heterogeneity measures applied to community differences that provide tests for randomness and measures of the percentage of true community variation, as well as estimates of the true variation. One measure comes from the meta-analysis literature and the other from the simple Pearson chi-square statistic. Simulations of populations and an example using real data are provided.

Results: The measure based on the simple chi-square statistic seems superior, offering better protection against Type I errors and providing more accurate estimates of the true community variance.

Conclusions: The heterogeneity measure based on Pearson's χ2 should be used to assess indices. Methods for improving poor indices are discussed.

Figure 1

Plot of intraclass correlations and $I_{DL}^2$ (blue +) and $I_P^2$ (green o) values against 100% agreement (red line).

Figure 2

$I_{DL}^2$ (blue +) and $I_P^2$ (green o) estimated and actual variances (red line) in 20,000 simulated samples.