A confidence interval for the wallace coefficient of concordance and its application to microbial typing methods

Abstract

Very diverse research fields frequently deal with the analysis of multiple clustering results, which should imply an objective detection of overlaps and divergences between the formed groupings. The congruence between these multiple results can be quantified by clustering comparison measures such as the Wallace coefficient (W). Since the measured congruence is dependent on the particular sample taken from the population, there is variability in the estimated values relatively to those of the true population. In the present work we propose the use of a confidence interval (CI) to account for this variability when W is used. The CI analytical formula is derived assuming a Gaussian sampling distribution and recurring to the algebraic relationship between W and the Simpson's index of diversity. This relationship also allows the estimation of the expected Wallace value under the assumption of independence of classifications. We evaluated the CI performance using simulated and published microbial typing data sets. The simulations showed that the CI has the desired 95% coverage when the W is greater than 0.5. This behaviour is robust to changes in cluster number, cluster size distributions and sample size. The analysis of the published data sets demonstrated the usefulness of the new CI by objectively validating some of the previous interpretations, while showing that other conclusions lacked statistical support.

Figure 1. Coverage and amplitude of 95% confidence intervals for Wallace coefficient obtained from simulated classifications.

Each dot represents a simulation with a particular set of parameters. The colors indicate the dimensions of the simulated contingency tables as indicated in the figure legend, which correspond to the number of clusters in each of the two classifications. All simulated tables in this plot had n = 300 elements and the distribution of row cluster sizes followed a Zipfian distribution with exponent a = 1.

Figure 2. Coverage and amplitude of 95% confidence intervals for Wallace coefficient obtained from simulated classifications.

Each dot represents a simulation with a particular set of parameters. The colors indicate the number of elements n of the simulated contingency tables as indicated in the figure legend. All simulated tables in this plot had 10×10 dimensions and the distribution of row cluster sizes followed a Zipfian distribution with exponent a = 1.

Figure 3. Coverage and amplitude of 95% confidence intervals for Wallace coefficient obtained from simulated classifications.

Each dot represents a simulation with a particular set of parameters. The colors indicate exponent a of the Zipfian distribution determining the distribution of row cluster sizes of the simulated contingency tables as indicated in the figure legend. All simulated tables in this plot had n = 300 elements and 10×10 dimensions.