Characterizing correlations in partial credit speech recognition scoring with beta-binomial distributions

Abstract

Partial credit scoring for speech recognition tasks can improve measurement precision. However, assessing the magnitude of this improvement with partial credit scoring is challenging because meaningful speech contains contextual cues, which create correlations between the probabilities of correctly identifying each token in a stimulus. Here, beta-binomial distributions were used to estimate recognition accuracy and intraclass correlation for phonemes in words and words in sentences in listeners with cochlear implants (N = 20). Estimates demonstrated substantial intraclass correlation in recognition accuracy within stimuli. These correlations were invariant across individuals. Intraclass correlations should be addressed in power analysis of partial credit scoring.

Fig. 1.

Example beta-binomial distributions of phoneme recognition accuracy within monosyllabic words across varying levels of correct response probability (μ) and intraclass correlation (ρ). Nonzero values of ρ indicate overdispersion of recognition accuracy across phonemes within words, which reflects the use of context to facilitate recognition.

Fig. 2.

Group-level distribution of correct phonemes within MSTB CNC words and words within PRESTO sentences are shown as bars. Beta-binomial distributions are shown as solid black lines and binomial distributions are shown as dashed gray lines. PRESTO sentences have a variable number of keywords per trial, so number of keywords correct is shown as a proportion out of total keywords.

Fig. 3.

Individual differences in recognition accuracy μ_p and intraclass correlation ρ_p estimated via Bayesian modeling. Most likely values for model parameters for each participant are shown as circles, with colors representing different participants. Shaded gradients around each most likely value show the corresponding posterior probability density for both model parameters for that participant, with darker shades corresponding to higher probability density. The dashed black line shows the most likely value for ρ_group.

Fig. 4.

The standard deviation of the mean for repeated draws from whole word scoring (n =1, μ = 0.5) and a three-phoneme beta-binomial distribution (n = 3, μ = 0.5, ρ = 0.35) as a function of the number of words tested. Dashed lines show the comparison of the number of trials required to equate standard deviation of the mean for the beta-binomial distribution with whole word scoring for 50 words.