The Effect of Visual Sort and Rate Versus Visual Analog Scales on the Reliability of Judgments of Dysphonia

Abstract

Purpose The reliability of auditory-perceptual judgments between listeners is a long-standing problem in the assessment of voice disorders. The purpose of this study was to determine whether a relatively novel experimental scaling method, called visual sort and rate (VSR), yielded stronger reliability than the more frequently used method of visual analog scales (VAS) for ratings of overall severity (OS) and breathiness (BR) in speakers with voicedisorders. Method Fifty speech samples were selected from a database of speakers with voice disorders. Twenty-two inexperienced listeners provided ratings of OS or BR in four rating blocks: VSR-OS, VSR-BR, VAS-OS, and VSR-BR. For the VAS task, listeners rated each speaker for BR or OS using a vertically oriented 100-mm VAS. For the VSR task, stimuli were distributed into sets of samples with a range of speaker severities in each set. Listeners sorted and ranked samples for OS or BR within each set, and final ratings were captured on a vertically oriented 100-mm VAS. Interrater variability, defined as the mean of the squared differences between a listener's ratings and group mean ratings, and intrarater reliability (Pearson r) were compared across rating tasks for OS and BR using paired t tests. Results Results showed that listeners had significantly less interrater variability (better reliability) when using VSR methods compared to VAS for judgments of both OS and BR. Intrarater reliability was high across rating tasks and dimensions; however, ratings of BR were significantly more consistent within individual listeners when using VAS than when using VSR. Conclusions VSR is an experimental method that decreases variability of auditory-perceptual judgments between inexperienced listeners when rating speakers with a range of dysphonic severities and disorders. Future research should determine whether a clinically viable tool may be developed based on VSR principles and whether such benefits extend to experienced listeners.

Figure 1.

Computer interfaces for (a) visual analog scale and (b) visual sort and rate.

Figure 2.

Differences in variability scores for each listener by rating task (visual analog scale [VAS] minus visual sort and rate [VSR]) for (a) overall severity and (b) breathiness. A positive value indicates a reduction in variability (i.e., better interrater reliability) for VSR compared to VAS, and a negative value indicates an increase in variability (i.e., worse interrater reliability) for VSR compared to VAS. A score of 0 indicates no difference in listener variability between the scale types. OS = overall severity; BR = breathiness.

Figure 3.

Voice sample variability scores for judgments of (a) overall severity (OS) and (b) breathiness (BR) as a function of severity and rating task, visual analog scale (VAS) versus visual sort and rate (VSR; note, for clarity of visualization, data points not pictured in panel b for VAS at (51, 1213) and (65, 1189), and for VSR at (53, 870), though all were included in regression analyses). Curves were fitted using polynomial regression for descriptive purposes.