The Reliability and Validity of Speech-Language Pathologists' Estimations of Intelligibility in Dysarthria

Abstract

This study examined the reliability and validity of speech-language pathologists' (SLP) estimations of speech intelligibility in dysarthria, including a visual analog scale (VAS) method and a percent estimation method commonly used in clinical settings. Speech samples from 20 speakers with dysarthria of varying etiologies were used to collect orthographic transcriptions from naïve listeners n=70 and VAS ratings and percent estimations of intelligibility from SLPs n=21. Intra- and interrater reliability for the two SLP intelligibility measures were evaluated, and the relationship between these measures was assessed. Finally, linear regression was used to evaluate the relationship between the naïve listeners' orthographic transcription scores and the two SLP intelligibility measures. The results indicated that the intrarater reliability for both SLP intelligibility measures was strong, and the interrater reliability between the SLP ratings was moderate to excellent. A moderate positive relationship between SLPs' VAS ratings and percent estimations was also observed. Finally, both SLPs' percent estimations and VAS ratings were predictive of naïve listeners' orthographic transcription scores, with SLPs' percent estimations being the strongest predictor. In conclusion, the average SLP percent estimations and VAS ratings are valid and reliable intelligibility measures. However, the validity and reliability of these measures vary between SLPs.

Keywords: dysarthria; speech intelligibility; speech-language pathologists.

Figure 1

Study Procedures for the SLP and Naïve Listeners. Passage refers to segments of The Grandfather passage. Consent was obtained prior to the study procedures depicted in this figure. VAS = visual analog scale ratings.

Figure 2

Distribution of Responses for Both the Naïve Listeners’ and SLPs’ Intelligibility Measures. For each of the three ratings investigated in the study, this figure shows the distribution of ratings (top), a box and whisker plot for the ratings (middle), in which the line within the box depicts the median, the left and right ends of the box represent the first and third quartile, respectively, and the left and right “whiskers” depict the smallest and largest rating. Finally, this figure shows the individual listener ratings across all 20 speakers, as indicated by the semi-transparent data points (bottom). Individual data points that appear to be darker signify a high-density region of overlapping responses. SLP = speech-language pathologists, VAS = visual analog scale.

Figure 3

Relationship between SLPs’ VAS Ratings and Percent Estimations. The lighter lines depict the relationship between percent estimations and VAS ratings for each SLP in the study, while the darker line represents the average relationship between these two measures. The black dotted line represents the perfect theoretical relationship between these two measures. Alternate visualization for this figure can be found in the supplementary materials (https://osf.io/sr9aw/) (accessed on 21 June 2022).

Figure 4

SLPs’ Percent Estimates and VAS Ratings of Intelligibility as Predictors of Naïve Listeners’ Orthographic Transcriptions. OT = orthographic transcription; SLP = Speech-language pathologists, VAS = visual analog scale. The light gray line connects the data points for each of the 20 speakers. Two speakers, AM1 and HDM10, are highlighted within the figure and discussed in Section 4.2 of the discussion.