Autoscore: An open-source automated tool for scoring listener perception of speech

Abstract

Speech perception studies typically rely on trained research assistants to score orthographic listener transcripts for words correctly identified. While the accuracy of the human scoring protocol has been validated with strong intra- and inter-rater reliability, the process of hand-scoring the transcripts is time-consuming and resource intensive. Here, an open-source computer-based tool for automated scoring of listener transcripts is built (Autoscore) and validated on three different human-scored data sets. Results show that not only is Autoscore highly accurate, achieving approximately 99% accuracy, but extremely efficient. Thus, Autoscore affords a practical research tool, with clinical application, for scoring listener intelligibility of speech.

FIG. 1.

(Color online) The interface of the online application.

FIG. 2.

(Color online) The error rates of scoring orthographic listener transcripts for a measure of words correct for Autoscore and human scorers for both in-house data sets.