Computing nasalance with MFCCs and Convolutional Neural Networks

Abstract

Nasalance is a valuable clinical biomarker for hypernasality. It is computed as the ratio of acoustic energy emitted through the nose to the total energy emitted through the mouth and nose (eNasalance). A new approach is proposed to compute nasalance using Convolutional Neural Networks (CNNs) trained with Mel-Frequency Cepstrum Coefficients (mfccNasalance). mfccNasalance is evaluated by examining its accuracy: 1) when the train and test data are from the same or from different dialects; 2) with test data that differs in dynamicity (e.g. rapidly produced diadochokinetic syllables versus short words); and 3) using multiple CNN configurations (i.e. kernel shape and use of 1 × 1 pointwise convolution). Dual-channel Nasometer speech data from healthy speakers from different dialects: Costa Rica, more(+) nasal, Spain and Chile, less(-) nasal, are recorded. The input to the CNN models were sequences of 39 MFCC vectors computed from 250 ms moving windows. The test data were recorded in Spain and included short words (-dynamic), sentences (+dynamic), and diadochokinetic syllables (+dynamic). The accuracy of a CNN model was defined as the Spearman correlation between the mfccNasalance for that model and the perceptual nasality scores of human experts. In the same-dialect condition, mfccNasalance was more accurate than eNasalance independently of the CNN configuration; using a 1 × 1 kernel resulted in increased accuracy for +dynamic utterances (p < .000), though not for -dynamic utterances. The kernel shape had a significant impact for -dynamic utterances (p < .000) exclusively. In the different-dialect condition, the scores were significantly less accurate than in the same-dialect condition, particularly for Costa Rica trained models. We conclude that mfccNasalance is a flexible and useful alternative to eNasalance. Future studies should explore how to optimize mfccNasalance by selecting the most adequate CNN model as a function of the dynamicity of the target speech data.

Fig 1. Overview of the proposed CNN nasalance model for hypernasality prediction.

Fig 2. Convolutional Neural Network model for hypernasality prediction.

Fig 3. Kernel shapes and phonetic information.

Fig 4. Train data annotation and classification.

Fig 5. Train data annotation and classification.

Fig 6. Correlation between e-Nasalance and perceptual scores (orange rectangle), and mfccNasalance and perceptual scores (blue) in the same-dialect condition (Spain).

Left figure shows the results for all the CNN configurations. The right figure shows the results for the CNNs using the optimal configuration (Spain: Syllable k11 = True, Words Kernels = Temporal, Sentences k11 = True).

Fig 7. Correlation between e-Nasalance and perceptual scores (orange rectangle), and mfccNasalance and perceptual scores (blue) in the different-dialect condition.

Top is for Costa Rica trained models, down for Chile. Left figures shows the results for all the CNN configurations. The right figures show the results for the CNNs using the optimal configuration (Costa Rica: Syllable k11 = True, Words Kernels = Temporal, Sentences k11 = True. Chile: Syllable Kernels = Spectral, Words k11 = False, Sentences Kernels = Spectral).