A Cantonese Audio-Visual Emotional Speech (CAVES) dataset

Abstract

We present a Cantonese emotional speech dataset that is suitable for use in research investigating the auditory and visual expression of emotion in tonal languages. This unique dataset consists of auditory and visual recordings of ten native speakers of Cantonese uttering 50 sentences each in the six basic emotions plus neutral (angry, happy, sad, surprise, fear, and disgust). The visual recordings have a full HD resolution of 1920 × 1080 pixels and were recorded at 50 fps. The important features of the dataset are outlined along with the factors considered when compiling the dataset. A validation study of the recorded emotion expressions was conducted in which 15 native Cantonese perceivers completed a forced-choice emotion identification task. The variability of the speakers and the sentences was examined by testing the degree of concordance between the intended and the perceived emotion. We compared these results with those of other emotion perception and evaluation studies that have tested spoken emotions in languages other than Cantonese. The dataset is freely available for research purposes.

Keywords: Auditory and visual expressions; Cantonese dataset; Dataset evaluation; Emotional speech.

Fig. 1

The setup in the recording booth showing the camera, screen microphone, lighting and participants’ seat

Fig. 2

A single frame extracted from video clip to illustrate the extent to which the video was cropped

Fig. 3

Percent accuracy scores for all emotion types by presentation conditions (model-based standard error)

Fig. 4

Mean percent correct recognition score for each speaker in the CAVES dataset. Note. Female speakers were given identifiers that started with ‘F’ with a number from 1 to 5 to denote each individual speaker. Similarly, males were given identifiers that started with ‘M’

Fig. 5

Mean percent correct recognition scores for all 50 sentences across the six emotion types