Evaluation of emotion classification schemes in social media text: an annotation-based approach

Abstract

Background: Emotion analysis of social media texts is an innovative method for gaining insight into the mental state of the public and understanding social phenomena. However, emotion is a complex psychological phenomenon, and there are various emotion classification schemes. Which one is suitable for textual emotion analysis?

Methods: We proposed a framework for evaluating emotion classification schemes based on manual annotation experiments. Considering both the quality and efficiency of emotion analysis, we identified five criteria, which are solidity, coverage, agreement, compactness, and distinction. Qualitative and quantitative factors were synthesized using the AHP, where quantitative metrics were derived from annotation experiments. Applying this framework, 2848 Sina Weibo posts related to public events were used to evaluate the five emotion schemes: SemEval's four emotions, Ekman's six basic emotions, ancient China's Seven Emotions, Plutchik's eight primary emotions, and GoEmotions' 27 emotions.

Results: The AHP evaluation result shows that Ekman's scheme had the highest score. The multi-dimensional scaling (MDS) analysis shows that Ekman, Plutchik, and the Seven Emotions are relatively similar. We analyzed Ekman's six basic emotions in relation to the emotion categories of the other schemes. The correspondence analysis shows that the Seven Emotions' joy aligns with Ekman's happiness, love demonstrates a significant correlation with happiness, but desire is not significantly correlated with any emotion. Compared to Ekman, Plutchik has two more positive emotions: trust and anticipation. Trust is somewhat associated with happiness, but anticipation is weakly associated with happiness. Each emotion of Ekman's corresponds to several similar emotions in GoEmotions. However, some emotions in GoEmotions are not clearly related to Ekman's, such as approval, love, pride, amusement, etc. CONCLUSION: Ekman's scheme performs best under the evaluation framework. However, it lacks sufficient positive emotion categories for the corpus.

Keywords: Annotation; Emotion classification scheme; Evaluation; Social media.

Fig. 1

The hierarchy model for the evaluation of the five emotion schemes

Fig. 2

Word count distribution of 2848 posts

Fig. 3

Distribution of emotions based on the SemEval scheme

Fig. 4

Distribution of emotions based on the Ekman scheme

Fig. 5

Distribution of emotions based on the Seven Emotions scheme

Fig. 6

Distribution of emotions based on the Plutchik scheme

Fig. 7

Distribution of emotions based on the GoEmotions scheme

Fig. 8

The scores of the five emotion schemes

Fig. 9

Performance of the five emotion schemes under the five criteria

Fig. 10

Multidimensional scaling of the five emotion schemes

Fig. 11

Ekman (red dot) versus SemEval (blue triangle)

Fig. 12

Ekman (red dot) versus Seven Emotions (blue triangle)

Fig. 13

Ekman (red dot) versus Plutichik (blue triangle)

Fig. 14

Ekman (red dot) versus GoEmotions (blue triangle)