Three-dimensional DenseNet self-attention neural network for automatic detection of student's engagement

Abstract

Today, due to the widespread outbreak of the deadly coronavirus, popularly known as COVID-19, the traditional classroom education has been shifted to computer-based learning. Students of various cognitive and psychological abilities participate in the learning process. However, most students are hesitant to provide regular and honest feedback on the comprehensiveness of the course, making it difficult for the instructor to ensure that all students are grasping the information at the same rate. The students' understanding of the course and their emotional engagement, as indicated via facial expressions, are intertwined. This paper attempts to present a three-dimensional DenseNet self-attention neural network (DenseAttNet) used to identify and evaluate student participation in modern and traditional educational programs. With the Dataset for Affective States in E-Environments (DAiSEE), the proposed DenseAttNet model outperformed all other existing methods, achieving baseline accuracy of 63.59% for engagement classification and 54.27% for boredom classification, respectively. Besides, DenseAttNet trained on all four multi-labels, namely boredom, engagement, confusion, and frustration has registered an accuracy of 81.17%, 94.85%, 90.96%, and 95.85%, respectively. In addition, we performed a regression experiment on DAiSEE and obtained the lowest Mean Square Error (MSE) value of 0.0347. Finally, the proposed approach achieves a competitive MSE of 0.0877 when validated on the Emotion Recognition in the Wild Engagement Prediction (EmotiW-EP) dataset.

Keywords: Attention network; Engagement recognition; Online learning; Spatio-temporal features.

Fig. 1

Proposed pipeline for automatic students’ engagement detection

Fig. 2

DenseNet, self-attention layer, and FC classification layers are the three components of the proposed DenseAttNet

Fig. 3

(a) 3D spatial self-attention architecture (b) 3D temporal self-attention architecture. Where ⊗ denotes element-wise multiplication operation and ⊕ denotes element-wise addition operation. Here, temporal or depth is an interchangeable term

Fig. 4

The spatial-attention module (which uses the Fig. 3a spatial self-attention architecture), the temporal-attention module (uses the Fig. 3b temporal self-attention architecture), and the hybrid module includes both spatial and temporal self-attention architecture

Fig. 5

Confusion matrix of the proposed DenseAttNet with a spatial self-attention module tested on DAiSEE with (a) Cross-Entropy loss (CE) (b) CB-FL (β = 0.9, γ = 1.0) (c) CB-FL β = 0.9, γ = 2.0) (d) CB-FL (β = 0.99, γ = 1.0) (e) CB-FL (β = 0.99, γ = 2.0) (f) CB-FL (β = 0.999, γ = 1.0)

Fig. 6

Confusion matrix of the proposed DenseAttNet with a temporal self-attention module tested on DAiSEE with (a) CE (b) CB-FL (β = 0.9, γ = 1.0) (c) CB-FL (β = 0.9, γ = 2.0) (d) CB-FL (β = 0.99, γ = 1.0) (e) CB-FL (β = 0.99, γ = 2.0) (f) CB-FL (β = 0.999, γ = 1.0)

Fig. 7

Confusion matrix of the proposed DenseAttNet with a hybrid self-attention module tested on DAiSEE with (a) CE (b) CB-FL (β = 0.9, γ = 1.0) (c) CB-FL (β = 0.9, γ = 2.0) (d) CB-FL (β = 0.99, γ = 1.0) (e) CB-FL (β = 0.99, γ = 2.0) (f) CB-FL (β = 0.999, γ = 1.0)

Fig. 8

Confusion matrix for binary engagement prediction on DAiSEE using the proposed DenseAttNet method. We have used hybrid self-attention module with (a) CB-FL (γ = 1, β = 0.9) (b) CB-FL (γ = 1, β = 0.99)

Fig. 9

Grad-CAM results on DAiSEE (a) engagement level “0” (b) engagement level “1” (c) engagement level “2” (d) engagement level “3”

Fig. 10

Models’ Grad-CAM visualization of engagement prediction on DAiSEE

Fig. 11

An illustration of the multi-label DenseAttNet models’ deep features learned from a multi-label DAiSEE dataset. Model trained using focal loss (a) β = 0.9 and γ = 1 (b) β = 0.99 and γ = 1. Where boredom (B), engagement (E), confusion (C), frustration (F), and None (-)

Fig. 12

This figure illustrates instances of non-frontal and occluded faces

Fig. 13

Ten consecutive sequential frames of the same subject taken from DAiSEE, each labeled differently. a Sequence with “very low engagement”, b sequence with “low engagement”, c sequence with “high engagement”, and d sequence with “very high engagement”

Fig. 14

Multi-label DenseAttNet algorithm-generated graph analytics samples of students’ engagement during a lecture in a traditional classroom setting