A Novel 3D Convolutional Neural Network-Based Deep Learning Model for Spatiotemporal Feature Mapping for Video Analysis: Feasibility Study for Gastrointestinal Endoscopic Video Classification

Abstract

Accurate analysis of medical videos remains a major challenge in deep learning (DL) due to the need for effective spatiotemporal feature mapping that captures both spatial detail and temporal dynamics. Despite advances in DL, most existing models in medical AI focus on static images, overlooking critical temporal cues present in video data. To bridge this gap, a novel DL-based framework is proposed for spatiotemporal feature extraction from medical video sequences. As a feasibility use case, this study focuses on gastrointestinal (GI) endoscopic video classification. A 3D convolutional neural network (CNN) is developed to classify upper and lower GI endoscopic videos using the hyperKvasir dataset, which contains 314 lower and 60 upper GI videos. To address data imbalance, 60 matched pairs of videos are randomly selected across 20 experimental runs. Videos are resized to 224 × 224, and the 3D CNN captures spatiotemporal information. A 3D version of the parallel spatial and channel squeeze-and-excitation (P-scSE) is implemented, and a new block called the residual with parallel attention (RPA) block is proposed by combining P-scSE3D with a residual block. To reduce computational complexity, a (2 + 1)D convolution is used in place of full 3D convolution. The model achieves an average accuracy of 0.933, precision of 0.932, recall of 0.944, F1-score of 0.935, and AUC of 0.933. It is also observed that the integration of P-scSE3D increased the F1-score by 7%. This preliminary work opens avenues for exploring various GI endoscopic video-based prospective studies.

Keywords: 3D convolutional neural network; P-scSE3D; deep learning; gastrointestinal endoscopic video classification; hyperKvasir dataset; spatiotemporal features.

Figure 1

Proposed model. (a) Model architecture, (b) residual with parallel attention (RPA) block, which is the core of the model, and (c) residual block.

Figure 2

Parallel spatial and channel squeeze-and-excitation (P-scSE) module [47,50].

Figure 3

Representative examples of (a,b) upper GI image frame from endoscopic video, and (c,d) lower GI image frame from endoscopic video (Bottom).

Figure 4

Video segment generation. (a) Original video, (b) video split into segments. Zero-padded frames are added to keep the segment size fixed, and (c) the structure of a segment. A frame gap is used to skip some frames.

Figure 5

(top) ROC curves and (bottom) confusion matrices for different test accuracies.

Figure 6

Loss and accuracy curves for training and validation for different test accuracies.

Figure 7

Explainable AI (XAI). Guided Grad-CAM is used as XAI. The 1st and 3rd rows indicate the original videos. The 2nd and 4th rows blend the heatmap on them. The yellow zone indicates the more focused zone used for classification.