Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2022 Sep:13433:749-759.
doi: 10.1007/978-3-031-16437-8_72. Epub 2022 Sep 16.

Attentional Generative Multimodal Network for Neonatal Postoperative Pain Estimation

Md Sirajus Salekin  1 , Ghada Zamzmi  1 , Dmitry Goldgof  1 , Peter R Mouton  2 , Kanwaljeet J S Anand  3 , Terri Ashmeade  1 , Stephanie Prescott  1 , Yangxin Huang  1 , Yu Sun  1
Affiliations

Attentional Generative Multimodal Network for Neonatal Postoperative Pain Estimation

Md Sirajus Salekin et al. Med Image Comput Comput Assist Interv. 2022 Sep.

Abstract

Artificial Intelligence (AI)-based methods allow for automatic assessment of pain intensity based on continuous monitoring and processing of subtle changes in sensory signals, including facial expression, body movements, and crying frequency. Currently, there is a large and growing need for expanding current AI-based approaches to the assessment of postoperative pain in the neonatal intensive care unit (NICU). In contrast to acute procedural pain in the clinic, the NICU has neonates emerging from postoperative sedation, usually intubated, and with variable energy reserves for manifesting forceful pain responses. Here, we present a novel multi-modal approach designed, developed, and validated for assessment of neonatal postoperative pain in the challenging NICU setting. Our approach includes a robust network capable of efficient reconstruction of missing modalities (e.g., obscured facial expression due to intubation) using an unsupervised spatio-temporal feature learning with a generative model for learning the joint features. Our approach generates the final pain score along with the intensity using an attentional cross-modal feature fusion. Using experimental dataset from postoperative neonates in the NICU, our pain assessment approach achieves superior performance (AUC 0.906, accuracy 0.820) as compared to the state-of-the-art approaches.

Keywords: Generative model; Multimodal learning; NICU; Neonatal pain; Postoperative pain.

PubMed Disclaimer

Figures

Fig. 1.
Fig. 1.
Proposed approach for postoperative pain assessment (best viewed in color).
Fig. 2.
Fig. 2.
t-SNE projection of spatio-temporal features using perplexity of 40. Each column represents face, body, and audio (left to right). Top and bottom rows are the baseline and proposed features, respectively (best viewed in color).
See this image and copyright information in PMC

References

    1. Bowman S, Vilnis L, Vinyals O, Dai A, Jozefowicz R, Bengio S: Generating sentences from a continuous space. In: Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pp. 10–21 (2016)
    1. Cao Q, Shen L, Xie W, Parkhi OM, Zisserman A: VGGFace2: a dataset for recognising faces across pose and age. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp. 67–74. IEEE; (2018)
    1. Cao Y, Fleet DJ: Generalized product of experts for automatic and principled fusion of Gaussian process predictions. arXiv preprint arXiv:1410.7827 (2014)
    1. Choi JH, Lee JS: EmbraceNet: a robust deep learning architecture for multi-modal classification. Inf. Fusion 51, 259–270 (2019)
    1. Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L: ImageNet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. IEEE; (2009)

LinkOut - more resources