Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2025 Feb 6;26(1):42.
doi: 10.1186/s12859-025-06066-8.

Instance-level semantic segmentation of nuclei based on multimodal structure encoding

Bo Guan  1 , Guangdi Chu  2 , Ziying Wang  3 , Jianmin Li  4 , Bo Yi  5
Affiliations

Instance-level semantic segmentation of nuclei based on multimodal structure encoding

Bo Guan et al. BMC Bioinformatics. .

Abstract

Background: Accurate segmentation and classification of cell nuclei are crucial for histopathological image analysis. However, existing deep neural network-based methods often struggle to capture complex morphological features and global spatial distributions of cell nuclei due to their reliance on local receptive fields.

Methods: This study proposes a graph neural structure encoding framework based on a vision-language model. The framework incorporates: (1) A multi-scale feature fusion and knowledge distillation module utilizing the Contrastive Language-Image Pre-training (CLIP) model's image encoder; (2) A method to transform morphological features of cells into textual descriptions for semantic representation; and (3) A graph neural network approach to learn spatial relationships and contextual information between cell nuclei.

Results: Experimental results demonstrate that the proposed method significantly improves the accuracy of cell nucleus segmentation and classification compared to existing approaches. The framework effectively captures complex nuclear structures and global distribution features, leading to enhanced performance in histopathological image analysis.

Conclusions: By deeply mining the morphological features of cell nuclei and their spatial topological relationships, our graph neural structure encoding framework achieves high-precision nuclear segmentation and classification. This approach shows significant potential for enhancing histopathological image analysis, potentially leading to more accurate diagnoses and improved understanding of cellular structures in pathological tissues.

Keywords: Cell nucleus segmentation; Graph neural networks; Histopathological image; Multimodal fusion.

PubMed Disclaimer

Conflict of interest statement

Declarations. Ethics approval and consent to participate: Not applicable. Consent for publication: Not applicable. Competing interests: Authors declare no conflict of interest.

Figures

Fig. 1
Fig. 1
A framework for cell nucleus segmentation and classification based on multi-modal structure encoding. a Network architecture for nucleus instance segmentation based on multimodal feature fusion and knowledge distillation, b graph neural network-based architecture for nucleus classification
Fig. 2
Fig. 2
Attention mechanism module based on spatial pyramid pooling for feature encoding
Fig. 3
Fig. 3
Multi-stage Feature Fusion Decoder Based on ConvNeXt Module
Fig. 4
Fig. 4
Nucleus Graph Construction. This includes feature maps extracted by the nuclear instance semantic segmentation branch and the inferred nuclear masks, local and spatial positional information of the nuclei, and high-level representations of the morphological information of the nuclei
Fig. 5
Fig. 5
Samples of cropped nucleus instance segmentation public dataset
Fig. 6
Fig. 6
Samples of cropped nucleus classification dataset
Fig. 7
Fig. 7
Visualization of nuclear instance segmentation results
Fig. 8
Fig. 8
Visualization of nuclear classification results
See this image and copyright information in PMC

References

    1. Basu A, Senapati P, Deb M, Rai R, Dhal KG. A survey on recent trends in deep learning for nucleus segmentation from histopathology images. Evol Syst. 2024;15(1):203–48. - PMC - PubMed
    1. Morales S, Engan K, Naranjo V. Artificial intelligence in computational pathology-challenges and future directions. Digital Signal Process. 2021;119: 103196.
    1. Huang Y, Yang X, Liu L, Zhou H, Chang A, Zhou X, Chen R, Yu J, Chen J, Chen C. Segment anything model for medical images? Med Image Anal. 2024;92: 103061. - PubMed
    1. Hatamizadeh A, Nath V, Tang Y, Yang D, Roth HR, Xu D. Swin unetr: swin transformers for semantic segmentation of brain tumors in mri images. In: International MICCAI brainlesion workshop, 2021;272–84.
    1. Zhou Y, Graham S, Alemi Koohbanani N, Shaban M, Heng P-A, Rajpoot N. Cgc-net: cell graph convolutional network for grading of colorectal cancer histology images. In: Proceedings of the IEEE/CVF international conference on computer vision workshops, 2019;0–0

LinkOut - more resources