GNNs surpass transformers in tumor medical image segmentation

Abstract

To assess the suitability of Transformer-based architectures for medical image segmentation and investigate the potential advantages of Graph Neural Networks (GNNs) in this domain. We analyze the limitations of the Transformer, which models medical images as sequences of image patches, limiting its flexibility in capturing complex and irregular tumor structures. To address it, we propose U-GNN, a pure GNN-based U-shaped architecture designed for medical image segmentation. U-GNN retains the U-Net-inspired inductive bias while leveraging GNNs' topological modeling capabilities. The architecture consists of Vision GNN blocks stacked into a U-shaped structure. Additionally, we introduce the concept of multi-order similarity and propose a zero-computation-cost approach to incorporate higher-order similarity in graph construction. Each Vision GNN block segments the image into patch nodes, constructs multi-order similarity graphs, and aggregates node features via multi-order node information aggregation. Experimental evaluations on multi-organ and cardiac segmentation datasets demonstrate that U-GNN significantly outperforms existing CNN- and Transformer-based models. U-GNN achieves a 6% improvement in Dice Similarity Coefficient (DSC) and an 18% reduction in Hausdorff Distance (HD) compared to state-of-the-art methods. The source code will be released upon paper acceptance.

Keywords: CNN; GNN; Medical image segmentation; Transformer.

Fig. 1

The overall architecture of U-GNN encompasses components such as an encoder, a bottleneck layer, a decoder, and skip connections. Among them, the construction of the encoder, the bottleneck layer, and the decoder is carried out based on the Vision GNN block.

Fig. 2

Vision GNN block.

Fig. 3

Illustration of first-order and multi-order similarity: First-order similarity captures node connections based on shallow features like color and texture, while multi-order similarity also considers the similarity of neighboring nodes, providing stronger semantic relevance.

Fig. 4

The segmentation results of different methods on the Synapse.

Fig. 5

U-GNN is capable of providing precise graph-level information for medical images, ensuring that image patches corresponding to tumor regions are closely connected and interact with each other.