Systematic evaluation of computational methods for cell segmentation

Abstract

Cell segmentation plays a crucial role in elucidating cell structure and function, understanding disease mechanisms, and aiding pathological diagnosis. Current surveys primarily categorize methods by their technical evolution stages, which may not fully capture the paradigm shift brought by deep learning. Moreover, their evaluation scope is largely confined to image-only approaches, overlooking the significant potential of multimodal data in enhancing cell/nucleus segmentation performance. Therefore, we propose a dual-dimensional classification framework for deep learning methods. It categorizes such methods into two types: task-oriented (e.g. semantic or instance segmentation) and data-oriented (e.g. single or multimodal inputs). Based on this, we systematically classify and summarize methods across various segmentation tasks and imaging modalities. We also develop a benchmark test that covers both single-modal and multimodal methods. This test uses five diverse datasets, among which four are from conventional microscopy and one integrates sequencing with image data. Furthermore, it assesses seven algorithms based on three dimensions: effectiveness, robustness, and efficiency. Key findings indicate that deep learning models generally outperform traditional algorithms, with their advantage becoming more pronounced when image data is integrated with sequencing information.

Keywords: cell segmentation; deep learning; image processing; nuclei segmentation; spatial transcriptome.

Figure 1

Quantitative comparison of six nuclei segmentation algorithms Cellpose, DeepCell, Mesmer, Splinedist, StarDist and watershed evaluated on four image-only datasets BBBC041Seg, BSST265, MoNuSeg and PanNuKe. (A–D) present the results for precision, recall, F1-score, and dice, respectively. Each boxplot illustrates the distribution of metric values for a given algorithm and dataset, while each point represents the metric value for one image of that dataset.

Figure 2

Evaluates the stability of six nuclei segmentation algorithms across four image-only datasets, which include BBBC041Seg, BSST265, MoNuSeg, and PanNuKe, by measuring the standard deviation of their performance metrics. (A–D) present the standard deviations for precision, recall, F1-score, and dice, respectively. Lower values indicate greater algorithmic stability on the corresponding dataset and metric.

Figure 3

Evaluates the stability of six nuclei segmentation algorithms across four image-only datasets, which include BBBC041Seg, BSST265, MoNuSeg, and PanNuKe, using confidence intervals of performance metrics. (A–D) present the confidence interval for precision, recall, F1-score, and dice, respectively. Narrower confidence intervals indicate better algorithmic performance for the corresponding dataset and metric.

Figure 4

Results of nuclei segmentation methods on image-only datasets. The first column displays the original images, and the second column shows the corresponding segmentation masks. Columns three through the last illustrate the segmentation results obtained using different nuclei segmentation methods. The red rectangle highlights region of interest, enabling detailed evaluation of each algorithm’s performance in nuclei boundary segmentation accuracy and instance segmentation.

Figure 5

Performance evaluation of seven nuclei segmentation methods on the multi-modal dataset MSBDS integrating image and sequencing data. (A) Shows the segmentation effectiveness. (B) Presents the stability measured by standard deviation. (C) Displays the confidence intervals across all evaluation metrics.

Figure 6

Results of nuclei segmentation methods on the dataset MSBDS integrating image and sequencing data. The first two columns display original images and corresponding ground truth masks, while subsequent columns present results from different segmentation algorithms. A red rectangle highlights a region of interest in the first row to facilitate detailed comparison of boundary accuracy and instance segmentation performance.

Figure 7

Performance visualization results of seven representative nuclei segmentation methods assessed on effectiveness (red), robustness (blue), and efficiency (green). For all colored dots, a darker shade denotes better performance. A cross indicates the method is not applicable (NA), and a red-bordered circle signifies a runtime over 100 s.

Figure 8

A Sankey diagram mapping the seven evaluated tools onto the proposed ‘task-target-modality-method’ taxonomy. The flow illustrates the methodological basis (from classical to deep learning) and the operational scope (modality and target) of each algorithm.

Figure 9

A decision-tree recommendation guide for users and developers. The flowchart assists researchers in selecting the optimal segmentation tool based on data modality (e.g. spatial omics versus standard imaging), tissue complexity, and specific task requirements (e.g. speed versus accuracy trade-offs).