Image segmentation of treated and untreated tumor spheroids by fully convolutional networks

Abstract

Background: Multicellular tumor spheroids (MCTS) are advanced cell culture systems for assessing the impact of combinatorial radio(chemo)therapy as they exhibit therapeutically relevant in vivo-like characteristics from 3-dimensional cell-cell and cell-matrix interactions to radial pathophysiological gradients. State-of-the-art assays quantify long-term curative endpoints based on collected brightfield image time series from large treated spheroid populations per irradiation dose and treatment arm. This analyses require laborious spheroid segmentation of up to 100,000 images per treatment arm to extract relevant structural information from the images (e.g., diameter, area, volume, and circularity). While several image analysis algorithms are available for spheroid segmentation, they all focus on compact MCTS with a clearly distinguishable outer rim throughout growth. However, they often fail for the common case of treated MCTS, which may partly be detached and destroyed and are usually obscured by dead cell debris.

Results: To address these issues, we successfully train 2 fully convolutional networks, UNet and HRNet, and optimize their hyperparameters to develop an automatic segmentation for both untreated and treated MCTS. We extensively test the automatic segmentation on larger, independent datasets and observe high accuracy for most images with Jaccard indices around 90%. For cases with lower accuracy, we demonstrate that the deviation is comparable to the interobserver variability. We also test against previously published datasets and spheroid segmentations.

Conclusions: The developed automatic segmentation can not only be used directly but also integrated into existing spheroid analysis pipelines and tools. This facilitates the analysis of 3-dimensional spheroid assay experiments and contributes to the reproducibility and standardization of this preclinical in vitro model.

Keywords: 3D cancer models; brightfield microscopy; cancer therapy; deep learning; fully convolutional networks; high-content screening; interobserver variability; organoids; segmentation; spheroids.

Figure 1:

Representative examples of images for automatic segmentation with the optimized U-Net (blue) compared to the manual segmentation (green) for 3D tumor spheroids after treatment. The overlap with the manual segmentation is excellent for standard size and larger spheroids obscured by cell debris (top row) and sufficient for small, heavily obscured spheroids (bottom row). Displayed are (A, D) the original images, which are also the input for the U-Net; (B, D) magnified image details around the spheroids, as indicated by the white box in (A, D) for visibility; and (C, E) corresponding contours from the segmentations. The metrics for evaluation of the automatic segmentation are (top row) Jaccard distance (JCD) , relative diameter deviation (RDD) , and relative circularity deviation (RCD) ; bottom row: , , and ; see text for details.

Figure 2:

Exemplary evaluation of the automatic segmentation (blue) with respect to the manual one (green) for (A) standard case and (B, C) rare artifacts; see text for details. (A) Correctly segmented spheroid: no contribution to invalid spheroid fraction (ISF) or ambiguous spheroid fraction (ASF); JCD, RDD, and RCD are well defined. (B) Excessive spheroids are detected beyond the actual spheroid: no contribution to ISF, one count added to ASF; JCD, RDD, and RCD are well defined and computed for the larger, upper spheroid. (C) No overlap between automatic and manual segmentation: 1 count added to ISF, no contribution to ASF; JCD, RDD, and RCD are set to 1.

Figure 3:

Postprocessing steps for the output of the FCN model to transform the probability heatmap to the spheroid contour. (A) Probability heatmap as output of the FCN model. Each pixel takes probability values between 0 (black) and 1 (white) predicting the target (spheroid). Note that due to the steep gradient, the gradual change from black to white is hard to see. (B) By using a threshold of 0.5, the pixels are classified into outside spheroid (black) and inside spheroid (white). (C) Contour of the spheroid border extracted as a polygonal chain (blue line displayed on original image).

Figure 4:

Validation of the automatic segmentation with the optimized U-Net on larger, independent datasets shows high accuracy for most cases. (A) JCD and (B) average radial error over diameter of the manually segmented (target) spheroid for 6,574 images of FaDu (blue triangles) and SAS (green stars) spheroids treated with different combinations and doses of X-ray irradiation and hyperthermia [16]. Manual segmentation is performed by a second biological expert (human H2, blue triangles and green stars) independently from the manual segmentation (human H1, red dots) of the training, validation, and test datasets. (Results for 104 images of the test dataset are displayed as red dots for comparison.) Note that the segmentation is developed only based on images of FaDu spheroids. Most deviations are small (, m, red horizontal lines as guide to the eye), average (median) values are , m for the whole dataset, and , m for spheroids larger m (red vertical lines as guide to the eye) than the initial, standard size of spheroids. Larger imprecisions for smaller spheroids are due to biologically difficult, unclear, or ambiguous cases; see text.

Figure 5:

For treated spheroids smaller than the initial, standard size before treatment (m), deviations from the manual segmentation are not higher than variations across segmentations by different humans, suggesting that the segmentation of images with small spheroids surrounded by heavy debris is often difficult or ambiguous: compared are segmentations from the optimized U-Net and 4 independent human experts (H2–H5) for the same 101 images, which are randomly selected from the pool of small spheroids of the extended Hold-out test dataset; see Fig. 4. The Friedman test score of all JCDs 217.2 () indicates significant differences among the pairwise segmentation deviations. The order according to the average JCD is (from low to high values) H5U-Net, H3U-Net, H3H5, H2H5 H4H5, H4U-Net, H2U-Net, H3H4, H2H4, H2H3, where the indicates a significant () difference between the sets of JCDs according to a Dunn–Bonferroni pairwise post hoc test.