A method for the evaluation of thousands of automated 3D stem cell segmentations

Abstract

There is no segmentation method that performs perfectly with any dataset in comparison to human segmentation. Evaluation procedures for segmentation algorithms become critical for their selection. The problems associated with segmentation performance evaluations and visual verification of segmentation results are exaggerated when dealing with thousands of three-dimensional (3D) image volumes because of the amount of computation and manual inputs needed. We address the problem of evaluating 3D segmentation performance when segmentation is applied to thousands of confocal microscopy images (z-stacks). Our approach is to incorporate experimental imaging and geometrical criteria, and map them into computationally efficient segmentation algorithms that can be applied to a very large number of z-stacks. This is an alternative approach to considering existing segmentation methods and evaluating most state-of-the-art algorithms. We designed a methodology for 3D segmentation performance characterization that consists of design, evaluation and verification steps. The characterization integrates manual inputs from projected surrogate 'ground truth' of statistically representative samples and from visual inspection into the evaluation. The novelty of the methodology lies in (1) designing candidate segmentation algorithms by mapping imaging and geometrical criteria into algorithmic steps, and constructing plausible segmentation algorithms with respect to the order of algorithmic steps and their parameters, (2) evaluating segmentation accuracy using samples drawn from probability distribution estimates of candidate segmentations and (3) minimizing human labour needed to create surrogate 'truth' by approximating z-stack segmentations with 2D contours from three orthogonal z-stack projections and by developing visual verification tools. We demonstrate the methodology by applying it to a dataset of 1253 mesenchymal stem cells. The cells reside on 10 different types of biomaterial scaffolds, and are stained for actin and nucleus yielding 128 460 image frames (on average, 125 cells/scaffold × 10 scaffold types × 2 stains × 51 frames/cell). After constructing and evaluating six candidates of 3D segmentation algorithms, the most accurate 3D segmentation algorithm achieved an average precision of 0.82 and an accuracy of 0.84 as measured by the Dice similarity index where values greater than 0.7 indicate a good spatial overlap. A probability of segmentation success was 0.85 based on visual verification, and a computation time was 42.3 h to process all z-stacks. While the most accurate segmentation technique was 4.2 times slower than the second most accurate algorithm, it consumed on average 9.65 times less memory per z-stack segmentation.

Keywords: 3D segmentation; confocal imaging; sampling; segmentation evaluation; stem cells; visual verification.

Figure 1

Shape variations of 2D middle cross sections of z-stacks representing cells on spun coat scaffold. The actin stained images are displayed by showing all values above zero intensity.

Figure 2

Statistics about the number of z-frames per z-stack over 10 scaffold types

Figure 3

An overview of segmentation accuracy estimation. “Alg.” stands for algorithm.

Figure 4

Illustration of the sampling methodology applied to 114 z-stacks from Collagen Fibrils scaffold collection. The two red dots in the lower right panel correspond to the two z-stacks selected for manual segmentation.

Figure 5

Two examples of three orthogonal max intensity projections of the min (top) and max (bottom) scores for Microfiber scaffold. Left column shows the projections of the original z-stack. Right column shows manually segmented three projections of the same z-stack. The ZX and YZ projections have been scaled in the Z direction.

Figure 6

Annotations, three orthogonal projections of a z-stack with actin channel and PPS scaffold, and the segmentation results obtained by executing the top two algorithmic sequences. The cells of interest are denoted by a red box. The z-stack voxels here were projected as cubic voxels without being scaled in the z-dimension. The size of the XY projections is 246 μm × 246 μm.

Figure 7

A 3D web-based visualization of 100+ z-stacks from the same collagen scaffold type. The insets illustrate the interactivity during visual inspection. The blue ball is used as a spatial scale.

Figure 8

Average FRG voxel count per scaffold after executing each step of the segmentation sequence A12: T1➔E➔F➔L➔M2➔L. The legend denotes the scaffold types.

Figure 9

Repeatability (precision) of manual segmentations estimated over three cell samples (S1, S2, S3) times 3 orthogonal projection images (XY, XZ, YZ) by four human subjects.

Figure 10

Top: Segmentation accuracy of six segmentation algorithms measured by average of the Dice index over 20 or 30 manually segmented cells. Bottom: Segmentation accuracy estimations per scaffold type established based on 30 cells that were manually segmented.

Figure 11

Top: Execution time efficiency for the top two performing sequences A12: T1➔E➔F➔L➔M2➔L and A22: T2 E➔F➔L➔M2➔L decomposed into O1, O2 and T➔E➔F➔L➔M2➔L computation times. Bottom: Memory benchmarks of two threshold optimization computations using O1 ~ Minimum error thresholding and O2 ~ Topological stable state thresholding approaches.