Automatic segmentation and supervised learning-based selection of nuclei in cancer tissue images

Abstract

Analysis of preferential localization of certain genes within the cell nuclei is emerging as a new technique for the diagnosis of breast cancer. Quantitation requires accurate segmentation of 100-200 cell nuclei in each tissue section to draw a statistically significant result. Thus, for large-scale analysis, manual processing is too time consuming and subjective. Fortuitously, acquired images generally contain many more nuclei than are needed for analysis. Therefore, we developed an integrated workflow that selects, following automatic segmentation, a subpopulation of accurately delineated nuclei for positioning of fluorescence in situ hybridization-labeled genes of interest. Segmentation was performed by a multistage watershed-based algorithm and screening by an artificial neural network-based pattern recognition engine. The performance of the workflow was quantified in terms of the fraction of automatically selected nuclei that were visually confirmed as well segmented and by the boundary accuracy of the well-segmented nuclei relative to a 2D dynamic programming-based reference segmentation method. Application of the method was demonstrated for discriminating normal and cancerous breast tissue sections based on the differential positioning of the HES5 gene. Automatic results agreed with manual analysis in 11 out of 14 cancers, all four normal cases, and all five noncancerous breast disease cases, thus showing the accuracy and robustness of the proposed approach.

Figure 1.

Flow diagram showing the computational framework.

Figure 2.

Representative image and the corresponding outputs at different segmentation steps. (a) Original DAPI channel nuclei image. (b) Preprocessed nuclei channel. (c) Seeded intensity watershed output on image foreground. (d) GDT output. (e) Merged output of intensity and GDT watershed. (f) Final segmentation output after the cluster-breaking watershed and tree-based merging.

Figure 3.

Multistage watershed segmentation algorithm.

Figure 4.

(a) Process of building up the merge tree for a node. (b) Merged fragment and optimal ellipse fit. (c) Nonoverlapping (XOR) area.

Figure 5.

(a) PRE for identifying accurately segmented nuclei. (b) Details of the feature processing.

Figure 6.

(a) Example nucleus for boundary accuracy assessment. (b) Example nucleus with control. (C) (Green) and test (T) (Red) segmentation. (c) Overlap area (purple) used to measure AS. (d) Distance transform-based boundary accuracy calculation. Distance transform was calculated with respect to control segmentation. (e) Normalized EDT calculation on control segmentation mask used to measure difference in relative distance measure. Control segmentation boundary is shown in white. (f) Difference in normalized EDT-based relative distance measure.

Figure 7.

PRE precision-recall plot for 1,620 configurations and the best configuration (closest to (1,1)). Configurations with high precision and low recall are shown in the red box.

Figure 8.

(a) Flow diagram showing steps for spatial gene localization analysis. (b) Original image of segmented nucleus showing red and green FISH spots marked by arrows. (c) Nucleus ROI showing segmented FISH spots. (d) EDT nucleus ROI showing normalized distance transform metric for each FISH spot. (e) Histogram of FISH signal positions binned by normalized EDT values for aggregate cancers, aggregate normals, and cancer samples C1, C10, and C12. (f) Cumulative distribution of FISH spots against normalized EDT values for aggregate cancers, aggregate normals, and cancer samples C1, C10, and C12.