A fully automatic framework for evaluating cosmetic results of breast conserving therapy

Abstract

The breast cosmetic outcome after breast conserving therapy is essential for evaluating breast treatment and determining patient's remedy selection. This prompts the need of objective and efficient methods for breast cosmesis evaluations. However, current evaluation methods rely on ratings from a small group of physicians or semi-automated pipelines, making the processes time-consuming and their results inconsistent. To solve the problem, in this study, we proposed: 1. a fully-automatic Machine Learning Breast Cosmetic evaluation algorithm leveraging the state-of-the-art Deep Learning algorithms for breast detection and contour annotation, 2. a novel set of Breast Cosmesis features, 3. a new Breast Cosmetic dataset consisting 3k+ images from three clinical trials with human annotations on both breast components and their cosmesis scores. We show our fully-automatic framework can achieve comparable performance to state-of-the-art without the need of human inputs, leading to a more objective, low-cost and scalable solution for breast cosmetic evaluation in breast cancer treatment.

Keywords: Breast Cosmesis scores; Breast cancer; Breast conserving therapy; Breast detection; Machine learning; Predictive model.

Fig. 10.

Confusion matrices of our framework with 3 machine learning algorithms estimating 6 scores (1 global score and 5 categorical scores).

Fig. 1.

Overview of the proposed Breast Cosmesis evaluation system.

Fig. 2.

Score distributions for (a) individual categories and (b) global score.

Fig. 3.

Landmark annotations used in our study. There are 30 landmarks on each breast, 12 landmarks on each areola, and 6 landmarks on each nipple.

Fig. 4.

Image preprocessing to mask out tattoos or markers on breast skin. (a) Segmentation from landmarks. (b) A channel after thresholding. (c) A channel after correction. (d) Resulting RGB image after this preprocessing.

Fig. 5.

Here the areola is highlighted using the black contour. (a) This diagram illustrates the definitions of the width and height of areola, denoted as ω_a and h_a, respectively, when the areola exists. The black cross denotes the center of areola. If the areola does not exist but the nipple exists, compute these values from the nipple instead. (b) For cases where both the areola and nipple exist, this diagram illustrates the definitions of the nipple location relative to the areola, denoted as D_ut, D_ub, D_hl, and D_hr. The yellow cross denotes the center of nipple. If only the areola exists, then these relative locations are set to be semi-vertical and semi-horizontal axis length of the existing areola. If only the nipple exists, then these relative locations are set to the semi-vertical and semi-horizontal axis length of the existing nipple. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

Fig. 6.

Example of breast mean shape warping and coordinate systems. To calculate EMD-XYLAB features, each breast of the patient needs to be warped to the mean breast shape of the same side. (a) A patient’s image before warping. (b) The same image after warping and masking. (c) mean breast shape used for warping and left (right) coordinate system for pixel location (x, y) used in the EMD-XYLAB features.

Fig. 7.

Color distribution differences between treated and untreated breast of a patient. The left breast of this patient is treated and the right is untreated. (a) Patient’s image in RGB color space. (b) 2D histogram in AB space for right breast. (c) 2D histogram in AB space for left breast. Comparing to the untreated breast, the pixels of treated breast are more linearly correlated between A and B channels. The color contour shows the breast pixels with corresponding color range in the histogram. Regions smaller than contour width are discarded for visual clarity. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

Fig. 8.

Quantitative results for landmarks detection using within landmark evaluation. (a) Visualization of the mean and standard error of the normalized Euclidean distance for each landmark across images. The radius of each circle represents the average Euclidean distance (across images) between the ground truth and the detected breast landmark after equal arc sampling. The thickness of each circle equals 2 times the standard error. The half-inter-pit distance is defined as half of the distance between the starting landmarks of left breast and right breast, and both are defined at the armpit of each side. (b) within landmarks result of detection error for left breast, (c) right breast, (d) areolas and (e) nipples.

Fig. 9.

Bar plot of full pipeline performance with 10-fold Cross Validation, for each individual categorical score and global score. Bar heights represent average accuracy with error bar denoting standard error.