Deep Learning-Based Objective and Reproducible Osteosarcoma Chemotherapy Response Assessment and Outcome Prediction

Abstract

Osteosarcoma is the most common primary bone cancer, whose standard treatment includes pre-operative chemotherapy followed by resection. Chemotherapy response is used for prognosis and management of patients. Necrosis is routinely assessed after chemotherapy from histology slides on resection specimens, where necrosis ratio is defined as the ratio of necrotic tumor/overall tumor. Patients with necrosis ratio ≥90% are known to have a better outcome. Manual microscopic review of necrosis ratio from multiple glass slides is semiquantitative and can have intraobserver and interobserver variability. In this study, an objective and reproducible deep learning-based approach was proposed to estimate necrosis ratio with outcome prediction from scanned hematoxylin and eosin whole slide images (WSIs). To conduct the study, 103 osteosarcoma cases with 3134 WSIs were collected. Deep Multi-Magnification Network was trained to segment multiple tissue subtypes, including viable tumor and necrotic tumor at a pixel level and to calculate case-level necrosis ratio from multiple WSIs. Necrosis ratio estimated by the segmentation model highly correlates with necrosis ratio from pathology reports manually assessed by experts. Furthermore, patients were successfully stratified to predict overall survival with P = 2.4 × 10^-6 and progression-free survival with P = 0.016. This study indicates that deep learning can support pathologists as an objective tool to analyze osteosarcoma from histology for assessing treatment response and predicting patient outcome.

Graphical abstract

Figure 1

Block diagram of the proposed method. Top: Currently, an osteosarcoma case with multiple slides is assessed via a microscope to estimate necrosis ratio and to predict outcome. Bottom: Deep learning–based segmentation by Deep Multi-Magnification Network was used to segment multiple tissue subtypes, to count the number of pixels for viable tumor (VT) and necrotic tumor (NT), to estimate necrosis ratio, and to predict outcome.

Figure 2

Osteosarcoma data set containing 103 cases with 3134 whole slide images (WSIs). Fifteen cases were used to train the segmentation model, and the other 88 cases were used to test the model. More specifically, 80 cases were used to evaluate necrosis ratio assessment, 75 cases were used to predict overall survival (OS), and 64 cases were used to predict progression-free survival (PFS). MSKCC, Memorial Sloan Kettering Cancer Center.

Figure 3

Multiclass segmentation of two osteosarcoma whole slide images. Whole slide images (A and C) and their segmentation predictions (B and D). Viable tumor is segmented in red, necrosis/nonviable bone is segmented in blue, necrosis/fibrosis without bone is segmented in yellow, normal bone is segmented in green, normal tissue is segmented in orange, normal cartilage is segmented in brown, and blank is segmented in gray. Scale bar = 5 mm (A and C).

Figure 4

Segmentation of viable tumor (A and B), necrosis/nonviable bone (C and D), and necrosis/fibrosis without bone (E and F). Viable tumor is segmented in red, necrosis/nonviable bone is segmented in blue, and necrosis/fibrosis without bone is segmented in yellow. Scale bars: 100 μm (A); 200 μm (C and E).

Figure 5

Outcome prediction. A: Patient stratification based on overall survival (OS) outcome at the conventional 90% cutoff threshold from manually assessed pathology reports, achieving P = 0.045. B: Patient stratification based on OS outcome at the same 90% cutoff threshold from our deep learning model, achieving P = 0.0031. The deep learning model performed a better stratification than manual assessment of glass slides. C: Patient stratification based on OS outcome at the 80% cutoff threshold from our deep learning model, achieving P = 2.4 × 10⁻⁶. The cutoff threshold for our deep learning model and our data set can be tuned to have better stratification because of its objective and reproducible manner. D: Patient stratification based on progression-free survival outcome at the 60% cutoff threshold from our deep learning model, achieving P = 0.016.