Length-scale study in deep learning prediction for non-small cell lung cancer brain metastasis

Abstract

Deep learning-assisted digital pathology has demonstrated the potential to profoundly impact clinical practice, even surpassing human pathologists in performance. However, as deep neural network (DNN) architectures grow in size and complexity, their explainability decreases, posing challenges in interpreting pathology features for broader clinical insights into physiological diseases. To better assess the interpretability of digital microscopic images and guide future microscopic system design, we developed a novel method to study the predictive feature length-scale that underpins a DNN's predictive power. We applied this method to analyze a DNN's capability in predicting brain metastasis from early-stage non-small-cell lung cancer biopsy slides. This study quantifies DNN's attention for brain metastasis prediction, targeting features at both the cellular scale and tissue scale in H&E-stained histological whole slide images. At the cellular scale, the predictive power of DNNs progressively increases with higher resolution and significantly decreases when the resolvable feature length exceeds 5 microns. Additionally, DNN uses more macro-scale features associated with tissue architecture and is optimized when assessing visual fields greater than 41 microns. Our study computes the length-scale requirements for optimal DNN learning on digital whole-slide microscopic images, holding the promise to guide future optical microscope designs in pathology applications and facilitating downstream deep learning analysis.

Fig. 1

(a) Preprocessing pipeline of H&E-stained whole slide images. The whole slide image is manually annotated by a human expert. The annotation mask is processed with thresholding to get rid of the background region. A thousand non-overlapping image tiles are randomly selected from the masked region. (b) One experiment of training-testing split in deep learning pipeline. Acc. stands for “accuracy”.

Fig. 2

A schematic diagram of the length-scale processing pipeline for (a) RFL and (b) MFL. The red arrows indicate the training flow, and the green arrows denote the testing flow. The black dashed line is associated with a specific length-scale data input. The different levels of the length-scales are obtained by information attrition from downsampling or cropping the input images. Acc. stands for accuracy.The experimentally derived accuracy-versus-RFL and -MFL curves are shown in detail in Fig. 3.

Fig. 3

Length-scale study curves for different (a) RFLs and (b) MFLs. The dotted arrow at the top indicates the length-scale from small size to large size. The black solid lines are the piecewise linear fittings to the average values of the three experiments. The “+” marker indicates the result from the original study.

Fig. 4

Visualization of (a) original image patches, (b) images at $\text{RFL}=5.1\mathrm{\mu }\text{m}$, and (c) the subtraction of (a) and (b). Note that while color information is preserved, the architectural and cellular information is largely lost at RFL 5.1 microns, as demonstrated by the preservation of these features in the subtraction images.

Fig. 5

(a) Whole slide image of a Met+ case. (b) Processed annotation mask. (c) Annotated H&E section. Slope maps for individual tiles examining the role of (d) RFLs and (e) MFLs, where orange indicates tiles where the DNN prediction for that tile was accurate, while blue indicating tiles were incorrect prediction. Zoom-in images in (d) and (e) the orange and blue boxes showing the same tissue area for RFL (d) and MFL (e).

Fig. 6

Representative histologic images of tiles where (a) the DNN made an accurate prediction, while (b) the DNN made no prediction or an inaccurate prediction. In (a), note areas of tumor (black arrows) and tumor microenvironment including immune cells and desmoplastic stroma (blue arrows). In (b) note that there are reactive pneumocytes, pulmonary macrophages and alveolar wall (green arrows), and fibrosis (red arrows). We refer the readers to the Histological analysis of DNN’s attention subsection in the paper for a detailed description of relevant observations.

Fig. 7

(a) Whole slide image of a Met- case. (b) Processed annotation mask. (c) Annotated H&E section. Slope maps for individual tiles examining the role of (d) RFLs and (e) MFLs, where orange indicates tiles where the DNN prediction for that tile was accurate, while blue indicating tiles were incorrect prediction. Zoom-in images in (d) and (e) the orange and blue boxes showing the same tissue area for RFL (d) and MFL (e).

Fig. 8

Representative histologic images of tiles where (a) the DNN made an accurate prediction and (b) made an inaccurate prediction. In (a), note areas of tumor (black arrows) and tumor microenvironment including immune cells and desmoplastic stroma (blue arrows). In (b) note that there is fibrosis and deposits of anthracitic pigment (red arrows) but no tumor cells. We refer the readers to the Histological analysis of DNN’s attention subsection in the paper for a detailed description of relevant observations.

Fig. 9

(a–d) Concatenation of 5-by-5 image tiles from Met+ case in Fig. 5. Areas a1 and b1 are the histology from a concatenated area where the DNN prediction was accurate and sensitive to changes in RFL (a2,b2) and MFL (a3,b3). Areas c1 and d1 are the histology from a concatenated area where the DNN prediction was low or inaccurate and insensitive to changes in RFL (c2,d2) and MFL (c3,d3). The color scheme is similar to Fig. 6. In a1 and b1, note areas of tumor cells (black arrows) and tumor microenvironment including immune cells and desmoplastic stroma (blue arrows). In c1 and d1, note that the tissue is largely devoid of tumor cells.

Fig. 10

(a–d) Concatenation of 5-by-5 image tiles from Met- case in Fig. 7. Areas a1 and b1 are the histology from a concatenated area where the DNN prediction was accurate and sensitive to changes in RFL (a2,b2) and MFL (a3,b3). Areas c1 and d1 are the histology from a concatenated area where the DNN prediction was low or inaccurate and insensitive to changes in RFL (c2,d2) and MFL (c3,d3). The color scheme is similar to Fig. 8. In a1 and b1, note areas of tumor cells (black arrows) and tumor microenvironment including immune cells and desmoplastic stroma (blue arrows). In c1 and d1, note that the tissue is largely devoid of tumor cells.