Network Visualization and Pyramidal Feature Comparison for Ablative Treatability Classification Using Digitized Cervix Images

Abstract

Uterine cervical cancer is a leading cause of women's mortality worldwide. Cervical tissue ablation is an effective surgical excision of high grade lesions that are determined to be precancerous. Our prior work on the Automated Visual Examination (AVE) method demonstrated a highly effective technique to analyze digital images of the cervix for identifying precancer. Next step would be to determine if she is treatable using ablation. However, not all women are eligible for the therapy due to cervical characteristics. We present a machine learning algorithm that uses a deep learning object detection architecture to determine if a cervix is eligible for ablative treatment based on visual characteristics presented in the image. The algorithm builds on the well-known RetinaNet architecture to derive a simpler and novel architecture in which the last convolutional layer is constructed by upsampling and concatenating specific RetinaNet pretrained layers, followed by an output module consisting of a Global Average Pooling (GAP) layer and a fully connected layer. To explain the recommendation of the deep learning algorithm and determine if it is consistent with lesion presentation on the cervical anatomy, we visualize classification results using two techniques: our (i) Class-selective Relevance Map (CRM), which has been reported earlier, and (ii) Class Activation Map (CAM). The class prediction heatmaps are evaluated by a gynecologic oncologist with more than 20 years of experience. Based on our observation and the expert's opinion, the customized architecture not only outperforms the baseline RetinaNet network in treatability classification, but also provides insights about the features and regions considered significant by the network toward explaining reasons for treatment recommendation. Furthermore, by investigating the heatmaps on Gaussian-blurred images that serve as surrogates for out-of-focus cervical pictures we demonstrate the effect of image quality degradation on cervical treatability classification and underscoring the need for using images with good visual quality.

Keywords: RetinaNet features; cervical cancer; class activation mapping; class relevance mapping; concatenated features; customized CNN; deep learning; network visualization; thermal ablation; treatability.

Figure 1

Examples of digitized cervix images and anatomical illustration of a cervix. (a) An example of os (the opening into the uterus) having different shapes. (b) An illustration of cervix anatomy and illustration of T-zone. The cervix squamous and columnar cell regions are separated by the squamocolumnar junction (SCJ). As a woman ages the SCJ migrates from its original location toward the os.

Figure 2

Examples of cervix images used in this study, the top row images are labeled as “treatable” and the bottom row images are labeled as “not treatable” (reasons for not treatable given by the expert: left—lesion in canal, middle—SCJ not visible, right—lesion too large for ablation).

Figure 3

High-level architecture of the feature pyramid and our fine-tuned classification models with global average pooling (GAP) and fully connected (FC) layers. Top figure shows the architecture using one single pyramidal feature as the last convolutional layer, bottom figure shows the customized architecture using concatenated pyramidal features as the last convolutional layer. As each RetinaNet feature has 256 channel, the concatenated feature has 256 × N (feature number) channels. As shown in the 5 stacked gray feature blocks above, the concatenated features (P₃, P₄, P₅, P₆, and P₇) have 1280 channels.

Figure 4

Example of Class-selective Relevance Map (CRM) heatmap images generated from the fine-tuned classification model. The heatmaps in the 1st row are generated from the classification model built with Concatenate (P₃, P₆, P₇) as the last convolutional layer. The heatmaps in the 2nd row are generated from the classification model built with P₆ as the last convolutional layer. The heatmaps in the 3rd row are generated from the classification model built with Concatenate (P₃, P₄, P₅, P₆, P₇) as the last convolutional layer. The bottom row images are without any heatmap displayed.

Figure 4

Example of Class-selective Relevance Map (CRM) heatmap images generated from the fine-tuned classification model. The heatmaps in the 1st row are generated from the classification model built with Concatenate (P₃, P₆, P₇) as the last convolutional layer. The heatmaps in the 2nd row are generated from the classification model built with P₆ as the last convolutional layer. The heatmaps in the 3rd row are generated from the classification model built with Concatenate (P₃, P₄, P₅, P₆, P₇) as the last convolutional layer. The bottom row images are without any heatmap displayed.

Figure 5

Top: CAM visualization of model built with concatenated features (P₃, P₆, P₇) as the last convolutional layer. Bottom: CRM visualization of the same model as top. Note that the 1st and 2nd column from the left are “treatable” images, 3rd is the “not treatable” images (reason given by human expert: the SCJ is not visible, so the network is expected to look at the os region). Original images without heatmaps are shown in the 3rd row, respectively.

Figure 5

Top: CAM visualization of model built with concatenated features (P₃, P₆, P₇) as the last convolutional layer. Bottom: CRM visualization of the same model as top. Note that the 1st and 2nd column from the left are “treatable” images, 3rd is the “not treatable” images (reason given by human expert: the SCJ is not visible, so the network is expected to look at the os region). Original images without heatmaps are shown in the 3rd row, respectively.

Figure 6

(a) CRM visualization, which is considered as “highlighted area out of the region of interest”. (b) CRM visualization, which is considered as “insufficient coverage of region of interest”. (c) Sample of CRM visualization with a lower threshold of 15% of the maximum score on the same image with (b).

Figure 7

(a) Image sample labeled as “treatable” but considered as “not treatable” by the human expert. (b,c) Image samples that the human expert considers as “bad image” (the full cervix is not visible and bad focus).

Figure 8

Example of CRM heatmap generated from the fine-tuned classification model built with concatenated features (P₃, P₆, P₇) on blurred images. The top row images are input images: left—original unblurred, middle—blurred with (19, 19) filter size, right—blurred with (49, 49) filter size. The bottom row images are heatmaps generated using the left, middle and right images in the top row, respectively. In the heatmap of the original unblurred image, the highlighted area is around the os region, the middle heatmap although visually looks similar with the left one, the highlighted area is off the os region.