Rule-based automatic diagnosis of thyroid nodules from intraoperative frozen sections using deep learning

Abstract

Frozen sections provide a basis for rapid intraoperative diagnosis that can guide surgery, but the diagnoses often challenge pathologists. Here we propose a rule-based system to differentiate thyroid nodules from intraoperative frozen sections using deep learning techniques. The proposed system consists of three components: (1) automatically locating tissue regions in the whole slide images (WSIs), (2) splitting located tissue regions into patches and classifying each patch into predefined categories using convolutional neural networks (CNN), and (3) integrating predictions of all patches to form the final diagnosis with a rule-based system. To be specific, we fine-tune the InceptionV3 model for thyroid patch classification by replacing the last fully connected layer with three outputs representing the patch's probabilities of being benign, uncertain, or malignant. Moreover, we design a rule-based protocol to integrate patches' predictions to form the final diagnosis, which provides interpretability for the proposed system. On 259 testing slides, the system correctly predicts 95.3% (61/64) of benign nodules and 96.7% (148/153) of malignant nodules, and classify 16.2% (42/259) slides as uncertain, including 19 benign and 16 malignant slides, which are a sufficiently small number to be manually examined by pathologists or fully processed through permanent sections. Besides, the system allows the localization of suspicious regions along with the diagnosis. A typical whole slide image, with 80, 000 × 60, 000 pixels, can be diagnosed within 1 min, thus satisfying the time requirement for intraoperative diagnosis. To the best of our knowledge, this is the first study to apply deep learning to diagnose thyroid nodules from intraoperative frozen sections. The code is released at https://github.com/PingjunChen/ThyroidRule.

Keywords: Deep learning; Frozen section; Rule-based protocol; Thyroid nodule; Whole slide image.

Fig. 1.

The pipeline of training and testing the model for diagnosis of thyroid nodules from frozen sections. Module (A) shows how patches of three categories are cropped from annotated slides. Uncertain patches and malignant patches are cropped from uncertain and malignant slides, respectively. Benign patches are cropped from benign regions in each of the three slide types. All cropped patches are used to fine-tune the deep learning model in module (B). In the testing stage, the trained model is applied to all patches inside localized tissues in testing slides. All patch predictions are integrated to form the final diagnosis according to a rule-based protocol. Note: “N” stands for negative, namely benign patch; “U” stands for uncertain patch; and “P” stands for positive, namely malignant patch.

Fig. 2.

Frozen section examples of (A) benign, (B) uncertain, and (C) malignant thyroid nodules.

Fig. 3.

Image patch cropping from annotated frozen sections. Contours are annotated and validated by certified pathologists. All regions in the benign slide are considered as benign. A few white background regions are specifically set as benign regions. Benign regions can also exist on uncertain and malignant slides. Benign, uncertain, and malignant patches are randomly cropped from these annotated regions with corresponding categories. The number of cropped patches is set to be proportional to the area of the annotated region.

Fig. 4.

Thyroid tissue localization pipeline. Tissue localization is applied at the 4th level of the whole slide image to eliminate background regions that are irrelevant to slide diagnosis, thereby speeding up the automatic diagnosis process. Color space conversion, image smoothing, inverse binarization, and binary image refinement are successively applied to generate the thyroid tissue mask.

Fig. 5.

Thyroid whole slide image tissue localization examples. The first row shows two sample whole slide images. The second row shows tissue localization results (marked in blue).

Fig. 6.

The proposed rule-based protocol for the diagnosis of thyroid frozen sections based on patch classification results.

Fig. 7.

The flow chart of the proposed rule-based diagnosis system for the thyroid frozen section. The main components in the system contains tissue region localization, patch splitting and category prediction based on CNN, and the rule-based slide diagnosis protocol.

Fig. 8.

Comparison of three different classifiers, including InceptionV3, VGG16BN, and ResNet50, on thyroid patch classification via fine-tuning and training from scratch.

Fig. 9.

Example predictions of patch-based classification of thyroid frozen sections. The first row shows benign, uncertain, and malignant input slides. The second row shows corresponding patch-based predictions. Three predicted binary maps (benign, uncertain, and malignant) are combined into a single image, in which green represents benign; blue, uncertain; and red, malignant.

Fig. 10.

Statistical analysis of the four most interested metrics in thyroid frozen section diagnosis. Four interested thyroid diagnosis metrics, including benign precision, malignant precision, uncertain sensitivity, and overall accuracy, are computed via slides random sampling from the whole testing slides.

Fig. 11.

Five benign slides that the proposed system misclassifies as malignant. Those regions demarcated using the red bounding boxes, appear very similar to the malignant regions on malignant thyroid slides.