Diagnosing oral and maxillofacial diseases using deep learning

Abstract

The classification and localization of odontogenic lesions from panoramic radiographs is a challenging task due to the positional biases and class imbalances of the lesions. To address these challenges, a novel neural network, DOLNet, is proposed that uses mutually influencing hierarchical attention across different image scales to jointly learn the global representation of the entire jaw and the local discrepancy between normal tissue and lesions. The proposed approach uses local attention to learn representations within a patch. From the patch-level representations, we generate inter-patch, i.e., global, attention maps to represent the positional prior of lesions in the whole image. Global attention enables the reciprocal calibration of path-level representations by considering non-local information from other patches, thereby improving the generation of whole-image-level representation. To address class imbalances, we propose an effective data augmentation technique that involves merging lesion crops with normal images, thereby synthesizing new abnormal cases for effective model training. Our approach outperforms recent studies, enhancing the classification performance by up to 42.4% and 44.2% in recall and F1 scores, respectively, and ensuring robust lesion localization with respect to lesion size variations and positional biases. Our approach further outperforms human expert clinicians in classification by 10.7 % and 10.8 % in recall and F1 score, respectively.

Figure 1

Overview of DOLNet: (a) the first stage for extracting patch-level representations from the proposed mutually influencing hierarchical attention. The initial patch encoding is obtained from individual local attention, and they are aggregated to create a global attention map, shown as a dotted red box, which inversely calibrates the patch representations. (b) The second stage corresponding to Eq. (1) with two branches for lesion classification and localization. (c) Preprocessed input image and patch split (best viewed in color).

Figure 2

LesionMix, proposed data augmentation: (a) two samples of “AB” and “Normal” cases to generate a synthetic “AB” sample, and (b) comparisons of the result of LesionMix with that of Mixup by mixing the sources in the ratio 0.5:0.5 and Cutmix. For each method, crops containing lesions are highlighted.

Figure 3

Characteristics of a dataset for this study: (a) the distribution of lesion size in the images, and (b) the spatial distribution of lesions across the images in terms of the normalized heat map corresponding to all lesion segmentations in our dataset (best viewed in color).

Figure 4

Effects of lesion size and location on classification by comparing the proposed method with (a) existing models, Kwon and Hu using the dataset $\mathcal{D}$ and (b) human clinicians using the dataset ${\mathcal{D}}_{\mathit{tiny}}$ and on localization by comparing with (c) the models. For the three plots with their horizontal axes labeled as size, the numbers correspond to identifiers of image groups (i.e., bins) according to the lesion size. Bins with larger ids contain images with larger lesions. Also, for the plots with the horizontal axes labeled as position, the numbers correspond to the four quadrants.

Figure 5

Qualitative results. Each of the rows contains a test image with its target class and lesion boundary annotated in red ($1\text{st}$ column), and the predictions from four methods as blue heatmaps indicating lesions: Kwon et al. ($2\text{nd}$), Hu et al. ($3\text{rd}$), the backbone of DOLNet ($4\text{th}$), and the full DOLNet (the last) (best viewed in color).