Deep Geodesic Learning for Segmentation and Anatomical Landmarking

Abstract

In this paper, we propose a novel deep learning framework for anatomy segmentation and automatic landmarking. Specifically, we focus on the challenging problem of mandible segmentation from cone-beam computed tomography (CBCT) scans and identification of 9 anatomical landmarks of the mandible on the geodesic space. The overall approach employs three inter-related steps. In the first step, we propose a deep neural network architecture with carefully designed regularization, and network hyper-parameters to perform image segmentation without the need for data augmentation and complex post-processing refinement. In the second step, we formulate the landmark localization problem directly on the geodesic space for sparsely-spaced anatomical landmarks. In the third step, we utilize a long short-term memory network to identify the closely-spaced landmarks, which is rather difficult to obtain using other standard networks. The proposed fully automated method showed superior efficacy compared to the state-of-the-art mandible segmentation and landmarking approaches in craniofacial anomalies and diseased states. We used a very challenging CBCT data set of 50 patients with a high-degree of craniomaxillofacial variability that is realistic in clinical practice. The qualitative visual inspection was conducted for distinct CBCT scans from 250 patients with high anatomical variability. We have also shown the state-of-the-art performance in an independent data set from the MICCAI Head-Neck Challenge (2015).

Fig. 1:

Anatomical landmarks on the mandible: Menton (Me), Gnathion (Gn), Pogonion (Pg), B Point (B), Infradentale (Id), Condylar Left (Cd_L), Condylar Right (Cd_R), Coronoid Left (Cor_L), and Coronoid Right (Cor_R). We aim to locate these landmarks automatically.

Fig. 2:

Examples of diverse CMF conditions are illustrated. (A) Surgical treatment, genioplasty with resultant chin advancement and fixation plate (implant) (adult), (B) missing condyle-ramus unit in the mandible in left dominant hemifacial microsomia (adult), (C) unerupted teeth in the anterior mandible with distorted anatomy (pediatric), (D) mid-sagittal plane with respect to lower jaw incisors have a serious degradation from the 90 degrees (pediatric), (E) bilateral bicortical positional screws (implants) in the ascending ramus of the mandible for rigid fixation after a bileteral sagittal split osteotomy (adult), (F) plate and screws (implants) in the anterior mandible for rigid fixation and reduction of an oblique fracture (adult).

Fig. 3:

Following the Mandible Segmentation with Fully Convolutional DenseNet, Linear Time Distance Transform (LTDT) of the Mandible Bone is generated. A second U-Net [8] is used to transform LTDTs into a combined Geodesic Map of the mandibular landmarks Menton (Me), Condylar Left (Cd_L), Condylar Right (Cd_R), Coronoid Left (Cd_L), and Coronoid Right (Cd_R). Finally, an LSTM Network is used to detect Infradentale (Id), B point (B), Pogonion (Pg), and Gnathion (Gn) mandibular landmarks according to the detected position of the Menton (Me) in previous step. All algorithms in this proposed pipeline run in pseudo-3D (slice-by-slice 2D). To ease understanding of the segmentation results, surface rendered volumes are presented instead of contour based binary images.

Fig. 4:

The example workflow of a single slice in the proposed pipeline (Figure 3). The outputs of the steps (Landmark Classification and LSTM Network) are zoomed in for visual illustration of the process.

Fig. 5:

(a) General architecture of the Tiramisu [17] is illustrated. The architecture is composed of downsampling and upsampling paths including Convolution, Dense Block, Concatenation (C), Skip Connection (dashed lines), Transition Down, and Transition Up layers. Concatenation layer appends the input of the dense block layer to the output of it. Skip connection copies the concatenated feature maps to the upsampling path. (b) A sample dense block with 4 layers is shown to its connections. With a growth rate of k, each layer in dense block appends k feature maps to the input. Hence, the output contains 4 × k features maps.

Fig. 6:

LSTM network input-output. Each row of the scaled sagittal boundary image is input to the corresponding LSTM block, and binary 1D vector of locations annotated as landmark (1), or no-landmark (0) is output.

Fig. 7:

Details of the network architecture (LSTM) for identifying closely-spaced landmarks. Gnathion (Gn), Pogonion (Pg), B Point (B), and Infradentale (Id) are determined once the Menton (Me) is detected through U-Net architecture as shown in Step 3 of the Figure 3. Input image resolution is RxK, and the LSTM cell is composed of 512 hidden units.

Fig. 8:

(a) Errors in pixel space, (b) errors in the volume space, (c) inter-observer reading variations in pixel space.

Fig. 9:

Experimental renderings demonstrating segmentation and landmark localization results of patients with high anatomical variability due to deformities and surgical intervention

Fig. 10:

Impact of segmentation accuracy on the landmark localization process.

Fig. 11:

Summary of qualitative evaluation of 250 scans from the NIDCR/NIH dataset evaluated by 2 experts, A and B

Fig. 12:

Qualitative Evaluation Scores of the Segmentation Results. The experts visually evaluated the performance of the segmentation of the 250 patient scans in the score range 1 to 4, where 1 is inferior. Examples of scans with scores 1–4 are presented.