Evaluation of a multiview architecture for automatic vertebral labeling of palliative radiotherapy simulation CT images

Abstract

Purpose: The purpose of this work was to evaluate the performance of X-Net, a multiview deep learning architecture, to automatically label vertebral levels (S2-C1) in palliative radiotherapy simulation CT scans.

Methods: For each patient CT scan, our automated approach 1) segmented spinal canal using a convolutional-neural network (CNN), 2) formed sagittal and coronal intensity projection pairs, 3) labeled vertebral levels with X-Net, and 4) detected irregular intervertebral spacing using an analytic methodology. The spinal canal CNN was trained via fivefold cross validation using 1,966 simulation CT scans and evaluated on 330 CT scans. After labeling vertebral levels (S2-C1) in 897 palliative radiotherapy simulation CT scans, a volume of interest surrounding the spinal canal in each patient's CT scan was converted into sagittal and coronal intensity projection image pairs. Then, intensity projection image pairs were augmented and used to train X-Net to automatically label vertebral levels using fivefold cross validation (n = 803). Prior to testing upon the final test set (n = 94), CT scans of patients with anatomical abnormalities, surgical implants, or other atypical features from the final test set were placed in an outlier group (n = 20), whereas those without these features were placed in a normative group (n = 74). The performance of X-Net, X-Net Ensemble, and another leading vertebral labeling architecture (Btrfly Net) was evaluated on both groups using identification rate, localization error, and other metrics. The performance of our approach was also evaluated on the MICCAI 2014 test dataset (n = 60). Finally, a method to detect irregular intervertebral spacing was created based on the rate of change in spacing between predicted vertebral body locations and was also evaluated using the final test set. Receiver operating characteristic analysis was used to investigate the performance of the method to detect irregular intervertebral spacing.

Results: The spinal canal architecture yielded centroid coordinates spanning S2-C1 with submillimeter accuracy (mean ± standard deviation, 0.399 ± 0.299 mm; n = 330 patients) and was robust in the localization of spinal canal centroid to surgical implants and widespread metastases. Cross-validation testing of X-Net for vertebral labeling revealed that the deep learning model performance (F₁ score, precision, and sensitivity) improved with CT scan length. The X-Net, X-Net Ensemble, and Btrfly Net mean identification rates and localization errors were 92.4% and 2.3 mm, 94.2% and 2.2 mm, and 90.5% and 3.4 mm, respectively, in the final test set and 96.7% and 2.2 mm, 96.9% and 2.0 mm, and 94.8% and 3.3 mm, respectively, within the normative group of the final test set. The X-Net Ensemble yielded the highest percentage of patients (94%) having all vertebral bodies identified correctly in the final test set when the three most inferior and superior vertebral bodies were excluded from the CT scan. The method used to detect labeling failures had 67% sensitivity and 95% specificity when combined with the X-Net Ensemble and flagged five of six patients with atypical vertebral counts (additional thoracic (T13), additional lumbar (L6) or only four lumbar vertebrae). Mean identification rate on the MICCAI 2014 dataset using an X-Net Ensemble was increased from 86.8% to 91.3% through the use of transfer learning and obtained state-of-the-art results for various regions of the spine.

Conclusions: We trained X-Net, our unique convolutional neural network, to automatically label vertebral levels from S2 to C1 on palliative radiotherapy CT images and found that an ensemble of X-Net models had high vertebral body identification rate (94.2%) and small localization errors (2.2 ± 1.8 mm). In addition, our transfer learning approach achieved state-of-the-art results on a well-known benchmark dataset with high identification rate (91.3%) and low localization error (3.3 mm ± 2.7 mm). When we pre-screened radiotherapy CT images for the presence of hardware, surgical implants, or other anatomic abnormalities prior to the use of X-Net, it labeled the spine correctly in more than 97% of patients and 94% of patients when scans were not prescreened. Automatically generated labels are robust to widespread vertebral metastases and surgical implants and our method to detect labeling failures based on neighborhood intervertebral spacing can reliably identify patients with an additional lumbar or thoracic vertebral body.

Keywords: automatic vertebral labeling; deep learning.

Fig. 1

The X‐Net architecture. Input arms receive a sagittal and coronal intensity projection image pair and output sagittal and coronal arrays containing a unique channel for each vertebral label. Red arrows indicate 2D convolutions, PReLU activation, and batch norm (BN) layers. Blue arrows indicate transpose operations. White and blue rectangular volumes depict the general shape of the feature space after each operation. CIP, custom intensity projection. [Color figure can be viewed at wileyonlinelibrary.com]

Fig. 2

Visual depiction of IP image pair formation. (a) A 3D diagram depicting the formation of the coronal and sagittal intensity projection images across the VOI (red rectangle shows cross section of VOI) centered about the spinal canal (red). Maximum pixel values or mean pixel values are projected across orthogonal directions to obtain the MIP and AIP. (b) A depiction of how intensity projection pairs are augmented for training and for ablative testing. (c). An example of a central medial augmentation taken from the central region of an intensity projection pair (grey regions are removed) with ground truth annotations in magenta. [Color figure can be viewed at wileyonlinelibrary.com]

Fig. 3

CIP and MIP image pairs. Custom (a–c) and maximum (d–e) intensity projection image pairs depicting sagittal (top row) and coronal (bottom row) views. “Lower” and “upper” indicate bounds for CT number normalization before application of CIP formation formula in 2.C.1. Red dashed lines indicate VOI boundaries formed from the spinal canal. Yellow dashed lines indicate the boundary further constraining the VOI at the midpoint of each axial slice from the spinal canal (thus excluding posterior processes from the VOI). In this patient, from the cross‐validation dataset, a lumbar vertebroplasty is present and creates a high‐intensity region in the IP images. [Color figure can be viewed at wileyonlinelibrary.com]

Fig. 4

(a) The frequency of cropped CIP image pairs, grouped by the number of vertebrae per cropping. Lumbar (red), thoracic(yellow), and cervical (blue) indicate if the cropped scan was centered upon lumbar, thoracic, or cervical regions. (b) F₁ score (lime green), precision (green), and sensitivity (dark green) as a function of the number of vertebral bodies within each cropping (irrespective of scan type). [Color figure can be viewed at wileyonlinelibrary.com]

Fig. 5

Intervertebral spacing distribution in the best performing cross‐validation split using X‐Net. This spacing is the 3D distance from one vertebral body to the next in millimeters. Box plots in red and blue indicate junctions where vertebral bodies were incorrectly or correctly labeled, respectively. [Color figure can be viewed at wileyonlinelibrary.com]

Fig. 6

Optimization of the method to detect labeling failures. An example of a sagittal CIP prediction featuring a large rate of change in vertebral body spacing detection (a) and ROC curve (b). Blue, green, and orange plots are sagittal, 3D, and coronal δ plots, respectively. A large δ at L3‐L1 exists. This patient has an additional lumbar vertebrae (L6). The presence of a large gap between predicted lumbar centroids was observed when more than five lumbar exist. [Color figure can be viewed at wileyonlinelibrary.com]

Fig. 7

Vertebral body identification rates for X‐Net (green), the X‐Net ensemble (blue), and Btrfly Net (red) for all vertebral bodies within the final test set (n = 94). [Color figure can be viewed at wileyonlinelibrary.com]

Fig. 8

Vertebral body identification rates for X‐Net (green), the X‐Net ensemble (blue), and Btrfly Net (red) for vertebral bodies in the normative group within the final test set (n = 74). [Color figure can be viewed at wileyonlinelibrary.com]

Fig. 9

Vertebral body identification rates for X‐Net (green), the X‐Net ensemble (blue), and Btrfly Net (red) for vertebral bodies in the outlier group within the final test set (n = 20). [Color figure can be viewed at wileyonlinelibrary.com]

Fig. 10

Model predictions of vertebral level on a normative patient with 21 vertebral bodies spanning C1‐L2. (a) X‐Net, (b) X‐Net ensemble, and (c) Btrfly Net predictions show TPs in yellow diamonds and FPs in red diamonds. Ground truth predictions are shown in magenta circles. The clinically viable region spanned T11‐C4, and all three predictions were 100% accurate in this region. IR, identification rate; error, localization error. [Color figure can be viewed at wileyonlinelibrary.com]

Fig. 11

Model predictions of vertebral levels for a patient with an extra lumbar body (L6) in the outlier group within the final test set. (a) X‐Net, (b) X‐Net ensemble, and (c) Btrfly Net predictions TPs in yellow diamonds and FPs in red diamonds. Ground truth predictions are shown in magenta circles. High pixel numbers above L1‐L2 were due to the presence of nephrectomy surgical clips. In sagittal views of (a),(b), and (c) large spaces were left at L5, L4, and L3, respectively. IR, identification rate; error, localization error. [Color figure can be viewed at wileyonlinelibrary.com]

Fig. 12

Model predictions of vertebral level on patient with extensive surgical implants (spanning L1‐T4) in the outlier group within the final test set. (a) X‐Net, (b) X‐Net ensemble, and (c) Btrfly Net predictions show TPs in yellow diamonds and FPs in red diamonds. Ground truth predictions are shown in magenta circles. IR, identification rate; error, localization error. [Color figure can be viewed at wileyonlinelibrary.com]