Automated Segmentation of Microvessels in Intravascular OCT Images Using Deep Learning

Abstract

Microvessels in vascular plaque are associated with plaque progression and are found in plaque rupture and intra-plaque hemorrhage. To analyze this characteristic of vulnerability, we developed an automated deep learning method for detecting microvessels in intravascular optical coherence tomography (IVOCT) images. A total of 8403 IVOCT image frames from 85 lesions and 37 normal segments were analyzed. Manual annotation was performed using a dedicated software (OCTOPUS) previously developed by our group. Data augmentation in the polar (r,θ) domain was applied to raw IVOCT images to ensure that microvessels appear at all possible angles. Pre-processing methods included guidewire/shadow detection, lumen segmentation, pixel shifting, and noise reduction. DeepLab v3+ was used to segment microvessel candidates. A bounding box on each candidate was classified as either microvessel or non-microvessel using a shallow convolutional neural network. For better classification, we used data augmentation (i.e., angle rotation) on bounding boxes with a microvessel during network training. Data augmentation and pre-processing steps improved microvessel segmentation performance significantly, yielding a method with Dice of 0.71 ± 0.10 and pixel-wise sensitivity/specificity of 87.7 ± 6.6%/99.8 ± 0.1%. The network for classifying microvessels from candidates performed exceptionally well, with sensitivity of 99.5 ± 0.3%, specificity of 98.8 ± 1.0%, and accuracy of 99.1 ± 0.5%. The classification step eliminated the majority of residual false positives and the Dice coefficient increased from 0.71 to 0.73. In addition, our method produced 698 image frames with microvessels present, compared with 730 from manual analysis, representing a 4.4% difference. When compared with the manual method, the automated method improved microvessel continuity, implying improved segmentation performance. The method will be useful for research purposes as well as potential future treatment planning.

Keywords: classification; deep learning; microvessel; optical coherence tomography; segmentation.

Figure 1

The DeepLab v3 plus network consists of atrous convolution, atrous spatial pyramid pooling, and encoder–decoder. The Xception network was used as the backbone network for feature extraction. The left and right figures are the pre-processed IVOCT image and predicted label, respectively (red arrow: microvessel). The sizes of input and output images are the same (200 × 448 pixels). In the input image, the black strip indicates the guidewire shadow removed during pre-processing.

Figure 2

Classification of microvessel candidates using a simple CNN model. Each candidate was classified as either microvessel or non-microvessel using a CNN model (green) consisting of three convolutional, two maximum pooling, and one fully connected layers. The red box is a microvessel, and the blue box is a non-microvessel.

Figure 3

Data augmentation for candidate classification network. The bounding boxes with microvessel (red) were augmented by rotating its angle from 30° to 180° with a 30° interval, providing a seven times greater number of microvessel cases. The bounding boxes without microvessel (blue) were not augmented.

Figure 4

Intra−observer variability analysis of microvessel areas. Panels are as follows: (left) linear regression plot and (right) Bland−Altman plot. Linear regression gave R² = 0.909. Dice was 0.970. In the Bland−Altman analysis and the mean bias was only about 0.0003 ± 0.001 mm², about 4.7% of the mean area. This result suggests excellent manual reproducibility within a single analyst.

Figure 5

Segmentation results with and without data augmentation and pre-processing. Panels are results from (A) manual annotation, (B) raw polar IVOCT images, (C) data augmentation on the raw polar IVOCT image, (D) pre-processing alone, and (E) data augmentation and pre-processing. Both data augmentation and pre-processing improved the segmentation results. In some cases (second and third columns), our method detected microvessels that were missed during manual annotation, as confirmed by the analyst. Microvessels are labeled in red.

Figure 6

Segmentation results with and without the candidate classification step. Panels are as follows: (A) manual annotation, (B) results before candidate classification, and (C) results after candidate classification. There were some false positives from the candidate segmentation step (B), which were effectively ruled out with the candidate classification step (C). The red is a microvessel.

Figure 7

Segmentation results of microvessel according to different semantic segmentation models. Panels are as follows: (A) manual annotation, (B) results obtained using U-Net, (C) results obtained using SegNet, and (D) results obtained using DeepLab v3 plus. In (C), the SegNet had similar results as compared with DeepLab v3 plus; however, there were some misdetections. The U-Net (B) had the worst detection results among all deep learning models. The red is a microvessel.

Figure 8

Examples of saliency maps obtained by different semantic segmentation networks ((A): U-Net, (B): SegNet, and (C): DeepLab v3+). The U-Net (A) showed a number of false activations along the angular rotation (r), while SegNet and DeepLab v3 plus focused on the specific regions with microvessels.

Figure 9

Microvessels from a representative IVOCT pullback in a plaque. On the right are 3D visualizations from manual (top) and automated (bottom) microvessel segmentations, as well as the vessel lumen. Microvessels look very similar between the top and bottom, except for within the yellow bounding boxes, showing a microvessel “behind” the vessel lumen. In this latter instance, the microvessel from the automated analysis is much more continuous than that from the manual analysis, indicating that the automated method detects more actual microvessel instances than the manual analysis. Missing annotations in the manual analysis are evident in the original IVOCT images on the left, where an annotation is missing despite image evidence. The longest microvessel is ~7.4 mm in length with a diameter of about ~107.4 μm. The red is lumen and the green is a microvessel.