A Deep Learning Approach for Segmentation, Classification, and Visualization of 3-D High-Frequency Ultrasound Images of Mouse Embryos

Abstract

Segmentation and mutant classification of high-frequency ultrasound (HFU) mouse embryo brain ventricle (BV) and body images can provide valuable information for developmental biologists. However, manual segmentation and identification of BV and body requires substantial time and expertise. This article proposes an accurate, efficient and explainable deep learning pipeline for automatic segmentation and classification of the BV and body. For segmentation, a two-stage framework is implemented. The first stage produces a low-resolution segmentation map, which is then used to crop a region of interest (ROI) around the target object and serve as the probability map of the autocontext input for the second-stage fine-resolution refinement network. The segmentation then becomes tractable on high-resolution 3-D images without time-consuming sliding windows. The proposed segmentation method significantly reduces inference time (102.36-0.09 s/volume ≈ 1000× faster) while maintaining high accuracy comparable to previous sliding-window approaches. Based on the BV and body segmentation map, a volumetric convolutional neural network (CNN) is trained to perform a mutant classification task. Through backpropagating the gradients of the predictions to the input BV and body segmentation map, the trained classifier is found to largely focus on the region where the Engrailed-1 (En1) mutation phenotype is known to manifest itself. This suggests that gradient backpropagation of deep learning classifiers may provide a powerful tool for automatically detecting unknown phenotypes associated with a known genetic mutation.

Fig. 1.

(a–f) 6 embryonic mice HFU volumes are shown with three views each: a B-mode image slice from the 3D volume, a manual BV (green) and body (red) segmentation, and a 3D rendering (visualized in natural orientation relative to the HFU probe). The numbers below each 3D rendering indicate corresponding image size in voxels. The arrow in a) indicates an ambiguous boundary due to contact between the body and uterine wall. The arrows in b), c) and d) indicate motion artifacts because of irregular physiological movements of the anesthetized pregnant mice. The arrows in e) and f) indicate missing head boundaries due to either specular reflections or shadowing from overlaying tissues.

Fig. 2.

The pipeline of the joint BV and body segmentation from 3D HFU images of mouse embryos.

Fig. 3.

Diagram of localization-auto-context (Loc-Con) module for the BV. A similar configuration is used for the body. The gradient produced by the refinement loss can flow back to the low-resolution segmentation network (blue arrows).

Fig. 4.

Procedure for rotating each BV and body segmentation map into a canonical orientation. The first Principle Component (PC) of BV and the first PC of body are used (indicated as dash line in (a)). The up-down direction (dash arrow in (a)) is determined by the comparative centroid positions of BV and body. The front-back direction (dash arrow in (b)) is determined by the structural characteristic of BV because the front BV region is wider than the back BV region (solid line in (b)).

Fig. 5.

Pictorial representation of the mutant classification network. The numbers below each box indicate channel, depth, height and width. “bn”, “relu”, “conv”, “max pool”, “global average pool” and “WX+b” indicate batch normalization, rectified linear unit, convolution, max pooling, global average pooling and fully connected layer operations, respectively.

Fig. 6.

Comparison of initial coarse segmentation and refined segmentation for four HFU volumes. Green indicates BV, red indicates body and the numbers below the predicted segmentation correspond to DSC. In a), b) and c), yellow arrows indicate that the refinement improved the segmentation in terms of boundary and structure. d) represents an image with motion artifacts where the manual segmentation was noisy in the body background boundary while the refinement network produced a smooth boundary which was closer to the true physical structure. The refined BV segmentation in d) is also more accurate than the initial BV segmentation.

Fig. 7.

Comparison of qualitative segmentation results among different methods for five HFU images. Green indicates BV, red indicates body and the numbers below the predicted segmentation are corresponding DSC. Yellow arrow in a) indicates ambiguous boundary due to the deep touching of the body and uterine wall. Image b) has severe motion artifacts. Yellow arrow in c) indicates missing head boundary. Image d) has different contrast with image b) and c). Yellow arrow in e) indicates severe missing signal of body, which leads to unsatisfactory automatic body segmentation results across different methods.

Fig. 8.

ROC curves and AUC scores of the mutant classification results with different input combinations (each obtained with six-fold cross validation).

Fig. 9.

Saliency images of the trained mutant classification neural network. The first row is the normal mouse embryo BV (green) and body (red) segmentation while the second row is mutant. Two images are presented for each sample. The blue arrow in the first image indicates the known structural differences between En1 mutant and normal BVs while the blue dots in the second image (salient points) indicate where the trained network focused when making the prediction.