Segmentation of beating embryonic heart structures from 4-D OCT images using deep learning

Abstract

Optical coherence tomography (OCT) has been used to investigate heart development because of its capability to image both structure and function of beating embryonic hearts. Cardiac structure segmentation is a prerequisite for the quantification of embryonic heart motion and function using OCT. Since manual segmentation is time-consuming and labor-intensive, an automatic method is needed to facilitate high-throughput studies. The purpose of this study is to develop an image-processing pipeline to facilitate the segmentation of beating embryonic heart structures from a 4-D OCT dataset. Sequential OCT images were obtained at multiple planes of a beating quail embryonic heart and reassembled to a 4-D dataset using image-based retrospective gating. Multiple image volumes at different time points were selected as key-volumes, and their cardiac structures including myocardium, cardiac jelly, and lumen, were manually labeled. Registration-based data augmentation was used to synthesize additional labeled image volumes by learning transformations between key-volumes and other unlabeled volumes. The synthesized labeled images were then used to train a fully convolutional network (U-Net) for heart structure segmentation. The proposed deep learning-based pipeline achieved high segmentation accuracy with only two labeled image volumes and reduced the time cost of segmenting one 4-D OCT dataset from a week to two hours. Using this method, one could carry out cohort studies that quantify complex cardiac motion and function in developing hearts.

Fig. 1.

Exemplar OCT images captured from a plane of a beating embryonic heart. (a)-(d) are four distinct phases of a cardiac cycle. vtr: ventricle; CJ: cardiac jelly; myo: myocardium.

Fig. 2.

Image processing pipeline for segmentation of beating embryonic heart structures from 4-D OCT datasets. Step 1 is selecting multiple volumes at different time points of a cardiac cycle as key-volumes. Cardiac structures of key-volumes were manually labeled. Step 2 is synthesizing additional labeled image volumes by transformations learned from registration model VoxelMorph. Step 3 is semantic heart structure segmentation using U-Net trained on synthesized labeled images of step 2 and manually labeled images of step 1.

Fig. 3.

Generation of additional labeled image volumes using learning-based registration. (a) A VoxelMorph network was trained using random image volume pairs of a 4-D OCT dataset. In each volume pair, one volume was the fixed volume (f) and the other was moving (m). The network was trained by optimizing a loss function (Loss) to output a registration field φ that aligned the input image pair. The loss function Loss has a similarity term L_sim and a smoothness term L_smooth. The similarity term was measuring the difference between f and the moved volume m ○ φ. (b) The trained network was used to register the key-volumes to other unlabeled volumes. The key-volumes and their segmentations were transformed using the resulting registration field to create synthesized images and segmentations.

Fig. 4.

Segmentation performance of three methods versus the number of available labeled key-volumes (the average of 3 heart samples). The three methods were the two baseline methods using only VoxelMorph or U-Net, and the proposed combined strategy using VoxelMorph and U-Net. Mean Accuracy and Mean Dice are the average Accuracy and Dice scores of all classes in all images of all 3 heart samples. Mean BFScore is the boundary F1 contour matching score between the predicted segmentation and the ground truth segmentation of all 3 heart samples.

Fig. 5.

Segmentation results using the proposed method demonstrated on three heart samples with the Dice score ranking from top 10% to top 90% of all test images. N is the number of available key-volumes. The blue, green, and red contours are the myocardium outer boundary, myocardium inner boundary, and lumen boundary, respectively.

Fig. 6.

3-D reconstructions of extracted myocardium, cardiac jelly, and lumen from a beating heart at three time phases. The first row is showing the volume rendering results. The second row is showing the surface rendering of extracted lumens overlaid on cross-sectional OCT images. See Visualization 1 (MP4, 3.22 MB) for the shape changes of segmented cardiac structures during heart beating.