Multi-Task Learning for Motion Analysis and Segmentation in 3D Echocardiography

Abstract

Characterizing left ventricular deformation and strain using 3D+time echocardiography provides useful insights into cardiac function and can be used to detect and localize myocardial injury. To achieve this, it is imperative to obtain accurate motion estimates of the left ventricle. In many strain analysis pipelines, this step is often accompanied by a separate segmentation step; however, recent works have shown both tasks to be highly related and can be complementary when optimized jointly. In this work, we present a multi-task learning network that can simultaneously segment the left ventricle and track its motion between multiple time frames. Two task-specific networks are trained using a composite loss function. Cross-stitch units combine the activations of these networks by learning shared representations between the tasks at different levels. We also propose a novel shape-consistency unit that encourages motion propagated segmentations to match directly predicted segmentations. Using a combined synthetic and in-vivo 3D echocardiography dataset, we demonstrate that our proposed model can achieve excellent estimates of left ventricular motion displacement and myocardial segmentation. Additionally, we observe strong correlation of our image-based strain measurements with crystal-based strain measurements as well as good correspondence with SPECT perfusion mappings. Finally, we demonstrate the clinical utility of the segmentation masks in estimating ejection fraction and sphericity indices that correspond well with benchmark measurements.

Fig. 1.

Overview of the proposed network architecture. The network is comprised of two task-specific branches: a motion tracking branch (in red) and a segmentation branch (in blue). Cross stitch units (in purple) placed at each layer in the network learn to linearly combine features from each branch that optimizes a combined loss function using a matrix of learnable parameters. A shape-consistency unit connects the outputted motion estimation from the motion branch with the output layer of the segmentation branch prior to being inputted to a final prediction layer to produce segmentation masks consistent between both branches by employing a similar cross stitch unit.

Fig. 2.

The shape-consistency unit. $U_{x,y,z}$ estimated from the motion tracking branch is used to spatially transform $S_{source}^{gt}$. Both the original and the transformed mask are then convolved and linearly combined with features from the output layer of the segmentation branch before being inputted into a final prediction layer

Fig. 3.

Long (bottom) and short-axis (top) visualizations of end-diastole to end-systole motion estimates from various competing methods for a representative porcine study. Columns 4 and 5 are versions of our proposed model with and without regularizations

Fig. 4.

Regression plot analyzing correlation between crystal derived strains and the calculated strains at every time frame in the infarcted regions of the left ventricular myocardium. (Left to right) radial, circumferential, longitudinal

Fig. 5.

A demonstration of the ability of our proposed model to identify regions of ischemia, 7 days post infarction. The top row presents the raw bMode image (top-left), estimated dense principal strain (top-middle), and the thresholded ischemic region (top-right). The bottom row represents the corresponding SPECT perfusion map. The red arrows display the predicted areas of low activity, which demonstrate good agreement between the two modalities.

Fig. 6.

Long and short-axis segmentation predictions from various competing methods (columns 2–5) compared against a manual tracing (column 1) for a representative porcine study. The blue shaded region represents the left ventricular myocardium whereas red represents the blood pool.

Fig. 7.

The effects on model performance from manually tuning cross-stitch parameters. The red curves represent a change in the average volumetric similarity (between myocardium and blood pool) when tuning ${\alpha }_{motion}$, whereas the blue curves represent the average dice score when tuning ${\beta }_{seg}$. The dotted vertical line represents the optimally learned value of that specific layer and parameter

Fig. 8.

The effects of manually tuning the cross-stitch unit of the shape-consistency unit illustrated using a representative porcine study. The middle column represents the optimally learned shape-consistency unit cross-stitch parameter whereas the left column represents an output that is biased toward motion features and the right represents an output that is biased toward segmentation features. The blue shaded region represents the myocardium, the red represents the blood pool, and the yellow represents an overlap in classification