Echocardiogram Vector Embeddings Via R3D Transformer for the Advancement of Automated Echocardiography

Abstract

Background: Ejection fraction (EF) estimation informs patient plans in the ICU, and low EF can indicate ventricular systolic dysfunction, which increases the risk of adverse events including heart failure. Automated echocardiography models are an attractive solution for high-variance human EF estimation, and key to this goal are echocardiogram vector embeddings, which are a critical resource for computational researchers.

Objectives: The authors aimed to extract the vector embeddings from each echocardiogram in the EchoNet dataset using a classifier trained to classify EF as healthy (>50%) or unhealthy (<= 50%) to create an embeddings dataset for computational researchers.

Methods: We repurposed an R3D transformer to classify whether patient EF is below or above 50%. Training, validation, and testing were done on the EchoNet dataset of 10,030 echocardiograms, and the resulting model generated embeddings for each of these videos.

Results: We extracted 400-dimensional vector embeddings for each of the 10,030 EchoNet echocardiograms using the trained R3D model, which achieved a test AUC of 0.916 and 87.5% accuracy, approaching the performance of comparable studies.

Conclusions: We present 10,030 vector embeddings learned by this model as a resource to the cardiology research community, as well as the trained model itself. These vectors enable algorithmic improvements and multimodal applications within automated echocardiography, benefitting the research community and those with ventricular systolic dysfunction (https://github.com/Team-Echo-MIT/r3d-v0-embeddings).

Keywords: EchoNet; R3D transformer; dataset; echocardiography; embeddings; video.

Graphical abstract

Figure 1

Example of a Shortcut (“Skip”) Connection Within a Residual Block ResNet models use residual blocks with shortcut (“skip”) connections across their convolutional layers. This connects layers within a residual block that otherwise have no direct backpropagation between them.

Figure 2

Distribution of Ejection Fraction for All Echocardiograms EF is a percentage between 0 to 100. Echocardiograms in the EchoNet dataset had an EF distribution specified by this figure, with a left skew and a mode around 60%. A class cutoff of EF = 50% results in a class imbalance.

Figure 3

R3D Model Performance for Various Sampling Periods Binary cross entropy (BCE) loss only increases once we sample from every 6 frames. This means 1-in-4 frame sampling likely does not exclude key frames necessary for EF classification.

Central Illustration

Echocardiogram Vector Embeddings Via R3D Transformer for the Advancement of Automated Echocardiograph Echocardiograms, represented as black frame sequences in the blue-shaded boxes, are fed into the R3D model, where 3D convolutions pick up spatiotemporal patterns and eventually reduce the video into a 400-dimensional fully connected layer before the sigmoid head. We extract the last fully-connected layer as the vector embeddings of an echocardiogram, and the principle component analysis (PCA) of these vector embeddings shows that EF patterns are preserved among them. Black arrows = show equivalence; black lines = highlight dimensions; blue lines = represent a 3D convolution; purple shapes = the vector embeddings; blue and green shapes = convolutional and pooling layers; orange shapes = fully connected and dropout layers; purple shape = sigmoid layer. Our workflow is 2-fold: we train the R3D transformer to discriminate between high and low EF and use the trained R3D to generate vector embeddings for each echocardiogram in the EchoNet dataset.

Figure 4

AUC Curve for Training, Validation, and Test Sets AUC (area under the curve) measures holistic discriminative ability and is 1.0 for a perfect binary classifier. Our test AUC of 0.92 thus demonstrates a strong learned representation by our model.

Figure 5

A 2-Dimensional View of the EchoNet Embeddings Condensing the embeddings from 400 dimensions to 2 principle components (PCs) results in the above scatterplot, where visible trends in EF show that our embeddings successfully capture EF variation.