Deep learning for transesophageal echocardiography view classification

Abstract

Transesophageal echocardiography (TEE) imaging is a vital tool used in the evaluation of complex cardiac pathology and the management of cardiac surgery patients. A key limitation to the application of deep learning strategies to intraoperative and intraprocedural TEE data is the complexity and unstructured nature of these images. In the present study, we developed a deep learning-based, multi-category TEE view classification model that can be used to add structure to intraoperative and intraprocedural TEE imaging data. More specifically, we trained a convolutional neural network (CNN) to predict standardized TEE views using labeled intraoperative and intraprocedural TEE videos from Cedars-Sinai Medical Center (CSMC). We externally validated our model on intraoperative TEE videos from Stanford University Medical Center (SUMC). Accuracy of our model was high across all labeled views. The highest performance was achieved for the Trans-Gastric Left Ventricular Short Axis View (area under the receiver operating curve [AUC] = 0.971 at CSMC, 0.957 at SUMC), the Mid-Esophageal Long Axis View (AUC = 0.954 at CSMC, 0.905 at SUMC), the Mid-Esophageal Aortic Valve Short Axis View (AUC = 0.946 at CSMC, 0.898 at SUMC), and the Mid-Esophageal 4-Chamber View (AUC = 0.939 at CSMC, 0.902 at SUMC). Ultimately, we demonstrate that our deep learning model can accurately classify standardized TEE views, which will facilitate further downstream deep learning analyses for intraoperative and intraprocedural TEE imaging.

Figure 1

Sample training images used for the deep-learning view classification task. Images are 2-dimensional still frames sampled from the video data used in model training. Eight standard TEE views were chosen, including: the ME 2-Chamber View, ME 4-Chamber View, ME AV SAX View, ME Bicaval View, ME LAA View, ME Long Axis View, TG LV SAX View, and Aortic View. TEE, transesophageal echocardiography; ME, mid-esophageal; AV, aortic valve; SAX, short axis; LAA, left atrial appendage; TG, trans-gastric; LV, left ventricular.

Figure 2

View classification model performance on the internal (CSMC) hold-out test set and the external (SUMC) test set. (a) AUC’s for each view class, demonstrating high accuracy (with AUC’s ranging from 0.816 to 0.957). No AUC was able to be calculated for the ME Left Atrial Appendage View in the randomly selected SUMC test set due to low sampling. (b) Confusion matrices showing model performance, with views labeled by a board-certified echocardiographer along the vertical axis and views predicted by the deep learning model on the horizontal axis. Numerical values in the matrices and the color intensity of the heatmaps represent the number of images with the indicated ground-truth and model-predicted labels. AUC, area under the receiver operating curve; CSMC, Cedars Sinai Medical Center; SUMC, Stanford University Medical Center; ME, mid-esophageal; AV, aortic valve; SAX, short axis; TG, trans-gastric; LV, left ventricular.

Figure 3

Clustering analysis showing the ability to distinguish among standard TEE views. t-SNE clustering analysis of input images demonstrates that meaningful representations of standard TEE views are clustered appropriately together. In other words, images are sorted into groups that reflect standard TEE classes. Embedding representation is consistent across CSMC and SUMC, suggesting robustness and generalizability of the approach. TEE, transesophageal echocardiography; t-SNE, t-Distributed Stochastic Neighbor Embedding; CSMC, Cedars Sinai Medical Center; SUMC, Stanford University Medical Center.

Figure 4

Micro-averaged receiver operating characteristic curves for model predictions in subsets containing all color flow Doppler videos versus no color flow Doppler videos. This evaluation was performed using a combination of the internal and external test sets due to the low prevalence of color flow Doppler videos in our data sets.

Figure 5

Representation of variation contained within classified views. Six of the eight TEE view class labels represent pooled or generalized categories, reflecting the high degree of anatomical and visual overlap that occurs among the twenty-eight standardized TEE views recommended by the American Society of Echocardiography and the Society of Cardiovascular Anesthesiologists. The images are 2-dimensional still frames sampled from the video data used in model training. (A) The “ME 2-Chamber View” class included ME 2-chamber (left) and ME mitral commissural (right) videos. (B) The “ME 4-Chamber View” class included ME 4-chamber (left) and ME 5-chamber (right) videos. (C) The “ME AV SAX View” class included ME AV SAX (left) and ME RV inflow-outflow (right) videos. (D) The “ME Long Axis View” class included ME long axis (left) and ME AV long axis (right) videos. (E) The “TG LV SAX View” class included short axis videos of the LV at all levels, such as the mid-papillary (left) and the basal (right) levels. (F) The “Aortic View” class included videos of the aorta at all levels, such as the descending thoracic SAX (left) and upper esophageal aortic arch LAX (right) levels. TEE, transesophageal echocardiography; ME, mid-esophageal; AV, aortic valve; SAX, short axis; RV, right ventricular; LV, left ventricular; TG, trans-gastric.