Automatic Hyoid Bone Tracking in Real-Time Ultrasound Swallowing Videos Using Deep Learning Based and Correlation Filter Based Trackers

Abstract

(1) Background: Ultrasound provides a radiation-free and portable method for assessing swallowing. Hyoid bone locations and displacements are often used as important indicators for the evaluation of swallowing disorders. However, this requires clinicians to spend a great deal of time reviewing the ultrasound images. (2) Methods: In this study, we applied tracking algorithms based on deep learning and correlation filters to detect hyoid locations in ultrasound videos collected during swallowing. Fifty videos were collected from 10 young, healthy subjects for training, evaluation, and testing of the trackers. (3) Results: The best performing deep learning algorithm, Fully-Convolutional Siamese Networks (SiamFC), proved to have reliable performance in getting accurate hyoid bone locations from each frame of the swallowing ultrasound videos. While having a real-time frame rate (175 fps) when running on an RTX 2060, SiamFC also achieved a precision of 98.9% at the threshold of 10 pixels (3.25 mm) and 80.5% at the threshold of 5 pixels (1.63 mm). The tracker's root-mean-square error and average error were 3.9 pixels (1.27 mm) and 3.3 pixels (1.07 mm), respectively. (4) Conclusions: Our results pave the way for real-time automatic tracking of the hyoid bone in ultrasound videos for swallowing assessment.

Keywords: SiamFC; correlation filters; deep learning; dysphagia; hyoid bone; real-time; swallowing; tracking; ultrasound videos.

Figure 1

The left side of the figure shows an example ultrasound image with labeled anatomical structures, as illustrated on the right side. The hyoid bone annotation point was placed at the intersection of the geniohyoid muscle (left) and the acoustic shadow (above). During inference, a bounding box tracked the hyoid bone location.

Figure 2

Workflow of Siamese trackers. The center of the yellow bounding box indicates the hyoid location in the last frame; the green box and orange box centered on the last frame hyoid location will crop exemplar patch and search patch on the previous frame and current frame, respectively. The green point indicates the peak response on the score map, while the yellow one denotes the center location of the current frame.

Figure 3

Workflow of multi-stage trackers. Precise region of interests (PrRoI) pooling layers [35] can convert features of different sizes into the same size while enabling the computation of the gradient of Intersection over Union (IoU) with respect to the bounding box coordinates. IoU Net outputs IoU scores for each proposal, and the top three ranked proposals are averaged to produce a robust prediction bounding box location.

Figure 4

Workflow of correlation filter trackers.

Figure 5

Hyoid center trace plots between the timestamps of hyoid onset (the frame when the hyoid starts to move) to offset (at the moment when the hyoid starts to move away from its maximum position of superior-anterior movement) in 2D Cartesian axis (a), polar axis (b), x-axis (c), and y-axis (d). From an example test sequence of 10 mL thin liquid swallow, female subject, from hyoid onset to offset. The blue line represents the ground truth of the hyoid path, and the yellow one represents the inference path. Length unit at the polar axis is in pixels.

Figure 6

Precision plot shows the mean distance precision of 10 test sequences in full-length at different location error thresholds. The legend shows the precisions of different trackers at the threshold of 10 pixels.

Figure 7

Performance plot. Center error of all frames from a test sequence in full-length where the female swallows 10 mL of paste liquid. The y-axis is a center error in pixel and the x-axis is frame number. Three example images from every 200 frames were chosen and displayed above the plot. The pink dot represents the center of inference, and the blue dot represents the center of ground truth (annotation).

Figure 8

A frame in which two acoustic shadows of hyoid bone were seen due to fast hyoid movement speed. Ground truth location (blue dot) and prediction (pink dot with pink bounding box).