Semiautomatic marker tracking of tongue positions captured by videofluoroscopy during primate feeding

Abstract

Videofluoroscopy (VF) is one of the most commonly used tools to assess oropharyngeal dysphagia as well as to visualize musculoskeletal structures of humans and animals engaged in various behaviors, including feeding. Despite its importance in clinical and scientific use, processing VF data has historically been extremely tedious because it is performed using manual frame-by-frame methods. With recent technological advances, the frame rate for scientific use has been increasing along with the use of high speed data capture systems. In the current study, we used non-human primates as a model animal to study human feeding behaviors to capture tongue movement based on markers implanted into the tongue. Here, we introduce a semi-automatic marker tracking algorithm that yields high tracking accuracy (> 90%) and dramatic speed improvements (faster than real time labeling). Furthermore, we quantify the sources of tracking errors and the tracking performance as a function of marker speeds. Our results indicate that there is more room for methodological improvements both in detection and prediction of marker positions. Moreover, correspondingly faster frame rates will be required to capture faster kinematic behaviors such as those of mice, which are extensively used to study both control and pathological conditions.

Fig. 1. Overview of tongue tracking algorithm

A. An exemplar frame of fluoroscopy data. Note the three opaque markers implanted in the tongue in anterior, middle, and posterior locations. B. The SURF algorithm was applied to the fluoroscopy image to identify interest points that putatively correspond to tongue markers. The 25 most salient visual features identified by the algorithm are shown as green circles with crosses in their center. Many salient features identified were of jaw screws and other markers that were not implanted on the tongue. Each of the three tongue markers were detected. C. A Kalman filter was used to make predictions about the spatial location of the tongue markers based on their location, velocity, and acceleration in previous frames. The predicted location of each tongue marker is indicated by a blue symbol (×). A dark-to-light color gradient indicates the anterior-to-posterior axis of the tongue. D. A matching algorithm was used to assign interest points (B) to the predicted marker locations (C). The interest point that was identified as a tongue marker is shown by a blue circle whose color corresponds to the predictions in C. Each tongue marker was successfully identified by the algorithm.

Fig. 2. Two types of tracking error

A. A representative frame containing a detection error of the anterior tongue marker. The actual location of the tongue marker based on labeling by a trained researcher is indicated by the yellow circle (all other conventions same as Fig. 1). Note that the Kalman filter predicted the marker would be very close to its actual location. B. A representative frame showing a prediction error of the anterior tongue marker. Here, the predicted location of the tongue marker (dark blue ×) was far from its actual location (yellow circle). Instead, a different interest point was incorrectly identified as the tongue marker.

Fig. 3. Tracking outcomes

We correctly identified the location of the tongue marker on over 90% of frames (top). On frames that contained an error, we quantified the proportion that were due to detection errors and prediction errors. For the anterior and middle tongue markers, detection and prediction errors occurred in roughly equal proportion. For the posterior tongue marker, detection errors comprised nearly all of the total errors.

Fig. 4. Tracking performance as a function of tongue speed

We found that both detection and prediction errors were more likely to occur as the tongue moved faster. Color scheme is the same as Fig. 3.

Fig. 5. Improvement in tracking speed

We compared the amount of time it took a human to label the tongue tracking data using a manual procedure with our semi-automatic tracking procedure. We found that algorithmic tracking of tongue position was nearly 500% faster than manual labeling.