Streamed optical flow adaptation from synthetic to real dental scenes

Abstract

Optical flow is integral to numerous video processing tasks such as stabilization. Although recent advancements in optical flow estimation have shown significant efficacy in general scenes, their applicability to challenging medical scenarios, which exhibit unique domain-specific visual phenomena, remains limited. Supervised learning methods facilitate the robust training of motion estimators. However, the absence of ground truth optical flow in many medical video-assisted applications poses a significant barrier to their progress. This is particularly evident in Video-Assisted Dentistry (VAD), where enhanced and continual vision could improve the educational, training, and fully operational dental workflows. Addressing this gap, we propose a new method for optical flow adaptation on real videos aided by a large dataset of synthetic dental videos with corresponding ground truth optical flows. Notably, the method employs a novel blind evaluation measure for unsupervised learning and incrementally fine-tunes optical flow neural networks. It facilitates their transition from synthetic to real dynamic dental scenes characterized by variable illumination, specular reflections, fluid dynamics, and sparse textures. In effect, this study demonstrates the effective streamed adaptation of multiple state-of-the-art optical flow networks to real medical scenes of video-assisted conservative dental treatments. Our code is available at https://github.com/camalab-ai/sofa-flow.