Deep Reinforcement Learning Based System for Intraoperative Hyperspectral Video Autofocusing

Abstract

Hyperspectral imaging (HSI) captures a greater level of spectral detail than traditional optical imaging, making it a potentially valuable intraoperative tool when precise tissue differentiation is essential. Hardware limitations of current optical systems used for handheld realtime video HSI result in a limited focal depth, thereby posing usability issues for integration of the technology into the operating room. This work integrates a focus-tunable liquid lens into a video HSI exoscope, and proposes novel video autofocusing methods based on deep reinforcement learning. A first-of-its-kind robotic focal-time scan was performed to create a realistic and reproducible testing dataset. We benchmarked our proposed autofocus algorithm against traditional policies, and found our novel approach to perform significantly (p < 0.05) better than traditional techniques (0.070 ±.098 mean absolute focal error compared to 0.146 ±.148). In addition, we performed a blinded usability trial by having two neurosurgeons compare the system with different autofocus policies, and found our novel approach to be the most favourable, making our system a desirable addition for intraoperative HSI.

Keywords: Autofocus; Computer Assisted Intervention; Deep Reinforcement Learning; Hyperspectral Imaging.

Fig. 1

Left) Existing fixed-focus HSI system being used during neurosurgery in an ethically approved study. Right) RGB reconstruction of an image taken with the fixed-focus HSI system following a craniotomy. The focus has been manually adjusted for the cavity visible through the craniotomy (circled).

Fig. 2

Schematic diagram of our intraoperative video HSI system with focustunable liquid lens, allowing electrically controllable focal length. The handheld portion of the system is shown in the dashed line box.

Fig. 3

Left) Robotic arm holding our optical system imaging a brain phantom. Right) Sample from our robotic focal-time scan, with the columns representing sequential focal stacks sampled at focal powers of 0.2, 0.5 and 0.8 (top to bottom). For low focal powers, the focal plane is behind the phantom (upper row). As the focal power increases, the focal plane intersects with the fissure (middle row), and then with the area surrounding the fissure (bottom row).

Fig. 4

Focal path (top) and error in focal power (bottom) for three autofocus policies on the robotic focal-time scan. The optimal focal power is shown in black. All paths have been smoothed with a moving average with a window of 5 frames for visualisation purposes.