Towards Autonomous Retinal Microsurgery Using RGB-D Images

Abstract

Retinal surgery is a challenging procedure requiring precise manipulation of the fragile retinal tissue, often at the scale of tens-of-micrometers. Its difficulty has motivated the development of robotic assistance platforms to enable precise motion, and more recently, novel sensors such as microscope integrated optical coherence tomography (OCT) for RGB-D view of the surgical workspace. The combination of these devices opens new possibilities for robotic automation of tasks such as subretinal injection (SI), a procedure that involves precise needle insertion into the retina for targeted drug delivery. Motivated by this opportunity, we develop a framework for autonomous needle navigation during SI. We develop a system which enables the surgeon to specify waypoint goals in the microscope and OCT views, and the system autonomously navigates the needle to the desired subretinal space in real-time. Our system integrates OCT and microscope images with convolutional neural networks (CNNs) to automatically segment the surgical tool and retinal tissue boundaries, and model predictive control that generates optimal trajectories that respect kinematic constraints to ensure patient safety. We validate our system by demonstrating 30 successful SI trials on pig eyes. Preliminary comparisons to a human operator in robot-assisted mode highlight the enhanced safety and performance of our system.

Keywords: Computer Vision for Medical Robotics; Medical Robots and Systems; Vision-Based Navigation.

Fig. 1.

(A) Problem statement; Our imaging system provides a simultaneous view of the surgical tool and the B-scan image which dynamically tracks the surgical instrument (B) Experimental setup

Fig. 2.

High-level workflow; 1) the microscope image and the iOCT B-scan images are acquired. 2) Three CNNs provide needle-tip detections and retinal layer segmentations. The surgeon provides two waypoint goals in the microscope and B-scan images via two mouse-clicks. The waypoint goal is first specified in the microscope image, then in the B-scan image before insertion. 3) The relevant task and motion is planned, and an optimized trajectory is sent to the robot for trajectory-tracking.

Fig. 3.

Key variables used are shown. Double arrows are shown to indicate that the pixel goals $i_{g\_ILM}$ and $i_{g\_subretina}$ correspond to $p_{g\_ILM}$ and $p_{g\_subretina}$ respectively in euclidean space.

Fig. 4.

(A) Microscope-OCT system setup. FC, fiber collimator; GS, galvo scanners; DM, dichroic mirror; IL, imaging lens; OL, objective lens. (B) The mapping between the laser scanning position and the applied voltage.

Fig. 5.

Network architectures: (A) Two networks are trained to detect the needle tip and its base (thus defining its axis), one for microscope and another for iOCT images. (B) A third network is trained to the ILM and RPE layer segmentations

Fig. 6.

Experimental metrics are shown (see Section V): (A) navigation error on the retinal surface goal $p_{g\_ILM}$ (B) needle-insertion error at the drug delivery site $p_{g\_subretina}$ (C) RCM error (D) total duration of the surgery by task (E) 2D navigation error to the clicked goal from the microscope view during robot-assisted mode (F) qualitative comparison between autonomous and robot-assisted modes (G) same comparison from a close-up side view during needle insertion (H) comparing the deviation of the needle-tip from the insertion axis during needle insertion; ADE (average displacement error), FDE (final displacement error).