Vision-based markerless registration using stereo vision and an augmented reality surgical navigation system: a pilot study

Abstract

Background: This study evaluated the use of an augmented reality navigation system that provides a markerless registration system using stereo vision in oral and maxillofacial surgery.

Method: A feasibility study was performed on a subject, wherein a stereo camera was used for tracking and markerless registration. The computed tomography data obtained from the volunteer was used to create an integral videography image and a 3-dimensional rapid prototype model of the jaw. The overlay of the subject's anatomic site and its 3D-IV image were displayed in real space using a 3D-AR display. Extraction of characteristic points and teeth matching were done using parallax images from two stereo cameras for patient-image registration.

Results: Accurate registration of the volunteer's anatomy with IV stereoscopic images via image matching was done using the fully automated markerless system, which recognized the incisal edges of the teeth and captured information pertaining to their position with an average target registration error of < 1 mm. These 3D-CT images were then displayed in real space with high accuracy using AR. Even when the viewing position was changed, the 3D images could be observed as if they were floating in real space without using special glasses.

Conclusion: Teeth were successfully used for registration via 3D image (contour) matching. This system, without using references or fiducial markers, displayed 3D-CT images in real space with high accuracy. The system provided real-time markerless registration and 3D image matching via stereo vision, which, combined with AR, could have significant clinical applications.

Fig. 1

The physical setup of the system. a The configuration of the markerless surgical navigation system based on stereo vision and augmented reality, and b a 3D rapid prototyping model

Fig. 2

Calibration of the integral videography (IV) display. This includes a five feature points for calibration, b The IV image displayed in real space, and c recognition results for the calibrated IV images (matching of left and right images via stereo vision)

Fig. 3

Automatic registration of the 3D-CT image and volunteer’s position. This included a extraction of characteristic points, b automatic detection of teeth contour and matching of right and left images via stereo vision, and c 3D contour reconstruction in the stereo camera frame

Fig. 4

The IV images are overlaid on the surgical site. This included the a mandible, b maxilla overlaid on the surgical site, and c visualization of the mandibular canal, tooth root, and impacted third molar

Fig. 5

Positional errors along different axes (x, y, and z)