Combining Augmented Reality and 3D Printing to Improve Surgical Workflows in Orthopedic Oncology: Smartphone Application and Clinical Evaluation

Abstract

During the last decade, orthopedic oncology has experienced the benefits of computerized medical imaging to reduce human dependency, improving accuracy and clinical outcomes. However, traditional surgical navigation systems do not always adapt properly to this kind of interventions. Augmented reality (AR) and three-dimensional (3D) printing are technologies lately introduced in the surgical environment with promising results. Here we present an innovative solution combining 3D printing and AR in orthopedic oncological surgery. A new surgical workflow is proposed, including 3D printed models and a novel AR-based smartphone application (app). This app can display the patient's anatomy and the tumor's location. A 3D-printed reference marker, designed to fit in a unique position of the affected bone tissue, enables automatic registration. The system has been evaluated in terms of visualization accuracy and usability during the whole surgical workflow. Experiments on six realistic phantoms provided a visualization error below 3 mm. The AR system was tested in two clinical cases during surgical planning, patient communication, and surgical intervention. These results and the positive feedback obtained from surgeons and patients suggest that the combination of AR and 3D printing can improve efficacy, accuracy, and patients' experience.

Keywords: 3D printing; augmented reality; computer-aided interventions; orthopedic oncology; smartphone.

Figure 1

Prosed step by step orthopedics oncology medical workflow.

Figure 2

Virtual 3D models from patients: (a) AR3DP0002; (b) AR3DP0003; (c) AR3DP0004; (d) AR3DP0005, with some transparency in the bone to display the inner tumor; (e) AR3DP0006; (f) AR3DP0007. Tumors are represented in red, bones in white and surgical guides in green. Surgical cutting planes are illustrated in semi-transparent gray in the cases that required them: (a) and (c). The 3D-printed marker reference is positioned in the surgical guide.

Figure 3

3D-printed augmented reality cubic marker used in the augmented reality system.

Figure 4

ARHealth smartphone application. (a) Demo mode using patient AR3DP0005 3D models, tumor is represented in red inside the bone, which is displayed in white with transparency texture; (b) Virtual visualization (tumor in red, bone in white with transparency texture) overlaid on top of the 3D-printed bone fragment (solid white) of patient AR3DP0007 using Clinic mode; (c) Surgical mode visualization of patient AR3DP0004 (tumor is represented in blue and cutting planes in semi-transparent green). A green frame surrounding the AR marker indicates that the reference is being tracked by the system.

Figure 5

3D printed patient-specific phantoms from patient: (a) AR3DP0002; (b) AR3DP0003; (c) AR3DP0004; (d) AR3DP0005; (e) AR3DP0006; (f) AR3DP0007. Bones are in white, the tumors are in red, and the resin surgical guides are fitted on their corresponding position.

Figure 6

Phantom of patient AR3DP0002 (buttock tumor) and smartphone with the ARHealth validation app. The Augmented Reality Tracking Error validation spheres are augmented on the phantom surface in deep blue.

Figure 7

Surgical Guide Placement Error obtained for each patient phantom. The upper and lower limits for each box represent the first and third quartile of the dataset, and the middle line indicates the median. The whiskers stand for the highest and lowest values (±1.5 times the standard deviation).

Figure 8

Augmented Reality Tracking Error for all the patients separated by user. The upper and lower limits for each box represent the first and third quartile of the dataset, the middle line indicates the median. The whiskers stand for the highest and lowest values (±1.5 times the standard deviation).

Figure 9

Integration of the augmented reality system at each step of the medical workflow. (a,b) Physician using ARHealth during surgical planning of patient AR3DP0006; (c,d) Medical staff explaining patient AR3DP0007 her condition using ARHealth; (e) One physicians using ARHealth during surgical intervention of patient AR3DP0007 after the surgical guide was placed on the patient, and other surgeon delimiting surgical margin while looking at the AR-display. (b,d,f) Smartphone visualization at the same moment of (a,c,e), respectively.