Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2022 Oct;17(10):1801-1811.
doi: 10.1007/s11548-022-02681-5. Epub 2022 May 30.

Video-based assessment of intraoperative surgical skill

Sanchit Hira #  1 , Digvijay Singh #  2 , Tae Soo Kim #  2   3 , Shobhit Gupta  4 , Gregory Hager  1   2   3 , Shameema Sikder  1   3   5 , S Swaroop Vedula  6
Affiliations

Video-based assessment of intraoperative surgical skill

Sanchit Hira et al. Int J Comput Assist Radiol Surg. 2022 Oct.

Abstract

Purpose: Surgeons' skill in the operating room is a major determinant of patient outcomes. Assessment of surgeons' skill is necessary to improve patient outcomes and quality of care through surgical training and coaching. Methods for video-based assessment of surgical skill can provide objective and efficient tools for surgeons. Our work introduces a new method based on attention mechanisms and provides a comprehensive comparative analysis of state-of-the-art methods for video-based assessment of surgical skill in the operating room.

Methods: Using a dataset of 99 videos of capsulorhexis, a critical step in cataract surgery, we evaluated image feature-based methods and two deep learning methods to assess skill using RGB videos. In the first method, we predict instrument tips as keypoints and predict surgical skill using temporal convolutional neural networks. In the second method, we propose a frame-wise encoder (2D convolutional neural network) followed by a temporal model (recurrent neural network), both of which are augmented by visual attention mechanisms. We computed the area under the receiver operating characteristic curve (AUC), sensitivity, specificity, and predictive values through fivefold cross-validation.

Results: To classify a binary skill label (expert vs. novice), the range of AUC estimates was 0.49 (95% confidence interval; CI = 0.37 to 0.60) to 0.76 (95% CI = 0.66 to 0.85) for image feature-based methods. The sensitivity and specificity were consistently high for none of the methods. For the deep learning methods, the AUC was 0.79 (95% CI = 0.70 to 0.88) using keypoints alone, 0.78 (95% CI = 0.69 to 0.88) and 0.75 (95% CI = 0.65 to 0.85) with and without attention mechanisms, respectively.

Conclusion: Deep learning methods are necessary for video-based assessment of surgical skill in the operating room. Attention mechanisms improved discrimination ability of the network. Our findings should be evaluated for external validity in other datasets.

Keywords: Cataract surgery; Deep learning; Surgical skill; Video-based assessment.

PubMed Disclaimer

Conflict of interest statement

• Competing Interests: None.

Figures

Fig. 1
Fig. 1
ROC plots for interest-point based methods. Numbers on plots are AUC and 95% confidence intervals.
Fig. 2
Fig. 2
Predictive performance of interest-point based methods.
Fig. 3
Fig. 3
ROC plots for deep learning methods. Numbers on plots are AUC and 95% confidence intervals.
Fig. 4
Fig. 4
Predictive performance of deep learning methods.
See this image and copyright information in PMC

References

    1. Aghdasi Nava, Bly Randall, W White Lee, Hannaford Blake, Moe Kris, and S Lendvay Thomas. Crowd-sourced assessment of surgical skills in cricothyrotomy procedure. Journal of surgical research, 196(2):302–306, 2015. - PMC - PubMed
    1. Agresti Alan. Categorical data analysis, volume 482. John Wiley & Sons, 2003.
    1. [] Bahdanau Dzmitry, Cho Kyunghyun, and Bengio Yoshua. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, 2015.
    1. Bettadapura Vinay, Schindler Grant, Plötz Thomas, and Essa Irfan. Augmenting bag-of-words: Data-driven discovery of temporal and structural information for activity recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2619–2626, 2013.
    1. Birkmeyer John D, Finks Jonathan F, O’Reilly Amanda, Oerline Mary, Carlin Arthur M, Nunn Andre R, Dimick Justin, Banerjee Mousumi, and Birkmeyer Nancy JO. Surgical skill and complication rates after bariatric surgery. New England Journal of Medicine, 369(15):1434–1442, 2013. - PubMed

LinkOut - more resources