Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 Jul 1;7(7):e2422520.
doi: 10.1001/jamanetworkopen.2024.22520.

Deep Learning Analysis of Surgical Video Recordings to Assess Nontechnical Skills

Rayan Ebnali Harari  1 , Roger D Dias  1 , Lauren R Kennedy-Metz  2 , Giovanna Varni  3 , Matthew Gombolay  4 , Steven Yule  5 , Eduardo Salas  6 , Marco A Zenati  1   7
Affiliations

Deep Learning Analysis of Surgical Video Recordings to Assess Nontechnical Skills

Rayan Ebnali Harari et al. JAMA Netw Open. .

Abstract

Importance: Assessing nontechnical skills in operating rooms (ORs) is crucial for enhancing surgical performance and patient safety. However, automated and real-time evaluation of these skills remains challenging.

Objective: To explore the feasibility of using motion features extracted from surgical video recordings to automatically assess nontechnical skills during cardiac surgical procedures.

Design, setting, and participants: This cross-sectional study used video recordings of cardiac surgical procedures at a tertiary academic US hospital collected from January 2021 through May 2022. The OpenPose library was used to analyze videos to extract body pose estimations of team members and compute various team motion features. The Non-Technical Skills for Surgeons (NOTSS) assessment tool was employed for rating the OR team's nontechnical skills by 3 expert raters.

Main outcomes and measures: NOTSS overall score, with motion features extracted from surgical videos as measures.

Results: A total of 30 complete cardiac surgery procedures were included: 26 (86.6%) were on-pump coronary artery bypass graft procedures and 4 (13.4%) were aortic valve replacement or repair procedures. All patients were male, and the mean (SD) age was 72 (6.3) years. All surgical teams were composed of 4 key roles (attending surgeon, attending anesthesiologist, primary perfusionist, and scrub nurse) with additional supporting roles. NOTSS scores correlated significantly with trajectory (r = 0.51, P = .005), acceleration (r = 0.48, P = .008), and entropy (r = -0.52, P = .004) of team displacement. Multiple linear regression, adjusted for patient factors, showed average team trajectory (adjusted R2 = 0.335; coefficient, 10.51 [95% CI, 8.81-12.21]; P = .004) and team displacement entropy (adjusted R2 = 0.304; coefficient, -12.64 [95% CI, -20.54 to -4.74]; P = .003) were associated with NOTSS scores.

Conclusions and relevance: This study suggests a significant link between OR team movements and nontechnical skills ratings by NOTSS during cardiac surgical procedures, suggesting automated surgical video analysis could enhance nontechnical skills assessment. Further investigation across different hospitals and specialties is necessary to validate these findings.

PubMed Disclaimer

Conflict of interest statement

Conflict of Interest Disclosures: Dr Gombolay reported grants from the National Institutes of Health during the conduct of the study; he reported receiving grants from the National Science Foundation (NSF), Office of Naval Research, NASA, The McCamish Foundation, MIT Lincoln Laboratory, Ford Motor Company, Konica Minolta, Computing Research Association, Apple, Sandia National Laboratories, Naval Research Laboratory, and Lockheed Martin Corporation outside the submitted work; he reported receiving honoraria and/or consulting fees from Johns Hopkins University Applied Physics Laboratory, NASA, Washington Injury Lawyers PLLC, KTH Royal Institute of Technology, Naval Research Lab, and NSF outside the submitted work. Dr Yule reported receiving personal fees from Johnson & Johnson Institute outside the submitted work. No other disclosures were reported.

Figures

Figure 1.
Figure 1.. Methodology Overview for Operating Room (OR) Video Recording and Keypoint Processing
Figure 2.
Figure 2.. Pipeline of Surgical Team Analysis
Generation of position keypoints with OpenPose and data processing are described in Figure 1. Aggregating data from all frames created a final file containing the x and y locations of keypoints of each OR team member in each time frame. According to our previous studies, we suggest using only the neck keypoints because changing the coordination of this keypoint resulted in higher accuracy in capturing the whole-body movements of a person. After dealing with outliers and missing data, the x and y coordinates of the neck keypoint were used to calculate the Euclidean distance between current and previous neck positions across consecutive frames. The average displacement per frame in pixels then was calculated across all team members to capture the entire team motion. Team displacement data measured were analyzed using time-domain and nonlinear methods. AVR indicates aortic valve replacement or repair; CABG, coronary artery bypass graft; NOTSS, Non-Technical Skills for Surgeons; OR, operating room.
Figure 3.
Figure 3.. Movement Visualization of Surgical Team Members
Figure 4.
Figure 4.. Correlation Analyses Between Total NOTSS Scores and Team Motion Features
See this image and copyright information in PMC

References

    1. Sinyard RD, Rentas CM, Gunn EG, et al. . Managing a team in the operating room: the science of teamwork and non-technical skills for surgeons. Curr Probl Surg. 2022;59(7):101172. doi:10.1016/j.cpsurg.2022.101172 - DOI - PubMed
    1. Savoldelli GL, Brindley PG, Jaffrelot M, Cardinal P. Crisis Resource Management and Nontechnical Skills: From Individuals to Teams, From Danger to Safety. In: Chiniara G, ed. Clinical Simulation. 2nd ed. Academic Press; 2019:347-372. doi:10.1016/B978-0-12-815657-5.00025-5 - DOI
    1. Uramatsu M, Fujisawa Y, Mizuno S, Souma T, Komatsubara A, Miki T. Do failures in non-technical skills contribute to fatal medical accidents in Japan? A review of the 2010-2013 national accident reports. BMJ Open. 2017;7(2):e013678. doi:10.1136/bmjopen-2016-013678 - DOI - PMC - PubMed
    1. Leuschner S, Leuschner M, Kropf S, Niederbichler AD. Non-technical skills training in the operating theatre: a meta-analysis of patient outcomes. Surgeon. 2019;17(4):233-243. doi:10.1016/j.surge.2018.07.001 - DOI - PubMed
    1. Sharma B, Mishra A, Aggarwal R, Grantcharov TP. Non-technical skills assessment in surgery. Surg Oncol. 2011;20(3):169-177. doi:10.1016/j.suronc.2010.10.001 - DOI - PubMed

Publication types