Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 Aug;134(8):3548-3554.
doi: 10.1002/lary.31391. Epub 2024 Mar 12.

Correcting for Rater Effects in Operating Room Surgical Skills Assessment

Ryan Chou  1 , Hajira Naz  2 , Kofi D O Boahene  3   4 , Jessica H Maxwell  5   6 , John R Wanamaker  5   6 , Patrick J Byrne  7 , Ira D Papel  3   8 , Theda C Kontis  3   8 , Gregory D Hager  9   10 , Lisa E Ishii  3   4 , Sonya Malekzadeh  5   6 , S Swaroop Vedula  9 , Masaru Ishii  3
Affiliations

Correcting for Rater Effects in Operating Room Surgical Skills Assessment

Ryan Chou et al. Laryngoscope. 2024 Aug.

Abstract

Objective: To estimate and adjust for rater effects in operating room surgical skills assessment performed using a structured rating scale for nasal septoplasty.

Methods: We analyzed survey responses from attending surgeons (raters) who supervised residents and fellows (trainees) performing nasal septoplasty in a prospective cohort study. We fit a structural equation model with the rubric item scores regressed on a latent component of skill and then fit a second model including the rating surgeon as a random effect to model a rater-effects-adjusted latent surgical skill. We validated this model against conventional measures including the level of expertise and post-graduation year (PGY) commensurate with the trainee's performance, the actual PGY of the trainee, and whether the surgical goals were achieved.

Results: Our dataset included 188 assessments by 7 raters and 41 trainees. The model with one latent construct for surgical skill and the rater as a random effect was the best. Rubric scores depended on how severe or lenient the rater was, sometimes almost as much as they depended on trainee skill. Rater-adjusted latent skill scores increased with attending-estimated skill levels and PGY of trainees, increased with the actual PGY, and appeared constant over different levels of achievement of surgical goals.

Conclusion: Our work provides a method to obtain rater effect adjusted surgical skill assessments in the operating room using structured rating scales. Our method allows for the creation of standardized (i.e., rater-effects-adjusted) quantitative surgical skill benchmarks using national-level databases on trainee assessments.

Level of evidence: N/A Laryngoscope, 134:3548-3554, 2024.

Keywords: OSATS; SGAT; rater bias; rater effect; septoplasty; surgical skill assessment.

PubMed Disclaimer

Conflict of interest statement

This study was supported by funding from the National Institute of Dental & Craniofacial Research of the National Institutes of Health under award number R01DE025265, and a Provost Undergraduate Research Award from Johns Hopkins University. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Figures

Figure 1:
Figure 1:. One-Factor Rater-Adjusted Structural Equation Model
The structural equation model used to estimate a single latent skill score adjusted for rater effects by including the attending surgeon as a random effect. The scores are modeled as ordinal variables.
Figure 2:
Figure 2:. Scores by Raters Before and After Rater Adjustment
The distributions of latent skill scores calculated from the rater-assessed SGAT scores (a) before adjusting for rater effects and (b) after adjusting for rater effects. The seven raters are shown on the X-axis.
Figure 3:
Figure 3:. Estimated Scores Relative to Validation Factors
The distributions of estimated scores relative to (a) the attending-estimated total skill level of the trainee, (b) the attending-estimated post graduation year of the trainee, (c) the actual post graduation year of the trainee, and (d) the attending-assessed achievement of surgical goals.
See this image and copyright information in PMC

Similar articles

See all similar articles

References

    1. Stulberg JJ, Huang R, Kreutzer L, et al. Association between surgeon technical skills and patient outcomes. JAMA Surg. 2020;155(10):960–968. doi: 10.1001/jamasurg.2020.3007 - DOI - PMC - PubMed
    1. Vedula SS, Ishii M, Hager GD. Objective assessment of surgical technical skill and competency in the operating room. Annu Rev Biomed Eng. 2017;19:301–325. doi: 10.1146/annurev-bioeng-071516-044435 - DOI - PMC - PubMed
    1. Obeid AA, Al-Qahtani KH, Ashraf M, Alghamdi FR, Marglani O, Alherabi A. Development and testing for an operative competency assessment tool for nasal septoplasty surgery. Am J Rhinol Allergy. 2014;28(4):e163–7. doi: 10.2500/ajra.2014.28.4051 - DOI - PubMed
    1. Martin JA, Regehr G, Reznick R, et al. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg. 1997;84(2):273–278. doi: 10.1046/j.1365-2168.1997.02502.x - DOI - PubMed
    1. Martin J, Reznick R. Reliability and validity of an instrument to evaluate operative skill in surgical residents. Canadian Journal of Surgery. 1994.

LinkOut - more resources