Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2023 May 1;158(5):515-521.
doi: 10.1001/jamasurg.2023.0014.

Model-Based Operative Performance Expectations for Quantifying Competency in General Surgery

Andrew E Krumm  1   2   3 , Kayla Marcotte  1   3 , Brian C George  1   3   4
Affiliations

Model-Based Operative Performance Expectations for Quantifying Competency in General Surgery

Andrew E Krumm et al. JAMA Surg. .

Abstract

Importance: Understanding how to translate workplace-based assessment (WBA) ratings into metrics that communicate the ability of a surgeon to perform a procedure would represent a critical advancement in graduate medical education.

Objective: To evaluate the association between past and future performance in a comprehensive assessment system for the purpose of assessing point-in-time competence among general surgery trainees.

Design, setting, and participants: This case series included WBA ratings from September 2015 to September 2021 from the WBA system of the Society for Improving Medical Professional Learning (SIMPL) for all general surgery residents who were provided a rating following an operative performance across 70 programs in the US. The study included ratings for 2605 trainees from 1884 attending surgeon raters. Analyses were conducted between September 2021 and December 2021 using bayesian generalized linear mixed-effects models and marginal predicted probabilities.

Exposures: Longitudinal SIMPL ratings.

Main outcomes and measures: Performance expectations for 193 unique general surgery procedures based on an individual trainee's prior successful ratings for a procedure, clinical year of training, and month of the academic year.

Results: Using 63 248 SIMPL ratings, the association between prior and future performance was positive (β, 0.13; 95% credible interval [CrI], 0.12-0.15). The largest source of variation was postgraduate year (α, 3.15; 95% CrI, 1.66-6.03), with rater (α, 1.69; 95% CrI, 1.60-1.78), procedure (α, 1.35; 95% CrI, 1.22-1.51), case complexity (α, 1.30; 95% CrI, 0.42-3.66), and trainee (α, 0.99; 95% CrI, 0.94-1.04) accounting for significant variation in practice ready ratings. After marginalizing overcomplexity and trainee and holding rater constant, mean predicted probabilities had strong overall discrimination (area under the receiver operating characteristic curve, 0.81) and were well calibrated.

Conclusions and relevance: In this study, prior performance was associated with future performance. This association, combined with an overall modeling strategy that accounted for various facets of an assessment task, may offer a strategy for quantifying competence as performance expectations.

PubMed Disclaimer

Conflict of interest statement

Conflict of Interest Disclosures: Dr Krumm reported internal funding from the Research, Innovation, Scholarship, and Education Fellowship Program at the University of Michigan Medical School. Dr George reported serving without compensation as the Executive Director of the Society for Improving Medical Professional Learning. No other disclosures were reported.

Figures

Figure.
Figure.. Probability of Practice Readiness for 3 Procedures, by Postgraduate Year (PGY)
See this image and copyright information in PMC

Comment in

References

    1. Caverzagie KJ, Nousiainen MT, Ferguson PC, et al. ; ICBME Collaborators . Overarching challenges to the implementation of competency-based medical education. Med Teach. 2017;39(6):588-593. doi:10.1080/0142159X.2017.1315075 - DOI - PubMed
    1. Norcini J, Burch V. Workplace-based assessment as an educational tool: AMEE guide no. 31. Med Teach. 2007;29(9):855-871. doi:10.1080/01421590701775453 - DOI - PubMed
    1. Mislevy RJ. Validity by design. Educ Res. 2007;36(8):463-469. doi:10.3102/0013189X07311660 - DOI
    1. Santen SA, Ryan M, Helou MA, et al. . Building reliable and generalizable clerkship competency assessments: Impact of ‘hawk-dove’ correction. Med Teach. 2021;43(12):1374-1380. doi:10.1080/0142159X.2021.1948519 - DOI - PubMed
    1. Van der Vleuten CPM, Norman GR, De Graaff E. Pitfalls in the pursuit of objectivity: issues of reliability. Med Educ. 1991;25(2):110-118. doi:10.1111/j.1365-2923.1991.tb00036.x - DOI - PubMed