Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2025 May 1;100(5):614-620.
doi: 10.1097/ACM.0000000000005985. Epub 2025 Jan 31.

"But Why?": Explanatory Feedback Is a Reliable Marker of High-Quality Narrative Assessment of Surgical Performance

Rachel Stork Poeppelman, Junsang Cho, Kristine Nachbor, Tejas C Sekhar, Jack Pruett, Adam Baim, Sasha Strul, Alex Barsam, Benjamin Langworthy, Evan L Waxman, Susan M Culican

"But Why?": Explanatory Feedback Is a Reliable Marker of High-Quality Narrative Assessment of Surgical Performance

Rachel Stork Poeppelman et al. Acad Med. .

Abstract

Purpose: This study examines the quality of short narrative comments collected using an online workplace-based assessment (WBA) tool.

Method: The quality of comments collected by a WBA tool at the UPMC Ophthalmology Residency Training Program was evaluated between July 2017-June 2020. A randomized rating exercise involving 10 meta-raters from 6 institutions was performed to evaluate the value of narratives from deidentified WBAs. The tool captured a single-item entrustment competency question with brief comments. Comments were evaluated using a Quality of Assessment of Learning (QuAL) score (range, 0-5; ≥3 considered high quality) and on whether the assessor provided a feedback rationale.

Results: A total of 838 unique WBAs were collected from 15 attending evaluators. Comments were brief (median [interquartile range] length, 11 [7-17] words), yet 514 (61.3%) were rated as high quality (QuAL score ≥3). Of all 838 comments, 98 (11.7%) included a specific reason the evidence or suggestion was provided to the learner. Of these 98 comments, 94 (95.9%) met the high-quality feedback threshold. A higher QuAL score was associated with a higher postgraduate year (PGY) level (estimate [SE], 1.603 [0.428], P < .001 for PGY2 [reference]; 1.003 [0.389], P = .01 for PGY3; 1.079 [0.360], P = .003 for PGY4), suggesting more advanced learners receive higher-quality narrative comments. A correlation was found between a higher entrustment rating and a lower QuAL score (estimate [SE], -0.199 [0.053], P < .001). When the PGY level was controlled for, this association got stronger (estimate [SE], -0.310 [0.057], P < .001).

Conclusions: Analysis of WBA comments from attending physicians evaluated using the QuAL score demonstrated that most comments were high quality despite their brevity. Residents in later training years and with lower entrustment ratings received higher-quality comments. High-quality narrative assessments were longer and addressed rationale as part of the comment.

PubMed Disclaimer

References

    1. Harris P, Snell L, Talbot M, Harden RM. Competency-based medical education: Implications for undergraduate programs. Med Teach. 2010;32(8):646–650. - PubMed
    1. Holmboe ES, Sherbino J, Long DM, Swing SR, Frank JR. The role of assessment in competency-based medical education. Med Teach. 2010;32(8):676–682. - PubMed
    1. Swanwick T, Chana N. Workplace-based assessment. Br J Hosp Med. 2009;70(5):290–293. - PubMed
    1. Anderson HL, Kurtz J, West DC. Implementation and use of workplace-based assessment in clinical learning environments: A scoping review. Acad Med. 2021;96(11S):S164–S174. - PubMed
    1. Hawkins RE, Welcher CM, Holmboe ES, et al. Implementation of competency-based medical education: Are we addressing the concerns and challenges? Med Educ. 2015;49(11):1086–1102. - PubMed

Publication types

LinkOut - more resources