Profiling and Predicting Faculty Assessment Behavior in Surgical Education

Abstract

Objective: To study and profile the digital assessment behaviors of surgical faculty and residents, and to build a classifier to predict assessment completion, enhancing formative feedback initiatives.

Background: As competency-based paradigms are integrated into surgical training, developing digital education tools for measuring competency and providing rapid feedback is crucial. Simply making assessments available is inadequate and results in disappointingly low user participation. To optimize engagement and efficacy of these tools, user assessment behaviors need to be studied.

Methods: User data was aggregated from a HIPAA-compliant electronic health record (EHR)-integrated medical education platform. Faculty and resident behaviors were analyzed with respect to factors, such as time, day, device type, automated reminders, and EHR integration.

Results: 10,729 assessments were completed by 254 attendings for 428 residents across 22 institutions, from 2022 to 2024. 86% of assessments were completed by faculty on weekdays, were significantly influenced by automated platform triggers and EHR integration, and distinct faculty behavior profiles contingent upon time to completion and comment length were established. Residents opened assessments at a median of 1.5 hours of faculty assessment completion, with 96% of assessments viewed by 24 hours.

Conclusions: Faculty assessment behaviors represent an actionable bottleneck which can be leveraged to optimize and tailor the design of digital education tools, to enhance formative feedback.

Keywords: assessment; competency-based medical education; faculty behaviors; surgical education.