Eye of the beholder: Risk calculators and barriers to adoption in surgical trainees

Abstract

Background: Accurate risk assessment before surgery is complex and hampered by behavioral factors. Underutilized risk-based decision-support tools may counteract these barriers. The purpose of this study was to identify perceptions of and barriers to the use of surgical risk-assessment tools and assess the importance of data framing as a barrier to adoption in surgical trainees.

Methods: We distributed a survey and risk assessment activity to surgical trainees at four training institutions. The primary outcomes of this study were descriptive risk assessment practices currently performed by residents, identifiable influences and obstacles to adoption, and the variability of preference sets when comparing modified System Usability Scores of a current risk calculator to a purpose-built calculator revision. Risk calculator comparison responses were compared with simple and multivariable regression to identify predictors for preferentiality.

Results: We collected responses from 124 surgical residents (39% response rate). Participants endorsed familiarity with direct verbal communication (100%), sketch diagrams (87%), and brochures (59%). The most contemporary risk communication frameworks, such as best-worst case scenario framing (38%), case-specific risk calculators (43%), and all-procedure calculators (52%) were the least familiar. Usage favored traditional models of communication with only 26% of residents regularly using a strategy other than direct verbal discussion or anatomic sketch diagrams. Barriers limiting routine use included lack of electronic and clinical workflow integration. The mean modified System Usability Scores domain scores were widely dispersed for all domains, and no domain demonstrated one calculator's superiority over another.

Conclusion: Risk assessment tools are underutilized by trainees. Of importance, preference sets of clinicians appear to be unpredictable and may benefit more from a customizable, bespoke approach.

Fig. 1.

Comparative proportion of respondents indicating familiarity, favorability, and utilization. Six queried risk-assessment and counseling strategies are listed along the x-axis of the bar graph. The y-axis represents the proportion of residents in agreement for each question. Familiarity (“Yes,” black), favorability (“Like” or “Strongly Like,” white), and utilization (“Use More than Half the Time” or “Almost always use,” gray) are plotted side by side for comparison.

Fig. 2.

Scatter plot of each modified System Usability Score domain comparing the NSQIP RC to an alternative risk calculator. Each pole of the mSUS scores represents a total preference for each respective calculator, with the midpoint of the spectrum representing total indifference for the two (identification of poles intentionally masked). Mean ( orange ), −2 standard deviations ( red ) , and +2 standard deviations ( green ) scores are marked for each domain. Domain scores were averaged across a low-complexity and a high-complexity vignette.