The Usefulness of Procedure-Based Assessments in Postgraduate Surgical Training Within the Intercollegiate Surgical Curriculum Programme; A Scoping Review

Abstract

Background: Workplace-based assessments are a mandatory component of postgraduate surgical training within the United Kingdom and Ireland. Procedure-based assessments (PBAs) and direct observation of practical skills (DOPS) are integrated within the Intercollegiate Surgical Curriculum Programme online platform and aim to assess trainees' performance in practical surgical skills; no reviews have previously investigated their educational usefulness in postgraduate surgical training. Usefulness was defined by the 5 criteria detailed by Van der Vleuten for determining the usefulness of educational assessment tools: validity; reliability; acceptability to learners and faculty; impact on future learning and practice; and costs (to the individual trainee, the institution, and society at large).

Methods: Scoping review methodology was used to examine the educational usefulness of PBA and DOPS assessments in postgraduate surgical training. A literature search of Ovid MEDLINE, EMBASE, and Web of Science databases was undertaken. The preferred reporting items for systematic reviews and meta-analyses standards for systematic reviews were followed.

Results/discussion: Ten studies met the inclusion criteria, with 1368 trainees and trainers included. A variety of study methodologies were identified. Although there is some evidence for the validity and reliability of both PBA and DOPS assessments, further work is required in both these domains including on the number of assessments required to ensure satisfactory reliability. This is a research priority, especially if these assessments become a component of summative assessment of trainee competency. The literature indicates that these assessments are generally acceptable to learners and faculty but their acceptability is negatively impacted upon by uncertainty over whether these assessments constitute a formative or summative assessment of trainees. With regards to costs, correct use of PBAs does require allocated time and resources to ensure their correct use and this must be factored into trainer and trainee job plans.

Conclusions: PBA and DOPS assessments are educationally useful tools in postgraduate surgical training. Further research is required to determine the number of assessments required to ensure adequate reliability. To ensure the educational benefits of these assessments are not diminished, clarification from postgraduate training schemes is required regarding whether these assessments an assessment for learning or an assessment of learning.

Keywords: direct observation of practical skills; procedure-based assessment; scoping review; surgical education.