Validity, reliability and support for implementation of independence-scaled procedural assessment in laparoscopic surgery

Abstract

Background: There is no widely used method to evaluate procedure-specific laparoscopic skills. The first aim of this study was to develop a procedure-based assessment method. The second aim was to compare its validity, reliability and feasibility with currently available global rating scales (GRSs).

Methods: An independence-scaled procedural assessment was created by linking the procedural key steps of the laparoscopic cholecystectomy to an independence scale. Subtitled and blinded videos of a novice, an intermediate and an almost competent trainee, were evaluated with GRSs (OSATS and GOALS) and the independence-scaled procedural assessment by seven surgeons, three senior trainees and six scrub nurses. Participants received a short introduction to the GRSs and independence-scaled procedural assessment before assessment. The validity was estimated with the Friedman and Wilcoxon test and the reliability with the intra-class correlation coefficient (ICC). A questionnaire was used to evaluate user opinion.

Results: Independence-scaled procedural assessment and GRS scores improved significantly with surgical experience (OSATS p = 0.001, GOALS p < 0.001, independence-scaled procedural assessment p < 0.001). The ICCs of the OSATS, GOALS and independence-scaled procedural assessment were 0.78, 0.74 and 0.84, respectively, among surgeons. The ICCs increased when the ratings of scrub nurses were added to those of the surgeons. The independence-scaled procedural assessment was not considered more of an administrative burden than the GRSs (p = 0.692).

Discussion/conclusion: A procedural assessment created by combining procedural key steps to an independence scale is a valid, reliable and acceptable assessment instrument in surgery. In contrast to the GRSs, the reliability of the independence-scaled procedural assessment exceeded the threshold of 0.8, indicating that it can also be used for summative assessment. It furthermore seems that scrub nurses can assess the operative competence of surgical trainees.

Keywords: GOALS; Global rating scale; Laparoscopic cholecystectomy; Laparoscopy; Minimal invasive surgery; OSATS; Procedure-based assessment; Surgical education.

Fig. 1

Independence-scaled procedural assessment form: Key steps of a procedure composed with the Delphi methodology combined to a scale based on the amount of assistants a trainee needs

Fig. 2

Validity of the independence-scaled procedural assessment and GRSs. Procedural assessment and GRSs scores improved significantly with surgical experience (OSATS p = 0.001, GOALS p < 0.001, Procedural assessment p < 0.001). However, the independence-scaled procedural assessment was the only one of the three assessment methods that could differentiate between the video of the intermediate and sub competent trainee among the surgical raters (p = 0.005)

Fig. 3

Results of the questionnaire distributed among surgeons and higher surgical trainees

Fig. 4

Two-step system for the development of procedure-specific assessments