Objective Structured Assessments of Technical Skills (OSATS) Does Not Assess the Quality of the Surgical Result Effectively

Abstract

Background: Performance assessment in skills training is ideally based on objective, reliable, and clinically relevant indicators of success. The Objective Structured Assessment of Technical Skill (OSATS) is a reliable and valid tool that has been increasingly used in orthopaedic skills training. It uses a global rating approach to structure expert evaluation of technical skills with the experts working from a list of operative competencies that are each rated on a 5-point Likert scale anchored by behavioral descriptors. Given the observational nature of its scoring, the OSATS might not effectively assess the quality of surgical results.

Questions/purposes: (1) Does OSATS scoring in an intraarticular fracture reduction training exercise correlate with the quality of the reduction? (2) Does OSATS scoring in a cadaveric extraarticular fracture fixation exercise correlate with the mechanical integrity of the fixation?

Methods: Orthopaedic residents at the University of Iowa (six postgraduate year [PGY]-1s) and at the University of Minnesota (seven PGY-1s and eight PGY-2s) undertook a skills training exercise that involved reducing a simulated intraarticular fracture under fluoroscopic guidance. Iowa residents participated three times during 1 month, and Minnesota residents participated twice with 1 month between their two sessions. A fellowship-trained orthopaedic traumatologist rated each performance using a modified OSATS scoring scheme. The quality of the articular reduction obtained was then directly measured. Regression analysis was performed between OSATS scores and two metrics of articular reduction quality: articular surface deviation and estimated contact stress. Another skills training exercise involved fixing a simulated distal radius fracture in a cadaveric specimen. Thirty residents, distributed across four PGY classes (PGY-2 and PGY-3, n = 8 each; PGY-4 and PGY-5, n = 7 each), simultaneously completed the exercise at individual stations. One of three faculty hand surgeons independently scored each performance using a validated OSATS scoring system. The mechanical integrity of each fixation construct was then assessed in a materials testing machine. Regression analysis was performed between OSATS scores and two metrics of fixation integrity: stiffness and failure load.

Results: In the intraarticular fracture model, OSATS scores did not correlate with articular reduction quality (maximum surface deviations: R = 0.17, p = 0.25; maximum contact stress: R = 0.22, p = 0.13). Similarly in the cadaveric extraarticular fracture model, OSATS scores did not correlate with the integrity of the mechanical fixation (stiffness: R = 0.10, p = 0.60; failure load: R = 0.30, p = 0.10).

Conclusions: OSATS scoring methods do not effectively assess the quality of the surgical result. Efforts must be made to incorporate assessment metrics that reflect the quality of the surgical result.

Clinical relevance: New objective, reliable, and clinically relevant measures of the quality of the surgical result obtained by a trainee are urgently needed. For intraarticular fracture reduction and extraarticular fracture fixation, direct physical measurement of reduction quality and of mechanical integrity of fixation, respectively, meet this need.

Fig. 1A–E

(A–C) The articular fracture reduction model is shown at various stages in the exercise, revealing the radioopaque surrogate bone specimen with soft tissue sleeve. (D) AP and (E) lateral fluoroscopic images of the intraarticular distal tibia fracture are shown, nearing final reduction with Kirschner wires having been placed by the resident.

Fig. 2A–B

This graphic depicts the basis of measurements to assess the final articular fracture reduction. (A) The measure of 3-D surface deviations directly reflect the degree of imprecision in the reduction, here shown illustratively for a fragment simply translated a fixed amount or rotated a fixed amount. (B) This graphic shows the basis for computationally estimating contact stress distributions, which indicate the influence of the surface deviations on the ankle contact mechanics.

Fig. 3A–D

This montage depicts the articular fracture simulation involving the fixing of a simulated extraarticular fracture of the distal radius in an upper extremity cadaver specimen. (A) The radiographic images in the upper left show the osteotomy with the cutting jig immediately proximal to the distal radioulnar joint (DRUJ) shown in the lower right. (B) Three residents are taking the examination at the same time here. A faculty member is grading one resident in the background. The faculty grading the residents in the foreground have stepped out of the picture, and a C-arm available to the residents is seen in the background. (C) This image shows the loading of the fracture fixation construct. (D) This illustrative displacement versus force data tracing shows load uptake and the basis for assessing stiffness and failure load.

Fig. 4

The maximum surface deviation was very weakly correlated with the OSATS score, suggesting that the two parameters measure different facets of performance in articular fracture reduction surgery.

Fig. 5

The maximum contact stress was very weakly correlated with the OSATS score, suggesting that the two parameters measure different facets of performance in articular fracture reduction surgery.

Fig. 6A–B

The results from these two trials with the articular fracture reduction exercise illustrate differences in the restoration of the articular surface and the degree of contact stress elevation. These trials were chosen because they showed a representative improvement in OSATS score going from (A) a baseline to (B) a followup exercise, which did not correlate with improvements in contact stress distributions.

Fig. 7

The stiffness of the fracture fixation construct that was achieved did not correlate well with the OSATS scores.

Fig. 8

The failure load of the fracture fixation construct that was achieved did not correlate well with the OSATS scores.