Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2011 Apr;54(2):116-22.
doi: 10.1503/cjs.032909.

Value of an objective assessment tool in the operating room

Ellen Hiemstra  1 , Wendela Kolkman, Ron Wolterbeek, Baptist Trimbos, Frank Willem Jansen
Affiliations

Value of an objective assessment tool in the operating room

Ellen Hiemstra et al. Can J Surg. 2011 Apr.

Abstract
in English, French

Background: Concerns about the achievement of surgical proficiency during residency are increasing. To objectify surgical skills, the Objective Structured Assessment of Technical Skills (OSATS) was developed and proven valid, feasible and reliable for use in laboratory settings. This study aimed to evaluate the value of this tool for intraoperative use.

Methods: Residents were assessed with an OSATS after every procedure they performed as the primary surgeon during a 3-month clinical rotation in gynecological surgery. We mapped individual learning curves (OSATS scores plotted against experience) and established the average procedure-specific learning curve. We used linear mixed models to assess the relation between performance and experience.

Results: Nine residents were recruited and 319 OSATS analyzed. Individual learning curves revealed progression beyond 24 of 30 OSATS points for 7 residents. Performance on the average procedure improved with experience, and the OSATS score increased by an average of 1.10 points per assessed procedure (p=0.008, 95% confidence interval 0.44-1.77). Median OSATS scores ranged from 18 to 30 among the 21 assessors.

Conclusion: Intraoperative implementation of OSATS seems to offer important advantages: structured feedback is facilitated, and learning curves enable insight into individual progression. However, doubts have been raised about the objectivity of the tool. Therefore, caution is warranted in using it for graduation and certification.

Contexte: La maîtrise de la chirurgie acquise au cours de la résidence préoccupe de plus en plus. Pour rendre les techniques chirurgicales objectives, on a mis au point l’évaluation structurée objective des habiletés techniques (Objective Structured Assessment of Technical Skills; OSATS) et on en a démontré la viabilité, la faisabilité et la fiabilité pour utilisation en laboratoire. Cette étude vise à déterminer la valeur de cet outil au cours des interventions.

Méthodes: Les médecins résidents ont été soumis à une évaluation OSATS après chaque intervention qu’ils ont pratiquée comme chirurgien principal au cours d’un stage clinique de 3 mois en chirurgie gynécologique. Nous avons établi les courbes d’apprentissage individuelles (courbes des résultats OSATS en fonction de l’expérience), ainsi que la courbe d’apprentissage moyenne particulière à l’intervention. Nous avons utilisé des modèles mixtes linéaires pour évaluer le lien entre le rendement et l’expérience.

Résultats: Nous avons recruté 9 médecins résidents et analysé 319 évaluations OSATS. Les courbes d’apprentissage individuelles ont révélé un progrès supérieur à 24 points OSATS sur 30 chez 7 médecins résidents. L’exécution de l’intervention moyenne s’est améliorée avec l’expérience et le résultat de l’évaluation OSATS a aug-menté en moyenne de 1,10 point par intervention évaluée (p = 0,008, intervalle de confiance à 95 %, 0,44–1,77). Les résultats médians de l’évaluation OSATS ont varié de 18 à 30 chez les 21 évaluateurs.

Conclusion: L’application de l’évaluation OSATS aux interventions semble offrir des avantages importants : les commentaires structurés sont facilités et les courbes d’apprentissage donnent une idée de l’évolution de chaque médecin. On a toutefois soulevé des doutes quant à l’objectivité de l’outil. La prudence s’impose donc dans l’utilisation de celui-ci aux fins d’obtention du diplôme et du certificat.

PubMed Disclaimer

Figures

Fig. 1
Fig. 1
Individual learning curves with regression lines. X axis = total caseload expressed in number of assessed procedures (regardless of the type of procedure performed), y axis = performance expressed in total Objective Structured Assessment of Technical Skills score, dotted line = individual regression line.
Fig. 2
Fig. 2
Average Objective Structured Assessment of Technical Skills (OSATS) scores plotted against procedure-specific learning curve for the first 10 of a specific type of procedure performed. An OSATS score of 24 was set as the performance standard. Dotted line = line based on linear mixed model analysis.
See this image and copyright information in PMC

References

    1. Hammond I, Karthigasu K. Training, assessment and competency in gynecologic surgery. Best Pract Res Clin Obstet Gynaecol. 2006;20:173–87. - PubMed
    1. Haluck RS, Krummel TM. Computers and virtual reality for surgical education in the 21st century. Arch Surg. 2000;135:786–92. - PubMed
    1. Kolkman W, Wolterbeek R, Jansen FW. Implementation of advanced laparoscopy into daily gynecologic practice: difficulties and solutions. J Minim Invasive Gynecol. 2006;13:4–9. - PubMed
    1. Shay BF, Thomas R, Monga M. Urology practice patterns after residency training in laparoscopy. J Endourol. 2002;16:251–6. - PubMed
    1. Kolkman W, Wolterbeek R, Jansen FW. Gynecological laparoscopy in residency training program: Dutch perspectives. Surg Endosc. 2005;19:1498–502. - PubMed