Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Review
. 2017 Nov-Dec;74(6):1081-1087.
doi: 10.1016/j.jsurg.2017.05.006. Epub 2017 May 20.

Multi-Institutional Development of a Mastoidectomy Performance Evaluation Instrument

Thomas Kerwin  1 , Brad Hittle  2 , Don Stredney  2 , Paul De Boeck  3 , Gregory Wiet  4
Affiliations
Review

Multi-Institutional Development of a Mastoidectomy Performance Evaluation Instrument

Thomas Kerwin et al. J Surg Educ. 2017 Nov-Dec.

Abstract

Objective: A method for rating surgical performance of a mastoidectomy procedure that is shown to apply universally across teaching institutions has not yet been devised. This work describes the development of a rating instrument created from a multi-institutional consortium.

Design: Using a participatory design and a modified Delphi approach, a multi-institutional group of expert otologists constructed a 15-element task-based checklist for evaluating mastoidectomy performance. This instrument was further refined into a 14-element checklist focusing on the concept of safety after using it to rate a large and varied population of performances.

Setting: Twelve otolaryngological surgical training programs in the United States.

Participants: A total of 14 surgeons from 12 different institutions took part in the construction of the instrument.

Results: By using 14 experts from 12 different institutions and a literature review, individual metrics were identified, rated as to the level of importance and operationally defined to create a rating scale for mastoidectomy performance. Initial use of the rating scale showed modest rater agreement. The operational definitions of individual metrics were modified to emphasize "safe" as opposed to "proper" technique. A second rating instrument was developed based on this feedback.

Conclusions: Using a consensus-building approach with multiple rounds of communication between experts is a feasible way to construct a rating instrument for mastoidectomy. Expert opinion alone using a Delphi method provides face and content validity evidence, however, this is not sufficient to develop a universally acceptable rating instrument. A continued process of development and experimentation to demonstrate evidence for reliability and validity making use of a large population of raters and performances is necessary to achieve universal acceptance.

Keywords: Medical Knowledge; Practice-Based Learning and Improvement; assessment; mastoidectomy; multi-institution; otology.

PubMed Disclaimer

Figures

Figure 1
Figure 1
Example slide of an individual metric level of importance and operational definition discussion based on survey to group of experts.
See this image and copyright information in PMC

References

    1. Catania AC. Learning. 2nd. Englewood Cliffs, N.J: Prentice-Hall; 1984.
    1. Shah J, Darzi A. Surgical skills assessment: an ongoing debate. BJU Int. 2001;88(7):655–660. - PubMed
    1. Michelson JD, Manning L. Competency assessment in simulation-based procedural education. Am J Surg. 2008;196(4):609–615. - PubMed
    1. Sethia R, Kerwin TF, Wiet GJ. Performance Assessment for Mastoidectomy: State of the Art Review. Otolaryngol Head Neck Surg. 2016 - PMC - PubMed
    1. Martin JA, Regehr G, Reznick R, et al. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg. 1997;84(2):273–278. - PubMed

LinkOut - more resources