Construct and Predictive Validity of a Cystoscopic Checklist to Evaluate Surgical Competency in the Operating Room

Abstract

Purpose: The aim of this study was to demonstrate the construct and predictive validity of a task-specific checklist for cystoscopy in the operating room.

Methods: This study is a prospective observational study evaluating the validity of a task-specific checklist for cystoscopic proficiency on residents performing cystoscopy in the operating room over a 4-year period. Residents were scored on 2 task-specific checklists: (1) assembly and (2) performance of a diagnostic cystoscopy. Construct validity was evaluated by comparing absolute checklist scores based on resident year with a Kruskal-Wallis and χ test, respectively. Paired analysis was performed with Wilcoxon sign rank and McNemar tests to assess differences between initial and final observations. Predictive validity was evaluated by determining the percent of competently performed task-specific checklists subsequent to 1 competent task-specific checklist completion.

Results: One hundred fifty-three evaluations were performed on 30 residents from all 4 levels of training. The median task-specific checklist scores for both (1) assembly and (2) performance of a diagnostic cystoscopy demonstrated construct validity with statistically significant trends demonstrating improved scores as resident year increased from intern to chief, P values of <0.001 and <0.0001, respectively. Once competency was obtained for 1 cystoscope assembly checklist, 92% of future assembly checklists were competently competent. For diagnostic cystoscopy once checklist competency was obtained, 69% of future checklists were completed competently. All scores and competencies were significantly improved from the initial to the final observation.

Discussion: The task-specific checklists for (1) assembly and (2) performance of a diagnostic cystoscopy demonstrated construct validity. Predicative validity for assembly of a cystoscope is high but less so for performance of a diagnostic cystoscope.