Sometimes Means Some of the Time: Residents' Overlapping Responses to Vague Quantifiers on the ACGME-I Resident Survey

Abstract

Background: Vague quantifiers used in the Accreditation Council for Graduate Medical Education-International (ACGME-I) resident survey are open to interpretation, raising concerns about the validity of survey scores. Residency programs may be unduly cited if survey responses are affected by differing judgments of vague quantifiers.

Objective: Through investigating frequency judgment overlap, we assessed the validity of vague quantifiers by quantifying variation in residents' frequency judgment of the following response options: never, rarely, sometimes, very often, and extremely often.

Methods: We conducted a cross-sectional survey of residents in 2 ACGME-I accredited institutions in Singapore. Participants assigned a frequency judgment to response options in 8 questions in the ACGME-I Resident Survey. Overlap in frequency judgment was computed using the minimum and maximum frequency judgment for each response option. This was ascertained to have occurred when the maximum frequency of the preceding category exceeded the minimum frequency of the downstream categories. The percentage of participants whose frequency judgment overlapped was computed.

Results: Of 652 residents, 289 (44%) responded; after exclusions of incomplete and careless responses, 119 responses (18%) were included in the study. Frequency judgment overlap was more frequent for vague quantifiers that are adjacent, ranging from 11% to 50% for questions in faculty, evaluation, and resources domains. The percentage of frequency judgment overlap was greatest for duty hour questions, with an overlap between 21% and 47% for adjacent categories.

Conclusions: Residents demonstrated wide variation in frequency judgment of vague quantifiers, especially on the duty hour questions in the ACGME-I resident survey.

Figure

Illustration of Frequency Judgment Overlap Across Vague Quantifiers