Comparing Multiple Methods to Measure Procedural Fidelity of Discrete-trial Instruction

Abstract

Procedural fidelity is the extent to which an intervention is implemented as designed and is an important component of research and practice. There are multiple ways to measure procedural fidelity, and few studies have explored how procedural fidelity varies based on the method of measurement. The current study compared adherence to discrete-trial instruction protocols by behavior technicians with a child with autism when observers used different procedural-fidelity measures. We collected individual-component and individual-trial fidelity with an occurrence-nonoccurrence data sheet and compared these scores to global fidelity and all-or-nothing, 3-point Likert scale, and 5-point Likert scale measurement methods. The all-or-nothing method required all instances of a component or trial be implemented without error to be scored correct. The Likert scales used a rating system to score components and trials. At the component level, we found that the global, 3-point Likert, and 5-point Likert methods were likely to overestimate fidelity and mask component errors, and the all-or-nothing method was unlikely to mask errors. At the trial level, we found that the global and 5-point Likert methods approximated individual-trial fidelity, the 3-point Likert method overestimated fidelity, and the all-or-nothing method underestimated fidelity. The occurrence-nonoccurrence method required the most time to complete, and all-or-nothing by trial required the least. We discuss the implications of measuring procedural fidelity with different methods of measurement, including false positives and false negatives, and provide suggestions for practice and research.

Supplementary information: The online version contains supplementary material available at 10.1007/s43494-023-00094-w.

Keywords: Autism; Behavior analysis; Discrete-trial instruction; Procedural integrity; Treatment integrity.

Fig. 1

Component-Level Procedural Fidelity Comparison. Note. Percentage of procedural fidelity as individual-component (gray bars), global (solid lines), all-or-nothing by component (dotted lines), 3-pt Likert by component (dashed lines), and 5-pt Likert by component (dotted-dashed lines) values. The letters underneath the x-axis correspond to the steps of the procedure. Components not visible (N/V) during an observation are represented with a dagger (†) and components that did not occur (N/A) are represented with an x and were excluded from calculations. The letters after each observation number correspond to each technician

Fig. 2

Trial-Level Procedural Fidelity Comparison. Note. Percentage of procedural fidelity as individual-trial (gray bars), global (solid lines), all-or-nothing by trial (dotted lines), 3-pt Likert by trial (dashed lines), and 5-pt Likert by trial (dotted-dashed lines) values. Trials that were not captured on the video are represented with an x and not factored into scores for all measures. The letters after each observation number correspond to each technician

Fig. 3

Mean Scoring Duration by Measurement Method. Note. Average duration of scoring for each scoring method in minutes. Data points correspond to individual durations for each observer. The brackets above indicate statistically significant differences in scoring duration for different scoring methods. Asterisks above each bracket correspond to obtained p values. * p = .002; **p = .001; ***p > .001