Assessing observer accuracy in continuous recording of rate and duration: three algorithms compared

Abstract

The three algorithms most frequently selected by behavior-analytic researchers to compute interobserver agreement with continuous recording were used to assess the accuracy of data recorded from video samples on handheld computers by 12 observers. Rate and duration of responding were recorded for three samples each. Data files were compared with criterion records to determine observer accuracy. Block-by-block and exact agreement algorithms were susceptible to inflated agreement and accuracy estimates at lower rates and durations. The exact agreement method appeared to be overly stringent for recording responding at higher rates (23.5 responses per minute) and for higher relative duration (72% of session). Time-window analysis appeared to inflate accuracy assessment at relatively high but not at low response rate and duration (4.8 responses per minute and 8% of session, respectively).

Keywords: continuous recording; interobserver agreement; observational data; observer accuracy; recording and measurement.

Figure 1

Criterion records for observational samples. Events are shown as a vertical line in each second in which they occurred in the rate samples. Seconds of occurrence of bouts of responding are shown as vertical bars for duration samples.

Figure 2

Mean percentage accuracy computed by the time-window analysis algorithm for behaviors measured as events and behaviors with duration at tolerances for agreement from 0 to ±5 s.

Figure 3

Mean percentage accuracy computed by exact agreement, block-by-block agreement, and time-window analysis with tolerance of ±2 s for low-rate (4.8 responses per minute), medium-rate (11.3 responses per minute), and high-rate (23.5 responses per minute) behaviors (top) and for low-duration (8%), medium-duration (44.4%), and high-duration (72%) behaviors (bottom).