Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2009 Fall;42(3):527-39.
doi: 10.1901/jaba.2009.42-527.

Assessing observer accuracy in continuous recording of rate and duration: three algorithms compared

Oliver C Mudford  1 , Neil T Martin, Jasmine K Y Hui, Sarah Ann Taylor
Affiliations

Assessing observer accuracy in continuous recording of rate and duration: three algorithms compared

Oliver C Mudford et al. J Appl Behav Anal. 2009 Fall.

Abstract

The three algorithms most frequently selected by behavior-analytic researchers to compute interobserver agreement with continuous recording were used to assess the accuracy of data recorded from video samples on handheld computers by 12 observers. Rate and duration of responding were recorded for three samples each. Data files were compared with criterion records to determine observer accuracy. Block-by-block and exact agreement algorithms were susceptible to inflated agreement and accuracy estimates at lower rates and durations. The exact agreement method appeared to be overly stringent for recording responding at higher rates (23.5 responses per minute) and for higher relative duration (72% of session). Time-window analysis appeared to inflate accuracy assessment at relatively high but not at low response rate and duration (4.8 responses per minute and 8% of session, respectively).

Keywords: continuous recording; interobserver agreement; observational data; observer accuracy; recording and measurement.

PubMed Disclaimer

Figures

Figure 1
Figure 1
Criterion records for observational samples. Events are shown as a vertical line in each second in which they occurred in the rate samples. Seconds of occurrence of bouts of responding are shown as vertical bars for duration samples.
Figure 2
Figure 2
Mean percentage accuracy computed by the time-window analysis algorithm for behaviors measured as events and behaviors with duration at tolerances for agreement from 0 to ±5 s.
Figure 3
Figure 3
Mean percentage accuracy computed by exact agreement, block-by-block agreement, and time-window analysis with tolerance of ±2 s for low-rate (4.8 responses per minute), medium-rate (11.3 responses per minute), and high-rate (23.5 responses per minute) behaviors (top) and for low-duration (8%), medium-duration (44.4%), and high-duration (72%) behaviors (bottom).
See this image and copyright information in PMC

References

    1. Bijou S.W, Peterson R.F, Ault M.H. A method to integrate descriptive and experimental field studies at the level of data and empirical concepts. Journal of Applied Behavior Analysis. 1968;1:175–191. - PMC - PubMed
    1. Boykin R.A, Nelson R.O. The effect of instructions and calculation procedures on observers' accuracy, agreement, and calculation correctness. Journal of Applied Behavior Analysis. 1981;14:479–489. - PMC - PubMed
    1. Cooper J.O, Heron T.E, Heward W.L. Applied behavior analysis (2nd ed.) Upper Saddle River, NJ: Pearson; 2007.
    1. Fradenberg L.A, Harrison R.J, Baer D.M. The effect of some environmental factors on interobserver agreement. Research in Developmental Disabilities. 1995;16:425–437. - PubMed
    1. Hawkins R.P, Dotson V.A. Reliability scores that delude: An Alice in Wonderland trip through the misleading characteristics of interobserver agreement scores in interval recording. In: Ramp E, Semb G, editors. Behavior analysis: Areas of research and application. Englewood Cliffs, NJ: Prentice Hall; 1975. pp. 359–376. In.

Publication types

LinkOut - more resources