Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2007 Sep;64(9):582-8.
doi: 10.1136/oem.2006.030528. Epub 2007 Jan 16.

Inter-rater agreement in the assessment of exposure to carcinogens in the offshore petroleum industry

Kjersti Steinsvåg  1 , Magne Bråtveit, Bente E Moen, Hans Kromhout
Affiliations

Inter-rater agreement in the assessment of exposure to carcinogens in the offshore petroleum industry

Kjersti Steinsvåg et al. Occup Environ Med. 2007 Sep.

Abstract

Objectives: To evaluate the reliability of an expert team assessing exposure to carcinogens in the offshore petroleum industry and to study how the information provided influenced the agreement among raters.

Methods: Eight experts individually assessed the likelihood of exposure for combinations of 17 carcinogens, 27 job categories and four time periods (1970-1979, 1980-1989, 1990-1999 and 2000-2005). Each rater assessed 1836 combinations based on summary documents on carcinogenic agents, which included descriptions of sources of exposure and products, descriptions of work processes carried out within the different job categories, and monitoring data. Inter-rater agreement was calculated using Cohen's kappa index and single and average score intraclass correlation coefficients (ICC) (ICC(2,1) and ICC(2,8), respectively). Differences in inter-rater agreement for time periods, raters, International Agency for Research on Cancer groups and the amount of information provided were consequently studied.

Results: Overall, 18% of the combinations were denoted as possible exposure, and 14% scored probable exposure. Stratified by the 17 carcinogenic agents, the probable exposure prevalence ranged from 3.8% for refractory ceramic fibres to 30% for crude oil. Overall mean kappa was 0.42 (ICC(2,1) = 0.62 and ICC(2,8) = 0.93). Providing limited quantitative measurement data was associated with less agreement than for equally well described carcinogens without sampling data.

Conclusion: The overall kappa and single-score ICC indicate that the raters agree on exposure estimates well above the chance level. The levels of inter-rater agreement were higher than in other comparable studies. The average score ICC indicates reliable mean estimates and implies that sufficient raters were involved. The raters seemed to have enough documentation on which to base their estimates, but provision of limited monitoring data leads to more incongruence among raters. Having real exposure data, with the inherent variability of such data, apparently makes estimating exposure in a rigid semiquantitative manner more difficult.

PubMed Disclaimer

Conflict of interest statement

Competing interests: none declared

References

    1. Teschke K. Exposure surrogates: job‐exposure matrices, self‐reports, and expert evaluations. In Nieuwenhuijsen MJ, ed. Exposure assessment in occupational and environmental epidemiology. Oxford: Oxford University Press, 2003118–132.
    1. Teschke K, Olshan A F, Daniels J L.et al Occupational exposure assessment in case‐control studies: opportunities for improvement. Occup Environ Med 200259575–594. - PMC - PubMed
    1. Teschke K, Hertzman C, Dimich‐Ward H.et al A comparison of exposure estimates by worker raters and industrial hygienists. Scand J Work Environ Health 19895424–429. - PubMed
    1. Hawkins N C, Evans J S. Subjective estimation of toluene exposures: a calibration study of industrial hygienists. Appl Ind Hyg 1989461–68.
    1. Landis J R, Koch G G. The measurement of observer agreement for categorical data. Biometrics 197733159–174. - PubMed

Publication types