Eye tracking: empirical foundations for a minimal reporting guideline

Kenneth Holmqvist^{1

2

3}, Saga Lee Örbom⁴, Ignace T C Hooge⁵, Diederick C Niehorster⁶, Robert G Alexander⁷, Richard Andersson⁸, Jeroen S Benjamins^{5

9}, Pieter Blignaut¹⁰, Anne-Marie Brouwer¹¹, Lewis L Chuang^{12

13}, Kirsten A Dalrymple¹⁴, Denis Drieghe¹⁵, Matt J Dunn¹⁶, Ulrich Ettinger¹⁷, Susann Fiedler¹⁸, Tom Foulsham¹⁹, Jos N van der Geest²⁰, Dan Witzner Hansen²¹, Samuel B Hutton²², Enkelejda Kasneci²³, Alan Kingstone²⁴, Paul C Knox²⁵, Ellen M Kok^{26

27}, Helena Lee²⁸, Joy Yeonjoo Lee²⁹, Jukka M Leppänen³⁰, Stephen Macknik⁷, Päivi Majaranta³¹, Susana Martinez-Conde⁷, Antje Nuthmann³², Marcus Nyström³³, Jacob L Orquin^{34

35}, Jorge Otero-Millan³⁶, Soon Young Park³⁷, Stanislav Popelka³⁸, Frank Proudlock³⁹, Frank Renkewitz⁴⁰, Austin Roorda³⁶, Michael Schulte-Mecklenbeck^{41

42}, Bonita Sharif⁴³, Frederick Shic^{44

45}, Mark Shovman^{46

47}, Mervyn G Thomas³⁹, Ward Venrooij⁴⁸, Raimondas Zemblys⁴⁹, Roy S Hessels⁵

Affiliations

¹ Department of Psychology, Nicolaus Copernicus University, Torun, Poland. kenneth.holmqvist@ur.de.
² Department of Computer Science and Informatics, University of the Free State, Bloemfontein, South Africa. kenneth.holmqvist@ur.de.
³ Department of Psychology, Regensburg University, Regensburg, Germany. kenneth.holmqvist@ur.de.
⁴ Department of Psychology, Regensburg University, Regensburg, Germany.
⁵ Experimental Psychology, Helmholtz Institute, Utrecht University, Utrecht, The Netherlands.
⁶ Lund University Humanities Lab and Department of Psychology, Lund University, Lund, Sweden.
⁷ Department of Ophthalmology, SUNY Downstate Health Sciences University, Brooklyn, NY, USA.
⁸ Tobii Pro AB, Danderyd, Sweden.
⁹ Social, Health and Organizational Psychology, Utrecht University, Utrecht, The Netherlands.
¹⁰ Department of Computer Science and Informatics, University of the Free State, Bloemfontein, South Africa.
¹¹ TNO, Soesterberg, The Netherlands.
¹² Department of Ergonomics, Leibniz Institute for Working Environments and Human Factors, Dortmund, Germany.
¹³ Institute of Informatics, LMU Munich, Munich, Germany.
¹⁴ Institute of Child Development, University of Minnesota, Minneapolis, USA.
¹⁵ School of Psychology, University of Southampton, Southampton, UK.
¹⁶ School of Optometry and Vision Sciences, Cardiff University, Cardiff, UK.
¹⁷ Department of Psychology, University of Bonn, Bonn, Germany.
¹⁸ Vienna University of Economics and Business, Vienna, Austria.
¹⁹ Department of Psychology, University of Essex, Essex, UK.
²⁰ Department of Neuroscience, Erasmus MC, Rotterdam, The Netherlands.
²¹ Machine Learning Group, Department of Computer Science, IT University of Copenhagen, Copenhagen, Denmark.
²² SR Research Ltd, Ottawa, Canada.
²³ Human-Computer Interaction, University of Tübingen, Tübingen, Germany.
²⁴ University of British Columbia, Columbia, Canada.
²⁵ Department of Eye and Vision Science, Institute of Life Course and Medical Sciences, University of Liverpool, Liverpool, UK.
²⁶ Department of Education and Pedagogy, Division Education, Faculty of Social and Behavioral Sciences, Utrecht University, Utrecht, The Netherlands.
²⁷ Department of Online Learning and Instruction, Faculty of Educational Sciences, Open University of the Netherlands, Heerlen, The Netherlands.
²⁸ University of Southampton, Southampton, UK.
²⁹ School of Health Professions Education, Faculty of Health, Medicine, and Life Sciences, Maastricht University, Maastricht, The Netherlands.
³⁰ Department of Psychology and Speed-Language Pathology, University of Turku, Turku, Finland.
³¹ TAUCHI Research Center, Computing Sciences, Faculty of Information Technology and Communication Sciences, Tampere University, Tampere, Finland.
³² Institute of Psychology, University of Kiel, Kiel, Germany.
³³ Lund University Humanities Lab, Lund University, Lund, Sweden.
³⁴ Department of Management, Aarhus University, Aarhus, Denmark.
³⁵ Center for Research in Marketing and Consumer Psychology, Reykjavik University, Reykjavik, Iceland.
³⁶ Herbert Wertheim School of Optometry and Vision Science, University of California, Berkeley, CA, USA.
³⁷ Comparative Cognition, Messerli Research Institute, University of Veterinary Medicine Vienna, Medical University of Vienna, Vienna, Austria.
³⁸ Department of Geoinformatics, Palacký University Olomouc, Olomouc, Czech Republic.
³⁹ The University of Leicester Ulverscroft Eye Unit, Department of Neuroscience, Psychology and Behaviour, University of Leicester, Leicester, UK.
⁴⁰ Department of Psychology, University of Erfurt, Erfurt, Germany.
⁴¹ University of Bern, Bern, Switzerland.
⁴² Max Planck Institute for Human Development, Berlin, Germany.
⁴³ School of Computing, University of Nebraska-Lincoln, Lincoln, Nebraska, USA.
⁴⁴ Center for Child Health, Behavior and Development, Seattle Children's Research Institute, Seattle, WA, USA.
⁴⁵ Department of General Pediatrics, University of Washington School of Medicine, Seattle, WA, USA.
⁴⁶ Eyeviation Systems, Herzliya, Israel.
⁴⁷ Department of Industrial Design, Bezalel Academy of Arts and Design, Jerusalem, Israel.
⁴⁸ Electrical Engineering, Mathematics and Computer Science (EEMCS), University of Twente, Enschede, The Netherlands.
⁴⁹ Smart Eye AB, Göteborg, Sweden.

PMID: 35384605
PMCID: PMC9535040
DOI: 10.3758/s13428-021-01762-8

Review

Eye tracking: empirical foundations for a minimal reporting guideline

Kenneth Holmqvist et al. Behav Res Methods. 2023 Jan.

. 2023 Jan;55(1):364-416.

doi: 10.3758/s13428-021-01762-8. Epub 2022 Apr 6.

Authors

Affiliations

¹ Department of Psychology, Nicolaus Copernicus University, Torun, Poland. kenneth.holmqvist@ur.de.
² Department of Computer Science and Informatics, University of the Free State, Bloemfontein, South Africa. kenneth.holmqvist@ur.de.
³ Department of Psychology, Regensburg University, Regensburg, Germany. kenneth.holmqvist@ur.de.
⁴ Department of Psychology, Regensburg University, Regensburg, Germany.
⁵ Experimental Psychology, Helmholtz Institute, Utrecht University, Utrecht, The Netherlands.
⁶ Lund University Humanities Lab and Department of Psychology, Lund University, Lund, Sweden.
⁷ Department of Ophthalmology, SUNY Downstate Health Sciences University, Brooklyn, NY, USA.
⁸ Tobii Pro AB, Danderyd, Sweden.
⁹ Social, Health and Organizational Psychology, Utrecht University, Utrecht, The Netherlands.
¹⁰ Department of Computer Science and Informatics, University of the Free State, Bloemfontein, South Africa.
¹¹ TNO, Soesterberg, The Netherlands.
¹² Department of Ergonomics, Leibniz Institute for Working Environments and Human Factors, Dortmund, Germany.
¹³ Institute of Informatics, LMU Munich, Munich, Germany.
¹⁴ Institute of Child Development, University of Minnesota, Minneapolis, USA.
¹⁵ School of Psychology, University of Southampton, Southampton, UK.
¹⁶ School of Optometry and Vision Sciences, Cardiff University, Cardiff, UK.
¹⁷ Department of Psychology, University of Bonn, Bonn, Germany.
¹⁸ Vienna University of Economics and Business, Vienna, Austria.
¹⁹ Department of Psychology, University of Essex, Essex, UK.
²⁰ Department of Neuroscience, Erasmus MC, Rotterdam, The Netherlands.
²¹ Machine Learning Group, Department of Computer Science, IT University of Copenhagen, Copenhagen, Denmark.
²² SR Research Ltd, Ottawa, Canada.
²³ Human-Computer Interaction, University of Tübingen, Tübingen, Germany.
²⁴ University of British Columbia, Columbia, Canada.
²⁵ Department of Eye and Vision Science, Institute of Life Course and Medical Sciences, University of Liverpool, Liverpool, UK.
²⁶ Department of Education and Pedagogy, Division Education, Faculty of Social and Behavioral Sciences, Utrecht University, Utrecht, The Netherlands.
²⁷ Department of Online Learning and Instruction, Faculty of Educational Sciences, Open University of the Netherlands, Heerlen, The Netherlands.
²⁸ University of Southampton, Southampton, UK.
²⁹ School of Health Professions Education, Faculty of Health, Medicine, and Life Sciences, Maastricht University, Maastricht, The Netherlands.
³⁰ Department of Psychology and Speed-Language Pathology, University of Turku, Turku, Finland.
³¹ TAUCHI Research Center, Computing Sciences, Faculty of Information Technology and Communication Sciences, Tampere University, Tampere, Finland.
³² Institute of Psychology, University of Kiel, Kiel, Germany.
³³ Lund University Humanities Lab, Lund University, Lund, Sweden.
³⁴ Department of Management, Aarhus University, Aarhus, Denmark.
³⁵ Center for Research in Marketing and Consumer Psychology, Reykjavik University, Reykjavik, Iceland.
³⁶ Herbert Wertheim School of Optometry and Vision Science, University of California, Berkeley, CA, USA.
³⁷ Comparative Cognition, Messerli Research Institute, University of Veterinary Medicine Vienna, Medical University of Vienna, Vienna, Austria.
³⁸ Department of Geoinformatics, Palacký University Olomouc, Olomouc, Czech Republic.
³⁹ The University of Leicester Ulverscroft Eye Unit, Department of Neuroscience, Psychology and Behaviour, University of Leicester, Leicester, UK.
⁴⁰ Department of Psychology, University of Erfurt, Erfurt, Germany.
⁴¹ University of Bern, Bern, Switzerland.
⁴² Max Planck Institute for Human Development, Berlin, Germany.
⁴³ School of Computing, University of Nebraska-Lincoln, Lincoln, Nebraska, USA.
⁴⁴ Center for Child Health, Behavior and Development, Seattle Children's Research Institute, Seattle, WA, USA.
⁴⁵ Department of General Pediatrics, University of Washington School of Medicine, Seattle, WA, USA.
⁴⁶ Eyeviation Systems, Herzliya, Israel.
⁴⁷ Department of Industrial Design, Bezalel Academy of Arts and Design, Jerusalem, Israel.
⁴⁸ Electrical Engineering, Mathematics and Computer Science (EEMCS), University of Twente, Enschede, The Netherlands.
⁴⁹ Smart Eye AB, Göteborg, Sweden.

PMID: 35384605
PMCID: PMC9535040
DOI: 10.3758/s13428-021-01762-8

Retraction in

Retraction Note: Eye tracking: empirical foundations for a minimal reporting guideline.
Holmqvist K, Örbom SL, Hooge ITC, Niehorster DC, Alexander RG, Andersson R, Benjamins JS, Blignaut P, Brouwer AM, Chuang LL, Dalrymple KA, Drieghe D, Dunn MJ, Ettinger U, Fiedler S, Foulsham T, van der Geest JN, Hansen DW, Hutton SB, Kasneci E, Kingstone A, Knox PC, Kok EM, Lee H, Lee JY, Leppänen JM, Macknik S, Majaranta P, Martinez-Conde S, Nuthmann A, Nyström M, Orquin JL, Otero-Millan J, Park SY, Popelka S, Proudlock F, Renkewitz F, Roorda A, Schulte-Mecklenbeck M, Sharif B, Shic F, Shovman M, Thomas MG, Venrooij W, Zemblys R, Hessels RS. Holmqvist K, et al. Behav Res Methods. 2024 Jan;56(1):511-512. doi: 10.3758/s13428-023-02285-0. Behav Res Methods. 2024. PMID: 37973712 Free PMC article. No abstract available.

Abstract

In this paper, we present a review of how the various aspects of any study using an eye tracker (such as the instrument, methodology, environment, participant, etc.) affect the quality of the recorded eye-tracking data and the obtained eye-movement and gaze measures. We take this review to represent the empirical foundation for reporting guidelines of any study involving an eye tracker. We compare this empirical foundation to five existing reporting guidelines and to a database of 207 published eye-tracking studies. We find that reporting guidelines vary substantially and do not match with actual reporting practices. We end by deriving a minimal, flexible reporting guideline based on empirical research (Section "An empirically based minimal reporting guideline").

Keywords: Data quality; Eye movements; Eye tracking; Replicability; Reporting guidelines; Reporting practices; Reporting standards; Reproducibility.

PubMed Disclaimer

Figures

**Fig. 1**
From eye orientation to higher-order eye-tracking measures. This is a crude division of the process from eye orientation to higher-order eye-tracking measures. There may be cases where a more fine-grained division is applicable

**Fig. 2**
Characteristics of eye-tracking data quality. A Horizontal gaze position (in Fick, , coordinates, see Haslwanter (1995)) of the right eye as a function of time. The gaze position was recorded from an adult participant with an EyeLink 1000 by Hooge et al., (2015). Call-outs indicate the relatively precise gaze-position signal (compared with panel B). B Horizontal gaze position in Fick coordinates of the right eye as a function of time. The gaze position was recorded from an infant participant with the Tobii TX300 by Hessels et al., (2016). *Call-outs* indicate the relatively imprecise gaze-position signal (compared with panel A), short gaps in the gaze-position signal (data loss), and an extreme gaze position reported by the eye tracker. The extreme gaze position is interesting because it can be considered an aspect of eye-tracking data quality not captured in the measures accuracy, precision, or data loss. C, D Gaze position signals (*black dots*) in a 2D representation, i.e. as if on a screen. Gaze position signals were recorded from adult participants by Hooge et al., (2019). Gaze position samples with high velocity were removed such that saccades are not visible. Orange markers represent validation targets. They are positioned to illustrate good/poor accuracy and do not correspond to the location of the actual validation targets in the experiment by Hooge et al., (2019). Call-outs indicate validation targets with corresponding precise and accurate, precise and inaccurate, imprecise and accurate, and imprecise and inaccurate gaze position signals, respectively. Note that the qualifications ‘precise’, ‘imprecise’, ‘accurate’, and ‘inaccurate’ are relative here and are often quantified

**Fig. 3**
Example of a head-boxed eye-tracking setup. The setup consists of a participant, eye tracker (camera and IR illuminator) and a computer screen. The geometry of this setup can be described by the relative orientations and distances of the monitor, camera and IR illuminator, and participant. Some eye trackers have a fixed relation with the computer screen (e.g. Tobii Pro Spectrum), while others do not and allow for more adjustments (e.g. SR Research EyeLink 1000). Note that the eye-tracker distance and screen distance are not identical. Screen height and width refer to both the physical and the pixel measures

See this image and copyright information in PMC

References

1. Açik A, Onat S, Schumann F, Einhäuser W, König P. Effects of luminance contrast and its modifications on fixation behavior during free viewing of images from different categories. Vision Research. 2009;49(12):1541–1553. doi: 10.1016/j.visres.2009.03.011. - DOI - PubMed
1. Açık, A., Sarwary, A., Schultze-Kraft, R., Onat, S., & König, P. (2010). Developmental changes in natural viewing behavior: bottom-up and top-down differences between children, young adults and older adults. Frontiers in Psychology, 1, 207. 10.3389/fpsyg.2010.00207. - PMC - PubMed
1. Ahern S, Beatty J. Pupillary responses during information processing vary with scholastic aptitude test scores. Science. 1979;205(4412):1289–1292. doi: 10.1126/science.472746. - DOI - PubMed
1. Ahlstrom C, Nyström M, Holmqvist K, Fors C, Sandberg D, Anund A, Kecklund G, Åkerstedt T. Fit-for-duty test for estimation of drivers’ sleepiness level: Eye movements improve the sleep/wake predictor. Transportation Research Part C: Emerging Technologies. 2013;26:20–32. doi: 10.1016/j.trc.2012.07.008. - DOI
1. Ahlstrom U, Friedman-Berg FJ. Using eye movement activity as a correlate of cognitive workload. International Journal of Industrial Ergonomics. 2006;36(7):623–636.

Publication types

Actions
Actions
Actions

MeSH terms

Actions
Actions
Actions
Actions

Grants and funding

LinkOut - more resources

Full Text Sources

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

Retracted article

Eye tracking: empirical foundations for a minimal reporting guideline

Affiliations

Eye tracking: empirical foundations for a minimal reporting guideline

Authors

Affiliations

Retraction in

Abstract

Figures

References

Publication types

MeSH terms

Grants and funding

LinkOut - more resources

Full Text Sources