Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2020 Mar;4(3):317-325.
doi: 10.1038/s41562-019-0813-1. Epub 2020 Feb 3.

The Confidence Database

Dobromir Rahnev  1 Kobe Desender  2   3 Alan L F Lee  4 William T Adler  5 David Aguilar-Lleyda  6 Başak Akdoğan  7 Polina Arbuzova  8   9   10 Lauren Y Atlas  11   12   13 Fuat Balcı  14 Ji Won Bang  15 Indrit Bègue  16 Damian P Birney  17 Timothy F Brady  18 Joshua Calder-Travis  19 Andrey Chetverikov  20 Torin K Clark  21 Karen Davranche  22 Rachel N Denison  23 Troy C Dildine  11   24 Kit S Double  25 Yalçın A Duyan  14 Nathan Faivre  26 Kaitlyn Fallow  27 Elisa Filevich  8   9   10 Thibault Gajdos  22 Regan M Gallagher  28   29   30 Vincent de Gardelle  31 Sabina Gherman  32   33 Nadia Haddara  34 Marine Hainguerlot  35 Tzu-Yu Hsu  36 Xiao Hu  37 Iñaki Iturrate  38 Matt Jaquiery  19 Justin Kantner  39 Marcin Koculak  40 Mahiko Konishi  41 Christina Koß  8   10 Peter D Kvam  42 Sze Chai Kwok  43   44   45 Maël Lebreton  46 Karolina M Lempert  47 Chien Ming Lo  36   48 Liang Luo  37 Brian Maniscalco  49 Antonio Martin  36 Sébastien Massoni  50 Julian Matthews  30   51 Audrey Mazancieux  26 Daniel M Merfeld  52 Denis O'Hora  53 Eleanor R Palser  54   55   56 Borysław Paulewicz  57 Michael Pereira  58 Caroline Peters  8   9   10 Marios G Philiastides  32 Gerit Pfuhl  59 Fernanda Prieto  60 Manuel Rausch  61 Samuel Recht  62 Gabriel Reyes  60 Marion Rouault  63 Jérôme Sackur  63   64 Saeedeh Sadeghi  65 Jason Samaha  66 Tricia X F Seow  67 Medha Shekhar  34 Maxine T Sherman  68   69 Marta Siedlecka  40 Zuzanna Skóra  40 Chen Song  70 David Soto  71   72 Sai Sun  73 Jeroen J A van Boxtel  30   74 Shuo Wang  75 Christoph T Weidemann  76 Gabriel Weindel  22 Michał Wierzchoń  40 Xinming Xu  43 Qun Ye  43 Jiwon Yeon  34 Futing Zou  43 Ariel Zylberberg  77
Affiliations

The Confidence Database

Dobromir Rahnev et al. Nat Hum Behav. 2020 Mar.

Abstract

Understanding how people rate their confidence is critical for the characterization of a wide range of perceptual, memory, motor and cognitive processes. To enable the continued exploration of these processes, we created a large database of confidence studies spanning a broad set of paradigms, participant populations and fields of study. The data from each study are structured in a common, easy-to-use format that can be easily imported and analysed using multiple software packages. Each dataset is accompanied by an explanation regarding the nature of the collected data. At the time of publication, the Confidence Database (which is available at https://osf.io/s46pr/) contained 145 datasets with data from more than 8,700 participants and almost 4 million trials. The database will remain open for new submissions indefinitely and is expected to continue to grow. Here we show the usefulness of this large collection of datasets in four different analyses that provide precise estimations of several foundational confidence-related effects.

PubMed Disclaimer

Figures

Figure 1.
Figure 1.. Datasets currently in the Confidence Database.
Pie charts showing the number of datasets split by category, publication year, number of participants, number of trials per participant, type of judgment, and rating scale. The label “Multiple” in the first pie chart indicates that the same participants completed tasks from more than one category. The maximum number of participants was 589 and the maximum trials per participant was 4,320 (“variable” indicates that different participants completed different number of trials).
Figure 2.
Figure 2.. Correlating confidence with choice and confidence RT.
(A) We found a medium-to-large negative correlation (r = −.24, p < 2.2e-16, n = 4,089) between confidence and choice RT, as well as a small negative correlation (r = −.07, p < 2.2e-16, n = 4,089) between confidence and confidence RT. Box shows the median and the interquartile (25-75%) range, whereas the whiskers show the 2-98% range. (B) The strength of the two correlations in panel A were themselves correlated across subjects (r = .23, p < 2.2e-16, n = 4,089).
Figure 3.
Figure 3.. Serial dependence in confidence RT.
We observed a large lag-1 autocorrelation (b = 1.346, t(1299601) = 153.6, p < 2.2e-16, n = 4,474). The autocorrelation decreased for higher lags but remained significant up to lag-7 (all p’s < 2.2e-16, n = 4,474). Error bars indicate SEM. Individual datapoints are not shown because the plots are based on the results of a mixed model analysis.
Figure 4.
Figure 4.. The prevalence of estimates of negative metacognitive sensitivity.
(A) Individual beta values and beta values density plot for the observed relationship between confidence and accuracy. Box shows the median and the interquartile (25-75%) range, whereas the whiskers show the 2-98% range. (B-F) Scatter plots, including lines of best fit, for the relationships between the beta value for confidence-accuracy relationship and the number of trials (B), average accuracy (C), median choice RT (D), median confidence RT (E), and the proportion of trials where the most common confidence judgment was given (F).
Figure 5.
Figure 5.. Confidence scale use for perception and memory studies.
The percent of 2-point, 3-point, 4-point, 5-point, 6-point, 7-to-11-point, and continuous confidence scales were plotted separately for perception and memory datasets. We combined the 7- to 11-point scales because of the low number of datasets with such scales. The two domains differed in how often they employed 3-point and continuous scales.

Similar articles

Cited by

References

    1. Mamassian P Visual Confidence. Annu. Rev. Vis. Sci 2, annurev-vision-111815–114630 (2016). - PubMed
    1. Weidemann CT & Kahana MJ Assessing recognition memory using confidence ratings and response times. R. Soc. Open Sci 3, 150670 (2016). - PMC - PubMed
    1. Peirce CS & Jastrow J On Small Differences in Sensation. Mem. Natl. Acad. Sci 3, 75–83 (1884).
    1. Ratcliff R, Van Zandt T & McKoon G Process dissociation, single-process theories, and recognition memory. J. Exp. Psychol. Gen 124, 352–74 (1995). - PubMed
    1. Azzopardi P & Cowey A Is blindsight like normal, near-threshold vision? Proc. Natl. Acad. Sci 94, 14190–14194 (1997). - PMC - PubMed

Publication types

MeSH terms