HMeta-d: hierarchical Bayesian estimation of metacognitive efficiency from confidence ratings

Abstract

Metacognition refers to the ability to reflect on and monitor one's cognitive processes, such as perception, memory and decision-making. Metacognition is often assessed in the lab by whether an observer's confidence ratings are predictive of objective success, but simple correlations between performance and confidence are susceptible to undesirable influences such as response biases. Recently, an alternative approach to measuring metacognition has been developed (Maniscalco and Lau 2012) that characterizes metacognitive sensitivity (meta-d') by assuming a generative model of confidence within the framework of signal detection theory. However, current estimation routines require an abundance of confidence rating data to recover robust parameters, and only provide point estimates of meta-d'. In contrast, hierarchical Bayesian estimation methods provide opportunities to enhance statistical power, incorporate uncertainty in group-level parameter estimates and avoid edge-correction confounds. Here I introduce such a method for estimating metacognitive efficiency (meta-d'/d') from confidence ratings and demonstrate its application for assessing group differences. A tutorial is provided on both the meta-d' model and the preparation of behavioural data for model fitting. Through numerical simulations I show that a hierarchical approach outperforms alternative fitting methods in situations where limited data are available, such as when quantifying metacognition in patient populations. In addition, the model may be flexibly expanded to estimate parameters encoding other influences on metacognitive efficiency. MATLAB software and documentation for implementing hierarchical meta-d' estimation (HMeta-d) can be downloaded at https://github.com/smfleming/HMeta-d.

Keywords: Bayes; confidence; metacognition; signal detection theory.

Figure 1

The meta-d’ model. (A) The right-hand panel shows schematic confidence-rating distributions conditional on correct and incorrect decisions. A subject with good metacognitive sensitivity will provide higher confidence ratings when they are correct, and lower ratings when incorrect, and these distributions will only weakly overlap (solid lines). Conversely a subject with poorer metacognitive sensitivity will show greater overlap between these distributions (dotted lines). These theoretical correct/error distributions are obtained by ‘folding’ a type 1 SDT model around the criterion [see Galvin et al. (2003), for further details], and normalizing such that the area under each curve sums to 1. The overlap between distributions can be calculated through type 2 ROC analysis (middle panel). The theoretical type 2 ROC is completely determined by an equal-variance Gaussian SDT model; we can therefore invert the model to determine the type 1 d’ that best fits the observed confidence rating data, which is labelled meta-d’. Meta-d’ can be directly compared with the type 1 d’ calculated from the subject’s decisions—if meta-d’ is equal to d’, then the subject approximates the ideal SDT prediction of metacognitive sensitivity. (B) Simulated data from a SDT model with d’ = 2. The y-axis plots the conditional probability of a particular rating given the first-order response is correct (green) or incorrect (red). In the right-hand panel, Gaussian noise has been added to the internal state underpinning the confidence rating (but not the decision) leading to a blurring of the correct/incorrect distributions. Open circles show fits of the meta-d’ model to each simulated dataset.

Figure 2

The hierarchical meta-d’ model. (A) Probabilistic graphical model for estimating metacognitive efficiency using hierarchical Bayes (HMeta-d). The nodes represent all the relevant variables for parameter estimation, and the graph structure is used to indicate dependencies between the variables as indicated by directed arrows. As is convention, unobserved variables are represented without shading and observed variables (in this case, confidence rating counts) are represented with shading. Point estimates for type 1 d’ and criterion are represented as black dots, and the box encloses participant-level parameters subscripted with s. The main text contains a description of each node and its prior distribution. Figure created using the Daft package in Python (http://daft-pgm.org; last accessed 31st August 2016). (B) Prior over the group-level estimate of log(meta-d’/d’) (${\mu }_M$). The solid line shows a kernel density estimate of samples from the prior; the histogram represents empirical meta-d’/d’ estimates obtained from 167 subjects (see main text for details).

Figure 3

HMeta-d output. (A) Example output from HMeta-d fit to simulated data with ground truth meta-d’/d’ fixed at 0.8 for 20 subjects. The left panel shows the first 1000 samples from each of three MCMC chains for parameter ${\mu }_{meta-d^{\mathrm{\text{'}}}/d^{\mathrm{\text{'}}}}$; the right panel shows all samples aggregated in a histogram. (B) Parameter recovery exercise using HMeta-d to fit data simulated from 7 groups of 20 subjects with different levels of meta-d’/d’ = [0.5 0.75 1.0 1.25 1.5 1.75 2]. Error bars denote 95% CI.

Figure 4

Empirical applications of HMeta-d. (A) HMeta-d fits to data from the perceptual metacognition task reported in Fleming et al. (2014). Each histogram represents posterior densities of ${\mu }_{meta-d^{\mathrm{\text{'}}}/d^{\mathrm{\text{'}}}}$ for two groups of subjects: HC = healthy controls; aPFC = anterior prefrontal cortex lesion patients. The right panel shows the difference (in log units) between the group posteriors. The white bar indicates the 95% CI which excludes zero. (B) Example of extending the HMeta-d model to estimate the correlation coefficient $\rho$ between metacognitive efficiencies in two domains. The dotted line shows the ground-truth correlation between pairs of meta-d’/d’ values for 100 simulated subjects.

Figure 5

Simulation experiments—medium metacognitive efficiency (meta-d’/d’ = 1). (A and B) Estimated meta-d’/d’ ratio for different fitting procedures while varying (A) d’ values or (B) type 2 criteria placements. Each data point reflects the average of 100 simulations each with N = 20 subjects. Error bars reflect standard errors of the mean. The ground truth value of meta-d’/d’ is shown by the dotted line.

Figure 6

Simulation experiments—low metacognitive efficiency (meta-d’/d’ = 0.5). For legend see Fig. 5.

Figure 7

Simulation experiments—high metacognitive efficiency (meta-d’/d’ = 1.5). For legend see Fig. 5.

Figure 8

Observed false positive rates for each fitting procedure. Average false positive rates for hypothesis tests against ground truth meta-d’/d’ values from the simulations in Figs 5–7. Individual data points reflect single experiments (the false positive rate for a particular combination of metacognitive efficiency level, parameters and trial count). Error bars reflect standard errors of the mean. For trial counts < 200, MLE or SSE methods result in unacceptably high false positive rates due to consistent over- or underestimation of metacognitive efficiency.