Response to Ruby et al: On a 'failed' attempt to manipulate conscious perception with transcranial magnetic stimulation to prefrontal cortex

Abstract

Does disruption of prefrontal cortical activity using transcranial magnetic stimulation (TMS) impair visual metacognition? An initial study supporting this idea (Rounis, Maniscalco, Rothwell, Passingham, & Lau, 2010) motivated an attempted replication and extension (Bor, Schwartzman, Barrett, & Seth, 2017). Bor et al. failed to replicate the initial study, concluding that there was not good evidence that TMS to dorsolateral prefrontal cortex impairs visual metacognition. This failed replication has recently been critiqued by some of the authors of the initial study (Ruby, Maniscalco, & Peters, 2018). Here we argue that these criticisms are misplaced. In our response, we encounter some more general issues concerning good practice in replication of cognitive neuroscience studies, and in setting criteria for excluding data when employing statistical analyses like signal detection theory. We look forward to further studies investigating the role of prefrontal cortex in metacognition, with increasingly refined methodologies, motivated by the discussions in this series of papers.

Keywords: Consciousness; Dorsolateral prefrontal cortex; Failed replication; Metacognition; Null result; Perception; Signal detection theory; Transcranial magnetic stimulation.

Fig. 1

Simulated data demonstrating the behaviour of meta-d’ under systematic variation of Type II false alarm and hit rates. For both graphs, meta d’ balance (an analytical version of meta d’) is a solid line, and meta d’ SSE is the dashed line. Dashed-dotted lines show d’. The vertical dashed line in this simulation is the boundary where the measure becomes unreliable (clearly non-linear). (A) Shows meta-d’ against type II false alarm rate. (B) Shows meta-d’ against type II hit rate. Note that, particularly for meta d’ SSE, for high type II hit rates, meta d’ becomes unreliable. For parameter and model details, see Fig. 5 from Barrett et al. (2013). Reproduced with permission.

Fig. 2

Each datapoint is the most extreme value (absolute distance from 0.5) out of 4 possible values (type I/II HR/FAR) for one session for each subject from experiment 1 of Bor et al. (2017) (180 datapoints, 90 subjects x 2 sessions). The extent of this most extreme value on the x axis (more extreme on the right) is plotted against meta d’ minus d’ (SSE) on the y axis. The vertical dotted line is the boundary by which we excluded subjects with values to the right of this in Bor et al. (2017), in order to exclude type I/II HR/FAR values too extreme to be to generate a reliable meta d’ - d’ measure.