Recursive hierarchical embedding in vision is impaired by posterior middle temporal gyrus lesions

Abstract

The generation of hierarchical structures is central to language, music and complex action. Understanding this capacity and its potential impairments requires mapping its underlying cognitive processes to the respective neuronal underpinnings. In language, left inferior frontal gyrus and left posterior temporal cortex (superior temporal sulcus/middle temporal gyrus) are considered hubs for syntactic processing. However, it is unclear whether these regions support computations specific to language or more generally support analyses of hierarchical structure. Here, we address this issue by investigating hierarchical processing in a non-linguistic task. We test the ability to represent recursive hierarchical embedding in the visual domain by contrasting a recursion task with an iteration task. The recursion task requires participants to correctly identify continuations of a hierarchy generating procedure, while the iteration task applies a serial procedure that does not generate new hierarchical levels. In a lesion-based approach, we asked 44 patients with left hemispheric chronic brain lesion to perform recursion and iteration tasks. We modelled accuracies and response times with a drift diffusion model and for each participant obtained parametric estimates for the velocity of information accumulation (drift rates) and for the amount of information accumulated before a decision (boundary separation). We then used these estimates in lesion-behaviour analyses to investigate how brain lesions affect specific aspects of recursive hierarchical embedding. We found that lesions in the posterior temporal cortex decreased drift rate in recursive hierarchical embedding, suggesting an impaired process of rule extraction from recursive structures. Moreover, lesions in inferior temporal gyrus decreased boundary separation. The latter finding does not survive conservative correction but suggests a shift in the decision criterion. As patients also participated in a grammar comprehension experiment, we performed explorative correlation-analyses and found that visual and linguistic recursive hierarchical embedding accuracies are correlated when the latter is instantiated as sentences with two nested embedding levels. While the roles of the inferior temporal gyrus and posterior temporal cortex in linguistic processes are well established, here we show that posterior temporal cortex lesions slow information accumulation (drift rate) in the visual domain. This suggests that posterior temporal cortex is essential to acquire the (knowledge) representations necessary to parse recursive hierarchical embedding in visual structures, a finding mimicking language acquisition in young children. On the contrary, inferior frontal gyrus lesions seem to affect recursive hierarchical embedding processing by interfering with more general cognitive control (boundary separation). This interesting separation of roles, rooted on a domain-general taxonomy, raises the question of whether such cognitive framing is also applicable to other domains.

Keywords: hierarchy; inferior frontal gyrus; lesion; syntax; visuospatial.

Figure 1

Experimental paradigm. (A) The presentation of the visual recursion/iteration task (REC, ITE) comprised four steps including a successive presentation of the steps 1–3 at the top of the screen to then present the two options for a forced choice at the bottom. Examples for REC and ITE screen shots are provided for step 4; note that the final choice images are identical for both tasks. Location of correct image was randomized (e.g. left in the ITE and right in the REC example provided). (B) Examples of fractals used in REC. There were different categories of ‘visual complexity’—3, 4 and 5—and different categories of foils. In ‘odd constituent’ foils, two elements within the whole hierarchy were misplaced; in ‘positional error’ foils, all elements within new hierarchical levels were internally consistent, but inconsistent with the previous iterations; in ‘repetition’ foils, no additional iterative step was performed after the third iteration.

Figure 2

Behavioural data. (A) Percentage of correct answers (acc [%], left) and response time (RT [s], right) in the ITE (blue), and REC (red). The order of visual tasks was either ‘I→R’ or ‘R→I’ as indicated by the light or darker shading. (B) We combined these data into a hierarchical DDM (text for details) and obtained posterior estimates for drift rate (drift v′) and boundary separation (boundary a′). Note that order no longer influences the performance since roughly equal numbers of patients showed values REC>ITE and ITE>REC for these measures (colour coding of individual data points as in A). The variability can be used across participants in the lesion-behaviour analysis (main text and Fig. 3). For detailed descriptive statistics, see Supplementary Table 2.

Figure 3

Lesion-behaviour analyses. (A) The area covered. Left: Coloured areas show a lesion in at least one patient, in the lighter area at least four lesions overlap representing the area in which the analysis was performed; right: area of maximal overlap (n = 15) projecting to the insular cortex as is typically seen in stroke dominated lesion studies. (B) Voxel-wise approach: Uncorrected (unc.) maps are shown for boundary separation (a′, red) and drift rate (v′, purple) for the REC, with ITE as nuisance variable. IFG lesions were associated with lower a′, meaning that participants collected less information before reaching a decision. On the other hand, lesions in the MTG and STG were associated with lower drift rate, meaning that these patients collected information slower. Only 39 voxels in the MTG (blue area circled for illustration purpose) survived correction for REC v′. (C) Statistical region of interest-symptom mapping, shows significant correlations between REC v′ and MTG for the AICHA (purple) and the Brodmann atlas (BA21, blue).

Figure 4

Relationship between REC, ITE and grammar task. (A) Example of the linguistic task of a different study in which the majority of the participants took part. Regarding the tasks reported here (ITE/REC) performance for one aspect of syntactic complexity, namely embedding, was correlated with results in the visual task. The three syntactic propositions were presented either sequentially (E0, i.e. no embedding) or with an embedded relative clause (EMB1, one embedding containing two propositions) or with a 2-fold centre-embedded structure (EMB2). Note that for the three example sentences (out of n = 132) the same image would be correct in the successive picture selection task (see screenshot in Supplementary Fig. 2). The task required selection of the correct picture from a set of four (one correct and three distractors for each proposition). (B) Scatterplots depicting the relationship between accuracy in the visual tasks (REC and ITE) and in the grammar tasks EMB1 (left) and EMB2 (right). Correlation coefficients are presented in the text. See also Table 1.