Decoding actions at different levels of abstraction

Abstract

Brain regions that mediate action understanding must contain representations that are action specific and at the same time tolerate a wide range of perceptual variance. Whereas progress has been made in understanding such generalization mechanisms in the object domain, the neural mechanisms to conceptualize actions remain unknown. In particular, there is ongoing dissent between motor-centric and cognitive accounts whether premotor cortex or brain regions in closer relation to perceptual systems, i.e., lateral occipitotemporal cortex, contain neural populations with such mapping properties. To date, it is unclear to which degree action-specific representations in these brain regions generalize from concrete action instantiations to abstract action concepts. However, such information would be crucial to differentiate between motor and cognitive theories. Using ROI-based and searchlight-based fMRI multivoxel pattern decoding, we sought brain regions in human cortex that manage the balancing act between specificity and generality. We investigated a concrete level that distinguishes actions based on perceptual features (e.g., opening vs closing a specific bottle), an intermediate level that generalizes across movement kinematics and specific objects involved in the action (e.g., opening different bottles with cork or screw cap), and an abstract level that additionally generalizes across object category (e.g., opening bottles or boxes). We demonstrate that the inferior parietal and occipitotemporal cortex code actions at abstract levels whereas the premotor cortex codes actions at the concrete level only. Hence, occipitotemporal, but not premotor, regions fulfill the necessary criteria for action understanding. This result is compatible with cognitive theories but strongly undermines motor theories of action understanding.

Keywords: MVPA; action concepts; action understanding; fMRI; motor system; occipitotemporal cortex.

Figure 1.

A, Investigated levels of abstraction. The concrete level (red) describes actions based on perceptual stimulus properties like concrete kinematics and object exemplars involved in the action. The intermediate level (green) generalizes across kinematics and object exemplars. The abstract level (blue) generalizes across kinematics and object category. B, Decoding scheme. Different abstraction levels were isolated by training a classifier to discriminate the opening and closing of a specific bottle or box and tested it using actions involving either the same object (concrete), a different object from the same object category (intermediate), or an object from a different object category (abstract; see Materials and Methods for details of the procedure). Expected patterns of results for different regions of coding actions at concrete but not intermediate and abstract levels (C); at concrete, intermediate, and abstract levels (D); and at the abstract level only (E). Dotted line represents decoding accuracy at chance = 50% (for details, see Materials and Methods, ROI MVPA).

Figure 2.

ROI MVPA results. Mean classification accuracies for decoding at concrete (red), intermediate (green), and abstract (blue) levels. Error bars indicate SEM, asterisks indicate statistical significance (different from 50% = chance, red = FDR corrected for the number of tests). Dotted line represents decoding accuracy at chance = 50%.

Figure 3.

Mean accuracy maps of the searchlight MVPA at each abstraction level (concrete, intermediate, and abstract). Individual accuracy maps were cortex-based aligned, averaged, and projected onto a common group surface (both flat maps and lateral views of inflated hemispheres). Decoding accuracy at chance is 50%. CS, central sulcus; IFS, inferior frontal sulcus; IPS, intraparietal sulcus; ITS, inferior temporal sulcus; PrCS, precentral sulcus; PoCS, postcentral sulcus; SFS, superior frontal sulcus; STS, superior temporal sulcus.

Figure 4.

Statistical maps of the searchlight MVPA. For intermediate and abstract levels, conjunctions (i.e., lowest common t value per vertex) of concrete/intermediate and concrete/intermediate/abstract levels, respectively, were used (see Materials and Methods for details). Alignment and projection procedures are the same as in Figure 3. Outlines around clusters indicate clusters surviving cluster size correction (dark red, concrete; dark green, intermediate; dark blue, abstract; thresholded at p = 0.005, corrected cluster threshold p = 0.05). CS, central sulcus; IFS, inferior frontal sulcus; IPS, intraparietal sulcus; ITS, inferior temporal sulcus; PrCS, precentral sulcus; PoCS, postcentral sulcus; SFS, superior frontal sulcus; STS, superior temporal sulcus.