Neural and behavioral signatures of the multidimensionality of manipulable object processing

Abstract

Understanding how we recognize objects requires unravelling the variables that govern the way we think about objects and the neural organization of object representations. A tenable hypothesis is that the organization of object knowledge follows key object-related dimensions. Here, we explored, behaviorally and neurally, the multidimensionality of object processing. We focused on within-domain object information as a proxy for the decisions we typically engage in our daily lives - e.g., identifying a hammer in the context of other tools. We extracted object-related dimensions from subjective human judgments on a set of manipulable objects. We show that the extracted dimensions are cognitively interpretable and relevant - i.e., participants are able to consistently label them, and these dimensions can guide object categorization; and are important for the neural organization of knowledge - i.e., they predict neural signals elicited by manipulable objects. This shows that multidimensionality is a hallmark of the organization of manipulable object knowledge.

Fig. 1. Experimental procedures and analysis pipeline.

A In order to extract object-related dimensions we collected similarity measures between our 80 manipulable objects through a pile-sorting experiment^,. Per individual (N = 60 particpants), we obtained a piling solution for each of the knowledge types (function, manipulation and vision) whereby objects piled together were similar to one another but different from objects in other piles. These piling solutions were coded into dichotomous matrices that represented pile membership. Participant-specific matrices were then averaged and transformed into a dissimilarity matrix (one per knowledge type). Finally, we used non-metric MDS^, to extract dimensions independently per knowledge type. B We wanted to test whether the obtained object-related dimensions were cognitively important for perceiving objects. Firstly, we had a different set of participants perform a label generation task for each dimension. Participants were presented with 20 objects – 10 from each of the extremes of the target dimension – and were asked to provide up to 5 labels that best explained the difference between the two sets of objects. Label frequency was used to select a label for each of the extracted dimensions. We further tested the importance of these object-related dimensions by having yet another set of participants learn to categorize objects according to their scores in each of the dimensions. Participants went first through 2 experimental phases where they were taught to categorize a subset of the objects in terms of whether they were close to one of the two extremes of a target dimension and were given clear feedback as to the correct responses. Importantly, in a third phase, they were asked to categorize all objects, including a subset of untrained objects, and were not given any feedback. Moreover, we added two control dimensions. In one of these controls, we took one of the real dimensions and randomly shuffled the scores of the dimension for the individual objects. For a more stringent control, we took lexical frequency values – i.e., count of the times a particular lexical entry appears in a text corpus per million – for each of the objects and rank ordered them in terms of these values. We used these dimensions to control for reliable generalization of object-related dimensional learning to untrained items. We tested whether participants generalized their learning to untrained items. Percent response performance towards the extreme object with the highest score in the dimension was calculated and fitted with a cumulative Gaussian curve. C Finally, we tested whether the object-related dimensions extracted were able to explain neural responses to objects. We presented the 80 objects in an event-related fMRI experiment using greyscale images, and participants had to categorize each image as either a manipulable object or an animal (the catch trials). We then used parametric mapping to analyze the fMRI data, and tested whether our object-related dimensions could explain the neural responses elicited by the 80 manipulable objects. We used parametric analysis^– over the fMRI data, and cast our key dimensions as first-order (i.e., linear) parametric modulators in a General Linear Model (GLM). That is, for each stimulus in the design matrix, the corresponding dimensional scores were assigned as modulation values.

Fig. 2. Interpretability of the objected-related dimensions.

Here, we present the selected 15 dimensions that govern the internal representation of our set of manipulable objects. Per dimension, we present the 20 objects with the most extreme scores (10 from each extreme). Labels were then selected based on the frequency of label generation by the participants. On the left side of each dimension, we present the collated selected label, as well as the percent generation score for that label (see Figure S3 for the label frequency plots).

Fig. 3. Supervised learning of object-related dimensions in a categorization task.

A–D Here we show percent response towards the object with the highest score for each of the object-related dimensions and the two control dimensions (i.e., towards the extreme with the highest score). Percent responses were averaged within each of the ten bins, and a cumulative Gaussian curve was fitted on the data of each individual. The presented plots are based on the average of all participants. Error bars correspond to SEM (N = 10 participants per dimension in a total of 270 participants); depicted cumulative Gaussian curve was fit on the average percent results for visualization purposes; E–G Violin plots of the R-square values of the Gaussian fit for each dimension for each participant.

Fig. 4. Neural effects of the object-related dimensions.

Here, we show an overlap map with all object-related dimension (F-maps per dimension against zero – i.e., against no modulation) per knowledge type, each dimension color coded by the colors in Fig. 3. Black corresponds to areas where at least two individual dimensions overlap (all individual F-maps cluster-forming height-thresholded at p < 0.001 – except for the dimensions “Cooking vs. Sports” and “Material properties” that are cluster-forming height-thresholded at p < 0.005 – and all corrected at FDR p < 0.05; all N = 26 participants).

Fig. 5. Content specificity in the neural responses.

Here, we show modulatory effects (i.e., the beta values of each dimension that are significantly different from zero) of the first two dimensions of each knowledge type (all individual F-maps cluster-forming height-thresholded at p < 0.001 – except for the dimension “Cooking vs. Sports that is height-thresholded at p < 0.005 – and corrected at FDR p < 0.05; all N = 26). In yellow we present the first dimensions of each knowledge type (“Cooking vs. Sports”; “Power vs. Precision”; “Metal vs. Other Materials”), whereas in light blue we present the second dimensions of each knowledge type (“Kitchen vs. Office”; “Dexterity vs. Force”; “Elongation vs. Round”). Each map shows voxels where signal is explained by the dimension when compared to no modulation (i.e., when compared to 0). Black corresponds to areas of overlap between the two dimensions presented per map.