Medial axis shape coding in macaque inferotemporal cortex

Abstract

The basic, still unanswered question about visual object representation is this: what specific information is encoded by neural signals? Theorists have long predicted that neurons would encode medial axis or skeletal object shape, yet recent studies reveal instead neural coding of boundary or surface shape. Here, we addressed this theoretical/experimental disconnect, using adaptive shape sampling to demonstrate explicit coding of medial axis shape in high-level object cortex (macaque monkey inferotemporal cortex or IT). Our metric shape analyses revealed a coding continuum, along which most neurons represent a configuration of both medial axis and surface components. Thus, IT response functions embody a rich basis set for simultaneously representing skeletal and external shape of complex objects. This would be especially useful for representing biological shapes, which are often characterized by both complex, articulated skeletal structure and specific surface features.

Figure 1

Adaptive shape sampling example. See text for details. (A) 1^st, 3^rd, and 5^th generations of a medial axis lineage (left, M1.1, M1.3, M1.5) and a surface shape lineage (right, S1.1, S1.3, S1.5). Stimuli are ordered within each generation by average response strength. Average response rate is indicated by background color (see scale bar). (B) 6^th, 8^th, and 10^th generations of the original medial axis lineage (left, M1.6, M1.8, M1.10) and 1^st, 5^th, and 10^th generations of a second, independent medial axis lineage (right, M2.1, M2.5, M2.10). (C) Partial family trees exemplifying shape evolution within the first (left) and second (right) medial axis lineages.

Figure 2

Medial axis template model example (for Figure 1 neuron). (A) Optimum template, in this case from second lineage (outline, upper right). The source stimulus is shown (upper left) along with its complete set of substructures. (B) Highest similarity substructures (colored outlines) for three example high response stimuli (top row) and three low response stimuli (bottom row) from the same (second) lineage. Average observed response rates are indicated by inset numbers, template similarity values are indicated by color (see scale bar). (C) Highest similarity substructures for high response (top row) and low response (bottom row) stimuli from the first lineage. (D–F) Optimum template based on both lineages, presented as in A–C.

Figure 3

Response strength comparison for medial axis and surface stimuli. (A) Scatter plot of mean response across top ten stimuli in each domain (n=111). (B) Histogram of Wilcoxon rank sum statistics testing whether responses to top ten medial axis stimuli were higher than responses to top ten surface stimuli. Filled bars indicate significantly (p < 0.05) higher medial axis responses (right, n = 40) or higher surface responses (left, n = 29).

Figure 4

Surface tuning example. (A, B) Selected generations from two medial axis lineages and one surface lineage. Details as in Figure 1. (C) Optimum surface template model (based on correlation with response rates in both lineages), projected onto the template source stimulus (left) and schematized (right) as a combination of two surface fragment icons (green, cyan) positioned relative to object center (cross). The acute convex point (green) is enlarged for visibility. (D) Surface template similarity (see scale bar) for three example high response stimuli (top row) and three low response stimuli (bottom row) from the first medial axis lineage. (E) Surface template similarity for high and low response stimuli from the second medial axis lineage.

Figure 5

Combined medial axis / surface tuning example. (A, B) Selected generations from two medial axis lineages and one surface lineage. Details as in Figure 1. (C) Optimum combined templates based on a single lineage (left) and based on both lineages (right). In each case, the black outline represents the medial axis template and the green and cyan surfaces represent the surface template. (D) Template similarity values (see scale bars) for example high (top row) and low (bottom row) response stimuli from the first medial axis lineage. (E) Template similarity values for example stimuli from the second medial axis lineage.

Figure 6

(A) Distribution of axial weights (horizontal axis) and nonlinear weights (vertical axis) in the composite medial axis / surface tuning models. (B–D) Example tests of axial tuning consistency. These tests were based on one high (top rows), one medium (middle rows), and one low (bottom rows) response stimulus drawn from the adaptive tests. The original stimuli (left column) were morphed with six different radius profiles (columns 2–7) to alter surface shape while maintaining medial axis shape. (E) Scatter plot of response invariance (horizontal axis) vs. axial tuning consistency (vertical axis). Our index of invariance is explained in Experimental Procedures. Consistency was defined as the fraction of variance explained by the first component of a singular value decomposition model, which measures the separability of axial and surface tuning.

Figure 7

Composite medial axis / surface tuning models. Models constrained by both lineages are shown for 59/66 neurons with significant cross-validation (r test, p < 0.005). For each neuron, the optimum model is projected onto a high response stimulus from the first medial axis lineage (left column) and a high response stimulus from the second lineage (right column). The model projections are shown in the top row, and the original shaded stimuli are shown in the bottom row. Medial axis and surface template similarity values are indicated by color (see scale bars). Models are arranged by decreasing medial axis weight from upper left to lower right.

Figure 8

3D rotation test. (A) Example results, for the Figure 1 neuron. Average response level (see scale bar) was measured for versions of the highest response stimulus from the adaptive sampling procedure rotated across a 180° range in 10° increments around the x (top), y (middle), and z (bottom) axes. (B) Same data as in (A), for rotation around the x (red), y (green), and z (blue) axes (error bars indicate +/- s.e.m.), compared to average response across all stimuli in the main experiment (black, dashed lines indicate +/- s.e.m.). (C) Distribution of tolerances for rotation around the x (top), y (middle), and z (bottom) axes. Tolerance was defined as the width in degrees of the range over which responses to the highest response stimulus remained significantly (t test, p < 0.05) greater than the average response to random stimuli tested as part of the adaptive sampling experiment.