Explicit encoding of multimodal percepts by single neurons in the human brain

Abstract

Different pictures of Marilyn Monroe can evoke the same percept, even if greatly modified as in Andy Warhol's famous portraits. But how does the brain recognize highly variable pictures as the same percept? Various studies have provided insights into how visual information is processed along the "ventral pathway," via both single-cell recordings in monkeys and functional imaging in humans. Interestingly, in humans, the same "concept" of Marilyn Monroe can be evoked with other stimulus modalities, for instance by hearing or reading her name. Brain imaging studies have identified cortical areas selective to voices and visual word forms. However, how visual, text, and sound information can elicit a unique percept is still largely unknown. By using presentations of pictures and of spoken and written names, we show that (1) single neurons in the human medial temporal lobe (MTL) respond selectively to representations of the same individual across different sensory modalities; (2) the degree of multimodal invariance increases along the hierarchical structure within the MTL; and (3) such neuronal representations can be generated within less than a day or two. These results demonstrate that single neurons can encode percepts in an explicit, selective, and invariant manner, even if evoked by different sensory modalities.

Figure 1. Example of a Neuron with Multimodal Responses to Oprah Winfrey

(A) A neuron in the hippocampus that responded selectively to pictures of the television host Oprah Winfrey (stimulus 40, 39, and 11), as well as to her written (stimulus 56) and spoken (stimulus 73) name. To a lesser degree, the neuron also fired to Whoopi Goldberg. They were no responses to any other picture, sound, or text presentations. For space reasons, only the largest 30 (out of 78) responses are displayed. In each case the raster plots for the six trials, peri-stimulus time histograms (PSTH) and the corresponding pictures are shown. The vertical dotted lines mark picture onset and offset, 1 s apart. (B) Median number of spikes (across trials) for all stimuli. Presentations of Oprah Winfrey are marked with red bars. Stimulus numbers corresponds to the ones shown above each picture in (A). The gray horizontal line shows the 5 SD above the baseline threshold used for defining significant responses.

Figure 2. Example of a Neuron with Multimodal Responses to Saddam Hussein

A neuron in the entorhinal cortex that fired selectively to three pictures of Saddam Hussein, his written name, and his spoken name. Conventions are the same as for Figure 1. The black and gray horizontal lines show the mean baseline firing and the threshold for defining significant responses, respectively.

Figure 3. Example of a Neuron with Multimodal Responses to a UCLA Researcher

A neuron in the amygdala selectively activated by pictures, sound, and text presentations of Arne, one of the researchers doing experiments at UCLA. This person was unknown to the patient before the experiments took place. From the same recording electrode, a second neuron firing to pictures of Michael Douglas and a multiunit firing to pictures of a family member of the patient were isolated after spike sorting (see Figures S1 and S6), thus showing that nearby neurons can have very different responses.

Figure 4. Population Results

(A) Percentage of responsive units, units with visual invariance, and units with responses to sound and text presentations. Anatomical connections between the different MTL areas and with visual and auditory cortices in the macaque monkey [10, 11] are marked with light gray and orange arrows, respectively. Note the increasing degree of visual invariance and number of responses to sound and text presentations along the hierarchical structure of the MTL, with the largest relative number of responses in the hippocampus and entorhinal cortex and the lowest in the parahippocampal cortex. (B) Normalized firing rates to picture, text, and sound presentations for all 17 neurons with triple invariance (i.e., with responses to picture, sound, and text). Bands around the average values denote SEM.