The role of low-level image features in the affective categorization of rapidly presented scenes

Abstract

It remains unclear how the visual system is able to extract affective content from complex scenes even with extremely brief (< 100 millisecond) exposures. One possibility, suggested by findings in machine vision, is that low-level features such as unlocalized, two-dimensional (2-D) Fourier spectra can be diagnostic of scene content. To determine whether Fourier image amplitude carries any information about the affective quality of scenes, we first validated the existence of image category differences through a support vector machine (SVM) model that was able to discriminate our intact aversive and neutral images with ~ 70% accuracy using amplitude-only features as inputs. This model allowed us to confirm that scenes belonging to different affective categories could be mathematically distinguished on the basis of amplitude spectra alone. The next question is whether these same features are also exploited by the human visual system. Subsequently, we tested observers' rapid classification of affective and neutral naturalistic scenes, presented briefly (~33.3 ms) and backward masked with synthetic textures. We tested categorization accuracy across three distinct experimental conditions, using: (i) original images, (ii) images having their amplitude spectra swapped within a single affective image category (e.g., an aversive image whose amplitude spectrum has been swapped with another aversive image) or (iii) images having their amplitude spectra swapped between affective categories (e.g., an aversive image containing the amplitude spectrum of a neutral image). Despite its discriminative potential, the human visual system does not seem to use Fourier amplitude differences as the chief strategy for affectively categorizing scenes at a glance. The contribution of image amplitude to affective categorization is largely dependent on interactions with the phase spectrum, although it is impossible to completely rule out a residual role for unlocalized 2-D amplitude measures.

Fig 1

The upper row depicts an intact scene (top left) and the same scene with the phase spectrum left intact but with amplitude spectra wholly randomized (top right). The lower panel depicts the local phase congruency (LPC) map of the intact image (bottom left), where darker greys indicate relatively lower phase congruency and lighter grays indicate higher LPC, corresponding to local edge and corner detail. The bottom right panel illustrates variants of the original image with an unaltered amplitude spectrum, though with varying amounts of phase randomization. Note the degradation of edge and corner detail as phase randomization increases.

Fig 2. Panel A illustrates sample intact images alongside the distinct image manipulations, involving within (ID) and between (BTW) semantic category swapping of 2-D amplitude spectra.

Panel B illustrates the local phase congruency (average of top 10 LPC values) for images in the intact, ID, and BTW conditions with the median for each group indicated by the red bar. Panel C illustrates the sequence for one rapid categorization trial. Note that the image exemplars depicted here are shown for illustrative purposes only and were not ones used in the actual experiment.

Fig 3. The upper panel illustrates the support vector machine (SVM) classification sequence for a representative mutilation and paired neutral image.

First, one half of the original images were subjected to a two-dimensional fast Fourier transformation (2-D FFT). Phase spectra were discarded and the SVM was trained with 80 amplitude-only features per image (20 spatial frequencies, four orientations). After training, the SVM was tested on the remaining half of the images. The lower panel illustrates the averaged spectral contour plots of our intact mutilation, disgust, and neutral images. The inner contour represents 60% and the outer contours 80 and 90%, respectively, of image energy.

Fig 4

Panels A and B depict perceptual sensitivity (d’) values in each of the image conditions, for the mutilation and disgust blocks. Horizontal lines indicate chance performance. Error bars are 95% confidence intervals.

Fig 5. Neutral and aversive image ratings plotted in a 2-D affective space with valence and arousal each rated between 1 and 9 (with a valence rating of 5.0 being perfectly neutral).

Each dot represents the mean rating for one participant for the indicated affective category.