Self-Relevance Predicts the Aesthetic Appeal of Real and Synthetic Artworks Generated via Neural Style Transfer

Abstract

What determines the aesthetic appeal of artworks? Recent work suggests that aesthetic appeal can, to some extent, be predicted from a visual artwork's image features. Yet a large fraction of variance in aesthetic ratings remains unexplained and may relate to individual preferences. We hypothesized that an artwork's aesthetic appeal depends strongly on self-relevance. In a first study (N = 33 adults, online replication N = 208), rated aesthetic appeal for real artworks was positively predicted by rated self-relevance. In a second experiment (N = 45 online), we created synthetic, self-relevant artworks using deep neural networks that transferred the style of existing artworks to photographs. Style transfer was applied to self-relevant photographs selected to reflect participant-specific attributes such as autobiographical memories. Self-relevant, synthetic artworks were rated as more aesthetically appealing than matched control images, at a level similar to human-made artworks. Thus, self-relevance is a key determinant of aesthetic appeal, independent of artistic skill and image features.

Keywords: aesthetic valuation; artwork; identity; machine learning; open data.

Fig. 1. Predicting aesthetic ratings from ratings of self-relevance and familiarity.

(a) For Experiment (Exp.) 1A, self-relevance ratings predicted aesthetic appeal with an average slope of 0.36 (N = 32). (b) For Experiment 1B, self-relevance ratings predicted aesthetic appeal (“feeling of beauty”) with an average slope of 0.31 (N = 208). (c) For Experiment 2, self-relevance ratings predicted aesthetic appeal with an average slope of 0.35 (N = 40). (d) In Experiment 2, familiarity also predicted ratings of aesthetic appeal, with the larger effect for ratings of recognized (R) or familiar (F) artworks versus unfamiliar (U) artworks. There was a smaller but significant effect for recognized versus familiar artworks. Rec = definitely recognized; Fam = familiar; Un = unknown (N = 40). (e) A mediation model of Experiment 2 data shows that the total effect of self-relevance on aesthetic appeal c was only partially mediated by familiarity (paths a and b), leaving a strong direct effect (c – ab). Thick dark blue (a–c) and red (d) lines indicate the average linear slope (with the standard error of the estimated slope in gray), and the thin light blue (a–c) and pink (d) lines show linear slopes for individual participants.

Fig. 2. Generating self-relevant artworks using a “style transfer” convolutional neural network.

(a) In a first session, participants responded to a Cultural Background and Lifestyle Questionnaire that inquired about specific autobiographical memories, aspects of identity, interests, preferences, and common activities. Images that contained content relevant to each observer’s individual responses were then sourced from the Internet. A style of an existing artwork was then transferred to the image using a style-transfer network based on the work by Huang and Belongie (2017) that consisted of an encoder network, adaptive instance normalization, and a decoder network, resulting in a new synthetic artwork with customized content. (b) In a second session, observers were shown artworks from four conditions: (1) self-relevant artworks custom generated on the basis of their questionnaire responses, (2) other-relevant artworks generated for a matched participant, (3) a control set of generated artworks, and (4) a set of real artworks. There were 20 artworks in each condition. VGG = Visual Geometry Group, Oxford, UK. Real Artwork credit: Bob Thompson (1937–1966), Homage to Nina Simone, 1965, oil on canvas, 48 × 72 1/8 inches; Collection of The Minneapolis Institute of Art; © Michael Rosenfeld Gallery LLC, New York, NY; Courtesy of Michael Rosenfeld Gallery LLC, New York, NY. Reprinted with permission. Content Source credit: Photo of Virtual Helsinki by VR-Studio Zoan.

Fig. 3

Effect of stimulus condition on ratings of (a) self-relevance and (b) aesthetic appeal in Experiment 2. Colored dots show average ratings for individual participants in each condition, after centering by participant, and shaded area indicates data density, smoothed with a gaussian kernel and trimmed. Black dots indicate the means, and error bars indicate 95% confidence intervals. Black horizontal brackets indicate significance. N = 40 participants.

Fig. 4

Artworks reflecting one’s self-construct were more appealing. Artworks in the self-relevant condition that reflected specific autobiographical (Autobio.) memories (238 items), identity (26 items), expressed preferences (87 items), and interests (86 items) were rated as significantly more aesthetically appealing than generated-control artworks (baseline, 800 items), whereas artworks reflecting common activities (39 items) or those derived from questions reflecting “mixed” (324 items) aspects of self-relevance were not. Black bars indicate 95% confidence intervals. N = 40 participants.