Predicting and Grouping Digitized Paintings by Style using Unsupervised Feature Learning

Abstract

Objective: To create an system to aid in the analysis of art history by classifying and grouping digitized paintings based on stylistic features automatically learned without prior knowledge.

Material and methods: 6,776 digitized paintings from 8 different artistic styles (Art Nouveau, Baroque, Expressionism, Impressionism, Realism, Romanticism, Renaissance, and Post-Impressionism) were utilized to classify (predict) and cluster (group) paintings according to style. The method of unsupervised feature learning with K-means (UFLK), inspired by deep learning, was utilized to extract features from the paintings. These features were then used in: (1) a support vector machine algorithm to classify the style of new test paintings based on a training set of paintings having known style labels; and (2) a spectral clustering algorithm to group the paintings into distinct style groups (anonymously, without employing any known style labels). Classification performance was determined by accuracy and F-score. Clustering performance was determined by: 1) the ability to recover the original stylistic groupings (using a cost analysis of all possible combinations of 8 group label assignments); 2) F-score; and 3) a reliability analysis. The latter analysis used two novel ways to determine the distribution of the null-hypothesis: (a) a uniform distribution projected onto the principal components of the original data; and (b) a randomized, weighted adjacency matrix. The ability to gain insights into art was tested by a semantic analysis of the clustering results. For this purpose, we represented the featural characteristics of each painting by an N-dimensional feature vector, and plotted the distance between vector endpoints (i.e., similarity between paintings). Then, we color-coded the endpoints with the assigned lowest-cost style labels. The scatterplot was visually inspected for separation of the paintings, where the amount of separation between color clusters provides semantic information on the interrelatedness between styles.

Results: The UFLK-extracted features resembled the edges/lines/colors in the paintings. For feature-based classification of paintings, the macro-averaged F-score was 0.469. Classification accuracy and F-score were similar/higher compared to other classification methods using more complex feature learning models (e.g., convolutional neural networks, a supervised algorithm). The clustering via UFLK-extracted features yielded 8 unlabeled style groupings. In 6 of 8 clusters, the most common true painting style matched the cluster style assigned by cost analysis. The clustering had an F-score of 0.212. (There are no comparison methods for clustering paintings.) For the semantic analysis, the featural characteristics of Baroque and Art Nouveau were found to be similar, indicating a relationship between these styles.

Discussion/conclusion: The UFLK method can extract features from digitized paintings. We were able to extract characteristics of art without any prior information about the nature of the features or the stylistic designation of the paintings. The methods herein may provide art researchers with the latest computational techniques for the documentation, interpretation, and forensics of art. The tools could assist the preservation of culturally sensitive works of art for future generations, and provide new insights into works of art and the artists who created them.

Keywords: classification; clustering; k-means; painting styles; unsupervised feature learning.

Fig. 1. Processing pipeline of painting style grouping and prediction

First, the paintings, without using any stylistic or feature information, are input into one of three feature extraction processes: (1) nonlinear feature extraction with UFLK, (2) linear feature extraction with PCA, and (3) no feature extraction in which only the raw pixels are used. Then, the paintings, which are now represented either in feature space or in native pixels space, are input into a cluster or classification algorithm. The output from the cluster algorithm is grouping of paintings according to style. The output from the classification is a prediction of style for each individual painting.

Fig. 2. Types of painting styles

Representative paintings of each style type employed in this study are displayed.

Fig. 3. Visualization of UFLK and PCA features for Scene from Tahitian Life

(A) It is demonstrated at the top left that the extracted UFLK features resemble lines, edges and solid blocks of colors. The UFLK features are displayed as an image composed of a collection of 400 features organized into 20 × 20 grid. Each element in the grid contains a feature with dimension size of 6 × 6 pixels with RGB color channels. On the top right, Paul Gauguin’s Scene from Tahitian Life is projected onto a few selected UFLK features, where it is demonstrated that the UFLK features highlight aspects such as the women’s skirts or the background from the painting. (B) On the bottom left, the extracted PCA features are more similar to undefined blobs, with not much color differentiation. Here the features are displayed in 20 × 20 grid, each consisting of 64 × 49 pixels with RGB color channels. In the PCA feature space, the painting does not provide any selection of specific elements from the painting, as found in UFLK features. Furthermore, the reconstruction on the bottom left demonstrates that many details of the painting are lost in the PCA extraction of features.

Fig. 4. Confusion matrices of classification and clustering of painting styles

Each entry of the confusion matrices are scaled by the total paintings in a column, which in essence provides precision, i.e., the ratio of clustered paintings that actually belong to the true style.

Fig. 5. Scatterplot of painting styles clusters

The groupings of the painting styles obtained from spectral clustering are plotted as a scatterplot by projecting onto its first two multidimensional scaled components (MDS) (28) of the data. MDS is essentially a visaulization of the similarity of observed data in a dataset. The size of each node, which represents a single painting in the dataset, is determined by its degree (i.e., the number of connections each painting has to other paintings). As a result, the larger a node, the more connections that painting has to other paintings. Also, the distance between nodes (paintings) demonstrates the similarity between paintings. Similar or related paintings and styles are in close proximity, while disparate paintings and styles are distant.