Reconstructing Superquadrics from Intensity and Color Images

Abstract

The task of reconstructing 3D scenes based on visual data represents a longstanding problem in computer vision. Common reconstruction approaches rely on the use of multiple volumetric primitives to describe complex objects. Superquadrics (a class of volumetric primitives) have shown great promise due to their ability to describe various shapes with only a few parameters. Recent research has shown that deep learning methods can be used to accurately reconstruct random superquadrics from both 3D point cloud data and simple depth images. In this paper, we extended these reconstruction methods to intensity and color images. Specifically, we used a dedicated convolutional neural network (CNN) model to reconstruct a single superquadric from the given input image. We analyzed the results in a qualitative and quantitative manner, by visualizing reconstructed superquadrics as well as observing error and accuracy distributions of predictions. We showed that a CNN model designed around a simple ResNet backbone can be used to accurately reconstruct superquadrics from images containing one object, but only if one of the spatial parameters is fixed or if it can be determined from other image characteristics, e.g., shadows. Furthermore, we experimented with images of increasing complexity, for example, by adding textures, and observed that the results degraded only slightly. In addition, we show that our model outperforms the current state-of-the-art method on the studied task. Our final result is a highly accurate superquadric reconstruction model, which can also reconstruct superquadrics from real images of simple objects, without additional training.

Keywords: color images; convolutional neural networks; deep learning; reconstruction; superquadrics.

Figure 1

Visualized overview of our reconstruction method. (1) Generate a large dataset of synthetic images that mimic the real environment. Each image contains a randomly positioned superquadric of a random shape. (2) Train the CNN predictor on the synthetic dataset to predict superquadric parameters. (3) Use the final CNN predictor (Textured on Wood (Fixed z) with Sh. & S.L.) on real images to reconstruct simple objects as superquadrics.

Figure 2

Possible superquadric shapes obtained by changing the shape parameters ${ϵ}_1$ and ${ϵ}_2$.

Figure 3

Visualization of the learning objective. To train the CNN predictor for superquadric recovery, we used a geometry-aware loss function based on the 3D occupancy grids constructed from predicted and ground truth parameters.

Figure 4

Visualization of the modified ResNet-18 model used for predicting superquadric parameters. The network input is a color image and the outputs are the different superquadric parameter groups. The notation $\{X\times Y,Z\}$ represents a convolutional layer with Z filters of size $X\times Y$.

Figure 5

Various images of superquadrics (SQs) generated by the renderer present in Figure 1. (a) Depth image; (b) intensity image; (c) color image; (d) color image with shadows; (e) textured SQ and background; (f) textured SQ on wood with shadows.

Figure 6

IoU score distributions of CNN predictors trained on different datasets. The distributions are based on predictions of the corresponding test images. Clear performance decay can be observed with increasing complexities of images.

Figure 7

Visualization of mean absolute error (MAE) distributions across the predicted parameters. Each trained model is represented by its own color. Size ($\overline{a}$) and shape ($\overline{ϵ}$) errors are averaged over all elements of the parameter group, due to the arbitrary ordering discussed before. Errors of the z parameter are not reported, because the distributions serve no purpose for most models, due to the Fixed z parameter.

Figure 8

Distributions of IoU values obtained when training the model on regular depth images and on intensity images with the Fixed z position requirement. (a) Depth; (b) Intensity (Fixed z).

Figure 9

Examples of superquadric reconstructions from across the whole IoU distribution range. The first column depicts input images, while the second and third show the overlap between ground truth (red) and predicted (black) superquadrics in wireframe form, from two different viewpoints.

Figure 9

Examples of superquadric reconstructions from across the whole IoU distribution range. The first column depicts input images, while the second and third show the overlap between ground truth (red) and predicted (black) superquadrics in wireframe form, from two different viewpoints.

Figure 10

Visualization of mean accuracy values obtained on images of superquadrics with different ground truth shape parameter values.

Figure 11

Examples of scene alterations used to counteract the Fixed z position requirement. Images depict identical superquadrics with minor scene alterations. The first image is the baseline. In the second, we enabled shadows in the scene. For the third image, we added a spotlight to cast larger shadows. All images contain blue superquadrics on grey backgrounds to allow for better visibility of shadows. (a) BoG (Free z); (b) add shadows; (c) add spotlight.

Figure 12

Distributions of IoU values obtained when training the model on variations of the Blue on Gray (BoG) images. (a) BoG (Fixed z); (b) BoG (Free z) with shadows; (c) BoG (Free z) with shadows and spotlight.

Figure 13

Qualitative results on real images obtained with our final CNN predictor trained on artificial data (Textured on Wood (Fixed z) with Sh. & S.L.). The first column depicts the input images, while the second column shows the predicted superquadric in a wireframe form. The last column combines the two images for easier evaluation.