Deformation Estimation of Textureless Objects from a Single Image

Abstract

Deformations introduced during the production of plastic components degrade the accuracy of their 3D geometric information, a critical aspect of object inspection processes. This phenomenon is prevalent among primary plastic products from manufacturers. This work proposes a solution for the deformation estimation of textureless plastic objects using only a single RGB image. This solution encompasses a unique image dataset of five deformed parts, a novel method for generating mesh labels, sequential deformation, and a training model based on graph convolution. The proposed sequential deformation method outperforms the prevalent chamfer distance algorithm in generating precise mesh labels. The training model projects object vertices into features extracted from the input image, and then, predicts vertex location offsets based on the projected features. The predicted meshes using these offsets achieve a sub-millimeter accuracy on synthetic images and approximately 2.0 mm on real images.

Keywords: deformation estimation; graph convolution; image dataset; label generation; single image; textureless deformed object.

Figure 1

Images of the five textureless plastic objects studied in this paper: (a) skateboard, (b) bracket, (c) round flat receptacle lid (rfrl), (d) oval flange, and (e) vent cover.

Figure 2

Images of the deformed versions of the skateboard; the wireframe illustrates the undeformed model: (a) $F=22.5 \mathrm{N}, MD=10.08 \mathrm{m}\mathrm{m}$; (b) $F=44.5 \mathrm{N}, MD=19.93 \mathrm{m}\mathrm{m}$; (c) $F=-22.5 \mathrm{N}$, $MD=-10.08 \mathrm{m}\mathrm{m}$; (d) $F=-44.5 \mathrm{N}, MD=-19.93 \mathrm{m}\mathrm{m}$.

Figure 3

Images of the deformed versions of the bracket; the wireframe illustrates the undeformed model: (a) $F=24 \mathrm{N}$, $MD=5.14 \mathrm{m}\mathrm{m}$; (b) $F=66 \mathrm{N}$, $MD=14.15 \mathrm{m}\mathrm{m}$; (c) $F=-24 \mathrm{N}$, $MD=-5.14 \mathrm{m}\mathrm{m}$; (d) $F=-66 \mathrm{N}$, $MD=-14.15 \mathrm{m}\mathrm{m}$.

Figure 4

Images of the deformed versions of the round flat receptacle lid; the wireframe illustrates the undeformed model: (a) $F=1050 \mathrm{N},MD=6.09 \mathrm{m}\mathrm{m}$; (b) $F=2250 \mathrm{N},MD=13.06 \mathrm{m}\mathrm{m}$; (c) $F=3450 \mathrm{N}$, $MD=20.03 \mathrm{m}\mathrm{m}$; (d) $F=4300 \mathrm{N},MD=4300 \mathrm{m}\mathrm{m}$.

Figure 5

Images of the deformed versions of the oval flange; the wireframe illustrates the undeformed model: (a) $F=1300 \mathrm{N}, MD=8.93 \mathrm{m}\mathrm{m}$; (b) $F=2650 \mathrm{N}, MD=18.20 \mathrm{m}\mathrm{m}$; (c) $F=-1300 \mathrm{N}$, $MD=-8.93 \mathrm{m}\mathrm{m}$; (d) $F=-2650 \mathrm{N},MD=-18.20 \mathrm{m}\mathrm{m}$.

Figure 6

Images of the deformed versions of the vent cover; the wireframe illustrates the undeformed model: (a) $F=150 \mathrm{N},MD=3.88 \mathrm{m}\mathrm{m}$; (b) $F=540 \mathrm{N}, MD=13.97 \mathrm{m}\mathrm{m}$; (c) $F=-150 \mathrm{N}$, $MD=-3.88 \mathrm{m}\mathrm{m}$; (d) $F=-540 \mathrm{N}, MD=-13.97 \mathrm{m}\mathrm{m}$.

Figure 7

This schematic depicts the camera position relative to the deformed vent cover, located at the origin $(0,0,0)$ of Blender’s global coordinate system. The red, green, and blue axes represent the X, Y, and Z axes, respectively.

Figure 8

Images of the deformed vent cover with $F=540.0 \mathrm{N}, MD=13.97 \mathrm{m}\mathrm{m}$, and $R_{Z_O}=0°$. (a) $T_{X_L}=0 \mathrm{m}\mathrm{m}$, (b) $T_{X_L}=25 \mathrm{m}\mathrm{m}$, (c) $T_{X_L}=35 \mathrm{m}\mathrm{m}$, (d) $T_{X_L}=45 \mathrm{m}\mathrm{m}$, (e) $R_{X_C}=0°$, (f) $R_{X_C}=10°$, (g) $R_{X_C}=20°$, and (h) $R_{X_C}=30°$. If not mentioned, $R_{X_C}=60°$ and $T_{X_L}=0 \mathrm{m}\mathrm{m}$.

Figure 9

Images of the 3D-printed deformed models captured by a smartphone camera. (a) Deformed model of the bracket with $F=66.0 \mathrm{N}, MD=14.15 \mathrm{m}\mathrm{m}$. (b) Deformed bracket component with $F=-66.0 \mathrm{N}, MD=14.15 \mathrm{m}\mathrm{m}$. (c) Deformed vent cover component with $F=270 \mathrm{N}, MD=6.98 \mathrm{m}\mathrm{m}$. (d) Deformed model of the vent cover object with $F=-500.0 \mathrm{N}, MD=-12.93 \mathrm{m}\mathrm{m}$.

Figure 10

Setup devised to capture pictures from the printed deformed models.

Figure 11

Real-world images of a 3D-printed, deformed vent cover ($F=-500.0 \mathrm{N}, MD=-12.93 \mathrm{m}\mathrm{m}$) used for training the machine learning model. (a) $R_{Z_O}=-0.70°$ and $R_{X_C}=85.36°$. (b) $R_{Z_O}=36.46°$ and $R_{X_C}=82.09°$. (c) $R_{Z_O}=-34.40°$, and $R_{X_C}=72.71°$. (d) $R_{Z_O}=-3.49°$ and $R_{X_C}=76.82°$.

Figure 12

Mesh comparison: (a) Deformed mesh model, Ansys output with 6268 vertices and 12,544 faces. (b) Initial undeformed mesh model (training model input) with 1145 vertices and 2298 faces.

Figure 13

The chamfer distance algorithm applied on two 2D distributions. False correspondences (red dashed line) for middle blue point, and true correspondence (green dashed line).

Figure 14

Sequential deformation (lower path) vs. direct application of the chamfer distance (upper path) on a vent cover object with significant geometric variation (e). (a) Initial undeformed model. (b) Least deformed model with $MD=0.25 \mathrm{m}\mathrm{m},F=10 \mathrm{N}$. (c) Average deformed model with $MD=7.76 \mathrm{m}\mathrm{m},F=300 \mathrm{N}$. (d) Deformed model with $MD=14.75 \mathrm{m}\mathrm{m},F=570 \mathrm{N}$. (e) Highest deformed model with $MD=15.00 \mathrm{m}\mathrm{m},F=580 \mathrm{N}$. Labels were generated using (f) direct chamfer distance and (g) sequential deformation algorithm.

Figure 15

Sequential deformation (SD) vs. chamfer distance (CD) on skateboard object with $MD=-19.93 \mathrm{m}\mathrm{m}$ (first row) and $MD=19.93 \mathrm{m}\mathrm{m}$ (second row). (a,e) Initial undeformed model. (b,f) Deformed models (Ansys output). (c,g) CD algorithm outputs. (d,h) SD algorithm outputs.

Figure 16

Sequential deformation (SD) vs. chamfer distance (CD) on bracket object with $MD=-14.15 \mathrm{m}\mathrm{m}$ (first row) and $MD=14.15 \mathrm{m}\mathrm{m}$ (second row). (a,e) Initial undeformed model. (b,f) Deformed models (Ansys output). (c,g) CD algorithm outputs. (d,h) SD algorithm outputs.

Figure 17

Sequential deformation (SD) vs. chamfer distance (CD) on RFRL object with $MD=24.97 \mathrm{m}\mathrm{m}$. (a) Initial undeformed model. (b) Deformed model (Ansys output). (c) CD algorithm output. (d) SD algorithm output.

Figure 18

Sequential deformation (SD) vs. chamfer distance (CD) on oval flange object with $MD=-18.20 \mathrm{m}\mathrm{m}$ (first row) and $MD=18.20 \mathrm{m}\mathrm{m}$ (second row). (a,e) Initial undeformed model. (b,f) Deformed models (Ansys output). (c,g) CD algorithm outputs. (d,h) SD algorithm outputs.

Figure 19

Sequential deformation (SD) vs. chamfer distance (CD) on vent cover object with $MD=15.00 \mathrm{m}\mathrm{m}$. (a) Initial undeformed model. (b) Deformed model (Ansys output). (c) CD algorithm output. (d) SD algorithm output.

Figure 20

Training model pipeline: Blue cubes represent 2D convolutional layers, and yellow cubes denote max-pooling layers. The “cam” block represents the camera model’s intrinsic and extrinsic parameters. Green rectangles symbolize graph convolutional layers, while magenta rectangles represent dense layers.

Figure 21

Visualization of actual Euclidean distance error between the predicted mesh and the ground truth for each vertex of the predicted skateboard $[F=33.5 \mathrm{N},MD=15.0 \mathrm{m}\mathrm{m}]$. (a) Input mesh of the training network (undeformed). (b) Input image of the deformed skateboard fed to the training model. (c–e) Predicted mesh of the deformed skateboard from different viewpoints. (f) Color bar representing the magnitude of the Euclidean distance error in millimeters (mm).

Figure 22

Visualization of actual Euclidean distance error between the predicted mesh and the ground truth for each vertex of the predicted skateboard $[F=-33.5 \mathrm{N},MD=-15.0 \mathrm{m}\mathrm{m}]$. (a) Input mesh of the training network (undeformed). (b) Input image of the deformed skateboard fed to the training model. (c–e) Predicted mesh of the deformed skateboard from different viewpoints. (f) Color bar representing the magnitude of the Euclidean distance error in millimeters (mm).

Figure 23

Visualization of actual Euclidean distance error between the predicted mesh and the ground truth for each vertex of the predicted bracket $[F=28 \mathrm{N},MD=6.00 \mathrm{m}\mathrm{m}]$. (a) Input mesh of the training network (undeformed). (b) Input image of the deformed bracket fed to the training model. (c–e) Predicted mesh of the deformed bracket from different viewpoints. (f) Color bar representing the magnitude of the Euclidean distance error in millimeters (mm).

Figure 24

Visualization of actual Euclidean distance error between the predicted mesh and the ground truth for each vertex of the predicted bracket $[F=-46 \mathrm{N},MD=-9.86 \mathrm{m}\mathrm{m}]$. (a) Input mesh of the training network (undeformed). (b) Input image of the deformed bracket fed to the training model. (c–e) Predicted mesh of the deformed bracket from different viewpoints. (f) Color bar representing the magnitude of the Euclidean distance error in millimeters (mm).

Figure 25

Visualization of actual Euclidean distance error between the predicted mesh and the ground truth for each vertex of the predicted RFRL $[F=2750 \mathrm{N},MD=15.96 \mathrm{m}\mathrm{m}]$. (a) Input mesh of the training network (undeformed). (b) Input image of the deformed RFRL fed to the training model. (c–e) Predicted mesh of the deformed RFRL from different viewpoints. (f) Color bar representing the magnitude of the Euclidean distance error in millimeters (mm).

Figure 26

Visualization of actual Euclidean distance error between the predicted mesh and the ground truth for each vertex of the predicted RFRL $[F=3950 \mathrm{N},MD=22.93 \mathrm{m}\mathrm{m}]$. (a) Input mesh of the training network (undeformed). (b) Input image of the deformed RFRL fed to the training model. (c–e) Predicted mesh of the deformed RFRL from different viewpoints. (f) Color bar representing the magnitude of the Euclidean distance error in millimeters (mm).

Figure 27

Visualization of actual Euclidean distance error between the predicted mesh and the ground truth for each vertex of the predicted oval flange $[F=1750 \mathrm{N},MD=12.02 \mathrm{m}\mathrm{m}]$. (a) Input mesh of the training network (undeformed). (b) Input image of the deformed oval flange fed to the training model. (c–e) Predicted mesh of the deformed oval flange from different viewpoints. (f) Color bar representing the magnitude of the Euclidean distance error in millimeters (mm).

Figure 28

Visualization of actual Euclidean distance error between the predicted mesh and the ground truth for each vertex of the predicted oval flange $[F=-2200 \mathrm{N},MD=-15.11 \mathrm{m}\mathrm{m}]$. (a) Input mesh of the training network (undeformed). (b) Input image of the deformed oval flange fed to the training model. (c–e) Predicted mesh of the deformed oval flange from different viewpoints. (f) Color bar representing the magnitude of the Euclidean distance error in millimeters (mm).

Figure 29

Visualization of actual Euclidean distance error between the predicted mesh and the ground truth for each vertex of the predicted vent cover $[F=310 \mathrm{N},MD=8.02 \mathrm{m}\mathrm{m}]$. (a) Input mesh of the training network (undeformed). (b) Input image of the deformed vent cover fed to the training model. (c–e) Predicted mesh of the deformed vent cover from different viewpoints. (f) Color bar representing the magnitude of the Euclidean distance error in millimeters (mm).

Figure 30

Visualization of actual Euclidean distance error between the predicted mesh and the ground truth for each vertex of the predicted vent cover $[F=390 \mathrm{N},MD=-10.09 \mathrm{m}\mathrm{m}]$. (a) Input mesh of the training network (undeformed). (b) Input image of the deformed vent cover fed to the training model. (c–e) Predicted mesh of the deformed vent cover from different viewpoints. (f) Color bar representing the magnitude of the Euclidean distance error in millimeters (mm).

Figure 31

Visualization of actual Euclidean distance error between the predicted mesh and the ground truth for each vertex of the predicted bracket $[F=-28 \mathrm{N},MD=-6.00 \mathrm{m}\mathrm{m}]$. (a) Input mesh of the training network (undeformed). (b) Real image of the actual deformed bracket fed to the training model. (c–e) Predicted mesh of the deformed bracket from different viewpoints. (f) Color bar representing the magnitude of the Euclidean distance error in millimeters (mm).

Figure 32

Visualization of actual Euclidean distance error between the predicted mesh and the ground truth for each vertex of the predicted vent cover $[F=500 \mathrm{N},MD=-10.09 \mathrm{m}\mathrm{m}]$. (a) Input mesh of the training network (undeformed). (b) Real image of the 3D-printed deformed vent cover fed to the training model. (c–e) Predicted mesh of the deformed vent cover from different viewpoints. (f) Color bar representing the magnitude of the Euclidean distance error in millimeters (mm).