A Novel Approach to Droplet's 3D Shape Recovery Based on Mask R-CNN and Improved Lambert⁻Phong Model

Abstract

Aiming at the demand for extracting the three-dimensional shapes of droplets in microelectronic packaging, life science, and some related fields, as well as the problems of complex calculation and slow running speed of conventional shape from shading (SFS) illumination reflection models, this paper proposes a Lambert⁻Phong hybrid model algorithm to recover the 3D shapes of micro-droplets based on the mask regions with convolutional neural network features (R-CNN) method to extract the highlight region of the droplet surface. This method fully integrates the advantages of the Lambertian model's fast running speed and the Phong model's high accuracy for reconstruction of the highlight region. First, the Mask R-CNN network is used to realize the segmentation of the highlight region of the droplet and obtain its coordinate information. Then, different reflection models are constructed for the different reflection regions of the droplet, and the Taylor expansion and Newton iteration method are used for the reflection model to get the final height of all positions. Finally, a three-dimensional reconstruction experimental platform is built to analyze the accuracy and speed of the algorithm on the synthesized hemisphere image and the actual droplet image. The experimental results show that the proposed algorithm based on mask R-CNN had better precision and shorter running time. Hence, this paper provides a new approach for real-time measurement of 3D droplet shape in the dispensing state.

Keywords: Lambert–Phong model; Mask R-CNN; segment highlight region; shape from shading.

Figure 1

Algorithm flow chart.

Figure 2

Mask R-CNN flow chart. CNN: convolutional neural network; ROI: region of interest; RPN: region proposal network.

Figure 3

(a) Binary droplet image of feature to be extracted. (b) Experimental image after detection by Mask R-CNN (where the red area is the highlight area and the feature area is called “star”).

Figure 4

Schematic diagram of experimental equipment.

Figure 5

Composite hemisphere.

Figure 6

(a) Original grayscale composite image (A–B is the cut line for the cross section); (b) 3D shape of the composite image and the side cross section contrast diagram using the Lambertian model algorithm (dark blue is the cross section of reconstruction, light blue is the cross section of the real shape); (c) 3D shape of the composite image and the side cross section contrast diagram using the Phong hybrid model algorithm; (d) highlight detection’s results of the composite image; (e) 3D shape of the composite image at the highlight region using the Phong hybrid model algorithm; (f) 3D shape of the composite image at the non-highlight region using the Lambertian model algorithm; (g) 3D shape of the composite image and the side cross section contrast diagram using the Lambert–Phong model algorithm.

Figure 7

(a) Real profile of the droplet; (b-1) 3D shape reconstruction of the real image using the Lambertian model algorithm; (b-2) 3D shape of the real image and the side cross section contrast diagram using the Lambertian model algorithm; (c-1) 3D shape reconstruction of the real image using the Phong hybrid model algorithm; (c-2) 3D shape of the real image and the side cross section contrast diagram using the Phong hybrid model algorithm; (d-1) 3D shape reconstruction of the real image using the Lambert–Phong model algorithm; (d-2) 3D shape of the real image and the side cross section contrast diagram using the Lambert–Phong model algorithm.

Figure 7

(a) Real profile of the droplet; (b-1) 3D shape reconstruction of the real image using the Lambertian model algorithm; (b-2) 3D shape of the real image and the side cross section contrast diagram using the Lambertian model algorithm; (c-1) 3D shape reconstruction of the real image using the Phong hybrid model algorithm; (c-2) 3D shape of the real image and the side cross section contrast diagram using the Phong hybrid model algorithm; (d-1) 3D shape reconstruction of the real image using the Lambert–Phong model algorithm; (d-2) 3D shape of the real image and the side cross section contrast diagram using the Lambert–Phong model algorithm.