Fast 3D Face Reconstruction from a Single Image Using Different Deep Learning Approaches for Facial Palsy Patients

Abstract

The 3D reconstruction of an accurate face model is essential for delivering reliable feedback for clinical decision support. Medical imaging and specific depth sensors are accurate but not suitable for an easy-to-use and portable tool. The recent development of deep learning (DL) models opens new challenges for 3D shape reconstruction from a single image. However, the 3D face shape reconstruction of facial palsy patients is still a challenge, and this has not been investigated. The contribution of the present study is to apply these state-of-the-art methods to reconstruct the 3D face shape models of facial palsy patients in natural and mimic postures from one single image. Three different methods (3D Basel Morphable model and two 3D Deep Pre-trained models) were applied to the dataset of two healthy subjects and two facial palsy patients. The reconstructed outcomes were compared to the 3D shapes reconstructed using Kinect-driven and MRI-based information. As a result, the best mean error of the reconstructed face according to the Kinect-driven reconstructed shape is 1.5±1.1 mm. The best error range is 1.9±1.4 mm when compared to the MRI-based shapes. Before using the procedure to reconstruct the 3D faces of patients with facial palsy or other facial disorders, several ideas for increasing the accuracy of the reconstruction can be discussed based on the results. This present study opens new avenues for the fast reconstruction of the 3D face shapes of facial palsy patients from a single image. As perspectives, the best DL method will be implemented into our computer-aided decision support system for facial disorders.

Keywords: 3D morphable model; 3D pre-trained model; Kinect-driven reconstruction; MRI; deep learning; fast 3D face reconstruction; single image.

Figure 1

The general framework for reconstructing a 3D face of an individual.

Figure 2

Pipeline to estimate the shape parameters of the 3DMM.

Figure 3

Patient-specific 3D face model was reconstructed using mean face, eigenfaces, and coefficients.

Figure 4

The network architecture for learning the parameters of the face model. The output of models, including coefficients that represent identity ($\alpha$), expression ($\beta$), texture ($\delta$), pose ($p$), lighting ($\gamma$), and identity confidence ($c$).

Figure 5

Reconstructed 3D face shape from the MRI images and segmentation. The 3D face shape was finally registered to the coordinate system of the image-based reconstructed face model before calculating the Hausdorff distances.

Figure 6

3D face reconstruction from an input image.

Figure 7

Comparison of 3D face reconstruction (grey) and 3D face reconstruction from MRI (yellow) using the first method (fitting a 3DMM).

Figure 8

Comparison of 3D face reconstruction (grey) and 3D face reconstruction from MRI (yellow) using the second method (DECA).

Figure 9

Comparison of 3D face reconstruction (grey) and 3D face reconstruction from MRI (yellow) using the third method (deep 3D face reconstruction).

Figure 10

The error of the best and the worst prediction cases of the third method compared with MRI ground truth data.

Figure 11

The error of the reconstructed face in mimic position of a healthy subject.

Figure 12

The error of the reconstructed face in mimic position of a facial palsy subject.

Figure 13

The 3D face reconstruction of facial palsy patients using method 3 (deep 3D face reconstruction) using collected images in open access dataset.

Figure 14

The 3D face reconstruction of the last six facial palsy patients using method 3 (deep 3D face reconstruction) using images from CHU Amiens.