Recognition-oriented facial 3D information estimation

doi:10.7523/j.ucas.2021.0058

Abstract

Abstract: 3D face recognition has the advantages of recognition accuracy and anti-counterfeiting strength over the popular 2D face recognition, and represents the development direction of face recognition. Due to the high cost of 3D facial acquisition, 3D facial recognition can not establish and optimize face recognition algorithms directly rely on massive face data as 2D facial recognition methods. How to obtain augmented training data for 3D faces accurately and efficiently is the most pressing problem in driving the development of 3D face recognition applications. A large amount of related research focuses on how to get better 3D face reconstruction visualization,but does not give much consideration to the subsequent recognition task,so that the recognition accuracy of 3D face recognition algorithms trained with these reconstructed images is much lower than expected. To address this problem, a recognition-oriented method for estimating facial 3D information is proposed. Different from the general method, this method directly builds an interactive bridge between information estimation and subsequent recognition:during the training process of 3D face information estimation, it directly bases on the corresponding recognition network to supervise and improve the estimation effect of 3D face information. For this purpose, we first construct a 3D face information representation, the depth-surface normal vector map (DN map), and then train a facial CycleGAN model with a real 3D dataset to learn a mapping from 2D face to DN map with preserved identity information and represent it in the form of U-Net network. Experiments are conducted on five datasets to compare with other methods, and the improvement is particularly significant in the ND-2006 dataset, with an improvement of 31.8%. In addition, experiments on performance improvement under data augmentation are conducted, here the performance improvement of the augmentation method based on the facial CycleGAN is more obvious under the same conditions of data augmentation, with a maximum improvement of 14.9% on the CASIA 3D dataset.

Key words: facial 3D information estimation, cycle generative adversarial networks, facial DN map, identity preserving loss, parameter pre-train

CLC Number:

TP391

CHEN Hanqin, QIN Jin, ZHAO Tong, YAN Yao. Recognition-oriented facial 3D information estimation[J]. Journal of University of Chinese Academy of Sciences, 2023, 40(2): 268-279.

References

[1] Bowyer K W, Chang K, Flynn P. A survey of approaches and challenges in 3D and multi-modal 3D +2D face recognition[J]. Computer Vision and Image Understanding, 2006, 101(1):1-15.
[2] Deng J K, Guo J, Xue N N, et al. ArcFace:additive angular margin loss for deep face recognition[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). June 15-20, 2019, Long Beach, CA, USA. IEEE, 2019:4685-4694.
[3] Parkhi O M, Vedaldi A, Zisserman A. Deep face recognition[C]//Procedings of the British Machine Vision Conference 2015. Swansea. British Machine Vision Association, 2015:1-12.
[4] Schroff F, Kalenichenko D, Philbin J. FaceNet:a unified embedding for face recognition and clustering[C]//2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). June 7-12, 2015, Boston, MA, USA. IEEE, 2015:815-823.
[5] Taigman Y, Yang M, Ranzato M, et al. DeepFace:closing the gap to human-level performance in face verification[C]//2014 IEEE Conference on Computer Vision and Pattern Recognition. June 23-28, 2014, Columbus, OH, USA. IEEE, 2014:1701-1708.
[6] Faltemier T C, Bowyer K W, Flynn P J. Using a multi-instance enrollment representation to improve 3D face recognition[C]//2007 First IEEE International Conference on Biometrics:Theory, Applications, and Systems. September 27-29, 2007, Crystal City, VA, USA. IEEE, 2007:1-6.
[7] Dou P F, Shah S K, Kakadiaris I A. End-to-end 3D face reconstruction with deep neural networks[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). July 21-26, 2017, Honolulu, HI. IEEE, 2017:1503-1512.
[8] Feng Y, Wu F, Shao X H, et al. Joint 3D face reconstruction and dense alignment with position map regression network[C]//Computer Vision-ECCV 2018, 2018:557-574. DOI:10.1007/978-3-030-01264-9_33.
[9] Richardson E, Sela M T, Kimmel R. 3D face reconstruction by learning from synthetic data[C]//2016 Fourth International Conference on 3D Vision (3DV). October 25-28, 2016, Stanford, CA, USA. IEEE, 2016:460-469.
[10] Zulqarnain Gilani S, Mian A. Learning from millions of 3D scans for large-scale 3D face recognition[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. June 18-23, 2018, Salt Lake City, UT, USA. IEEE, 2018:1896-1905.
[11] Mu G D, Huang D, Hu G S, et al. Led3D:a lightweight and efficient deep approach to recognizing low-quality 3D faces[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). June 15-20, 2019, Long Beach, CA, USA. IEEE, 2019:5766-5775.
[12] Gilani S Z, Mian A, Shafait F, et al. Dense 3D face correspondence[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(7):1584-1598.
[13] Gilani S Z, Mian A. Towards large-scale 3D face recognition[C]//2016 International Conference on Digital Image Computing:Techniques and Applications (DICTA). November 30-December 2, 2016, Gold Coast, QLD, Australia. IEEE, 2016:1-8.
[14] Gilani S Z, Mian A, Eastwood P. Deep, dense and accurate 3D face correspondence for generating population specific deformable models[J]. Pattern Recognition, 2017, 69:238-250.
[15] Isola P, Zhu J Y, Zhou T H, et al. Image-to-image translation with conditional adversarial networks[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). July 21-26, 2017, Honolulu, HI, USA. IEEE, 2017:5967-5976.
[16] Wang W X, Fu Y W, Qian X L, et al. FM2u-net:face morphological multi-branch network for makeup-invariant face verification[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). June 13-19, 2020, Seattle, WA, USA. IEEE, 2020:5729-5739.
[17] Johnson J, Alahi A, Li F F. Perceptual losses for real-time style transfer and super-resolution[C]//Computer Vision-ECCV 2016, 2016:694-711. DOI:10.1007/978-3-319-46475-6_43.
[18] Zhu J Y, Park T, Isola P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks[C]//2017 IEEE International Conference on Computer Vision (ICCV). October 22-29, 2017, Venice, Italy. IEEE, 2017:2242-2251.
[19] Zhang K P, Zhang Z P, Li Z F, et al. Joint face detection and alignment using multitask cascaded convolutional networks[J]. IEEE Signal Processing Letters, 2016, 23(10):1499-1503.
[20] Ronneberger O, Fischer P, Brox T. U-net:convolutional networks for biomedical image segmentation[M]//Lecture Notes in Computer Science. Cham:Springer International Publishing, 2015:234-241.
[21] He K M, Zhang X Y, Ren S Q, et al. Deep residual learning for image recognition[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). June 27-30, 2016, Las Vegas, NV, USA. IEEE, 2016:770-778.
[22] Yi D, Lei Z, Liao S C, et al. Learning face representation from scratch[EB/OL]. arXiv:1411.7923. (2014-11-28)[2021-01-20]. https://arxiv.org/pdf/1411.7923.pdf.
[23] Zhu X Y, Lei Z, Liu X M, et al. Face alignment across large poses:a 3D solution[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). June 27-30, 2016, Las Vegas, NV, USA. IEEE, 2016:146-155.
[24] Tran A T, Hassner T, Masi I, et al. Regressing robust and discriminative 3D morphable models with a very deep neural network[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). July 21-26, 2017, Honolulu, HI, USA. IEEE, 2017:1493-1502.
[25] Guo J Z, Zhu X Y, Yang Y, et al. Towards fast, accurate and stable 3D dense face alignment[M]//Computer Vision-ECCV 2020. Cham:Springer International Publishing, 2020:152-168.
[26] Vijayan V, Bowyer K W, Flynn P J, et al. Twins 3D face recognition challenge[C]//2011 International Joint Conference on Biometrics (IJCB). October 11-13, 2011, Washington, DC, USA. IEEE, 2011:1-7.
[27] Zhong C, Sun Z N, Tan T N. Learning efficient codes for 3D face recognition[C]//200815th IEEE International Conference on Image Processing. October 12-15, 2008, San Diego, CA, USA. IEEE, 2008:1928-1931.
[28] Savran A, Alyüz N, Dibeklioğlu H, et al. Bosphorus database for 3D face analysis[M]//Lecture Notes in Computer Science. Berlin, Heidelberg:Springer Berlin Heidelberg, 2008:47-56.
[29] Gupta S, Markey M K, Bovik A C. Anthropometric 3D face recognition[J]. International Journal of Computer Vision, 2010, 90(3):331-349.