基于形状和边判别的三维模型无监督配准*

doi:10.7523/j.ucas.2024.056

摘要/Abstract

摘要： 非刚性三维模型的配准在动画驱动、纹理传递、语义理解等不同应用领域具有广泛的应用,是计算机图形学的重要研究话题。然而基于骨架变形的非刚性物体,如人体、动物,往往按照关节的运动变化带动整体的形状变形。因而该类物体的非刚性变形和复杂的相互作用,给三维配准任务带来了困难和挑战。尽管近期的研究利用深度学习技术在重建具有骨架结构的三维形状时可以取得不错的效果,但这些方法往往严重依赖于大型带标注的数据集。在无监督的情况下,依然容易生成过度拉伸和部分交叉的模型网格。论文针对三维模型的配准问题,提出了一种基于GAN的无监督网络框架,利用三维生成器生成拟合扫描点云的模型网格,同时使用两个新型的判别器对变形网格进行正则化约束,在公开数据集上取得了较好的实验效果。

关键词: 三维重建, 三维配准, 对抗网络, 非刚性物体

Abstract: Registration of non-rigid 3D models has a wide range of applications in different application areas such as animation driving, texture transfer, semantic understanding, etc. and is an important research topic in computer graphics.However, non-rigid objects based on skeletal deformation, such as human bodies and animals, often induce overall shape deformations in accordance with the movements of the joints. The non-rigid deformations and complex interactions of such objects thus pose difficulties and challenges to the 3D alignment task. Although recent studies using deep learning techniques can achieve good results in reconstructing 3D shapes with skeleton structures, these methods tend to rely heavily on large datasets with annotations. In the unsupervised case, they are still prone to generating overstretched and partially crossed model meshes.The paper proposes an unsupervised GAN-based network framework for the registration of 3D models, using a 3D generator to generate model meshes that fits the scanned point clouds, and two novel discriminators to impose regularization constraints on the deformed meshes, which achieves good results on public datasets.

Key words: 3D human reconstruction, 3D registration, adversarial networks, non-rigid objects

中图分类号:

TP183

蒋宇涛, 刁俊淇, 肖俊, 王颖. 基于形状和边判别的三维模型无监督配准^*[J]. 中国科学院大学学报, DOI: 10.7523/j.ucas.2024.056.

JIANG Yutao, DIAO Junqi, XIAO Jun, WANG Ying. Unsupervised 3D registration of articulated shapes by shape and edge regularization[J]. Journal of University of Chinese Academy of Sciences, DOI: 10.7523/j.ucas.2024.056.

参考文献

[1] Romero J, Tzionas D, Black M J. Embodied hands: Modeling and capturing hands and bodies together[J]. ACM Trans. Graph,2017,36(6): 245:1-245:17. DOI:10.1145/3130800.3130883.
[2] Loper M, Mahmood N, Romero J, et al.SMPL: A skinned multi-person linear model[M]//Seminal Graphics Papers: Pushing the Boundaries, Volume 2. 2023: 851-866. DOI:10.1145/3596711.3596800.
[3] Yang H T, Zhu H, Wang Y R, et al.FaceScape: a large-scale high quality 3D face dataset and detailed riggable 3D face prediction[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle, WA, USA. IEEE, 2020: 598-607. DOI: 10.1109/CVPR42600.2020.00068.
[4] Sumner R W, Popović J.Deformation transfer for triangle meshes[J]. ACM Transactions on Graphics, 2004, 23(3): 399-405. DOI: 10.1145/1015706.1015736.
[5] Amberg B, Romdhani S, Vetter T.Optimal step nonrigid ICP algorithms for surface registration[C]//2007 IEEE Conference on Computer Vision and Pattern Recognition. Minneapolis, MN, USA. IEEE, 2007: 1-8. DOI: 10.1109/CVPR.2007.383165.
[6] Feng A W, Casas D, Shapiro A.Avatar reshaping and automatic rigging using a deformable model[C]//Proceedings of the 8th ACM SIGGRAPH Conference on Motion in Games. 2015: 57-64. DOI:10.1145/2822013.2822017.
[7] Zuffi S, Black M J.The stitched puppet: A graphical model of 3d human shape and pose[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015: 3537-3546. DOI:10.1109/cvpr.2015.7298976.
[8] Halimi O, Imanuel I, Litany O, et al. The whole is greater than the sum of its nonrigid parts[EB/OL]. 2020: 2001.09650.(2020-01-27)[2024-05-09].http://arxiv.org/abs/2001.09650v1.
[9] Deprelle T, Groueix T, Fisher M, et al.Learning elementary structures for 3d shape generation and matching[J]. Advances in Neural Information Processing Systems 32, 2019: 7433-7443.
[10] Zhou B, Franco J S, Bogo F, et al.Reconstructing human body mesh from point clouds by adversarial gp network[C]//Proceedings of the Asian Conference on Computer Vision. 2020. DOI:10.1007/978-3-030-69525-5_8.
[11] Groueix T, Fisher M, Kim V G, et al.3d-coded: 3d correspondences by deep deformation[C]//Proceedings of the european conference on computer vision (ECCV). 2018: 230-246. DOI:10.1007/978-3-030-01216-8_15.
[12] Li C L, Simon T, Saragih J, et al.Lbs autoencoder: Self-supervised fitting of articulated meshes to point clouds[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019: 11967-11976. DOI:10.1109/cvpr.2019.01224.
[13] Wang Y F, Aigerman N, Kim V G, et al.Neural cages for detail-preserving 3D deformations[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle, WA, USA. IEEE, 2020: 72-80. DOI: 10.1109/CVPR42600.2020.00015.
[14] Bhatnagar B L, Sminchisescu C, Theobalt C, et al.Combining implicit function learning and parametric models for 3d human reconstruction[C]//Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16. Springer International Publishing, 2020: 311-329. DOI:10.1007/978-3-030-58536-5_19.
[15] Ginzburg D, Raviv D.Cyclic functional mapping: Self-supervised correspondence between non-isometric deformable shapes[C]//Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part V 16. Springer International Publishing, 2020: 36-52. DOI:10.1007/978-3-030-58558-7_3.
[16] Eisenberger M, Lahner Z, Cremers D.Smooth shells: Multi-scale shape registration with functional maps[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020: 12265-12274. DOI:10.1109/CVPR42600.2020.01228.
[17] Varol G, Romero J, Martin X, et al.Learning from synthetic humans[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2017: 109-117. DOI:10.1109/CVPR.2017.492.
[18] Bogo F, Romero J, Loper M, et al.FAUST: Dataset and evaluation for 3D mesh registration[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2014: 3794-3801. DOI:10.1109/CVPR.2014.491.
[19] Zuffi S, Kanazawa A, Jacobs D W, et al.3D menagerie: Modeling the 3D shape and pose of animals[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2017: 6365-6373. DOI:10.1109/CVPR.2017.586.
[20] Varol G, Ceylan D, Russell B, et al.Bodynet: Volumetric inference of 3d human body shapes[C]//Proceedings of the European conference on computer vision (ECCV). 2018: 20-36. DOI:10.1007/978-3-030-01234-2\_2.
[21] Rakotosaona M J, Ovsjanikov M. Intrinsic point cloud interpolation via dual latent space navigation[C]//European Conference on Computer Vision. Cham: Springer, 2020: 655-672.10.1007/978-3-030-58536-5_39.
[22] Ranjan A, Bolkart T, Sanyal S, et al.Generating 3D faces using convolutional mesh autoencoders[C]//Proceedings of the European conference on computer vision (ECCV). 2018: 704-720. DOI:10.1007/978-3-030-01219-9\_43.
[23] Kanazawa A, Tulsiani S, Efros A A, et al.Learning category-specific mesh reconstruction from image collections[C]//Proceedings of the European Conference on Computer Vision (ECCV). 2018: 371-386. DOI:10.1007/978-3-030-01267-0\_23.
[24] Xu H Y, Bazavan E G, Zanfir A, et al.GHUM & GHUML: generative 3D human shape and articulated pose models[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle, WA, USA. IEEE, 2020: 6183-6192. DOI: 10.1109/CVPR42600.2020.00622.
[25] Chibane J, Alldieck T, Pons-Moll G.Implicit functions in feature space for 3d shape reconstruction and completion[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020: 6970-6981. DOI:10.1109/CVPR42600.2020.00700.
[26] Saito S, Huang Z, Natsume R, et al.Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2019: 2304-2314. DOI:10.1109/ICCV.2019.00239.
[27] Sun J, Ovsjanikov M, Guibas L.A concise and provably informative multi‐scale signature based on heat diffusion[C]//Computer graphics forum. Oxford, UK: Blackwell Publishing Ltd, 2009, 28(5): 1383-1392. DOI:10.1111/J.1467-8659.2009.01515.X.
[28] Marin R, Melzi S, Rodolà E, et al.FARM: Functional automatic registration method for 3D human bodies[J]. Computer Graphics Forum, 2020, 39(1): 160-173. DOI: 10.1111/cgf.13751.
[29] Ovsjanikov M, Corman E, Bronstein M, et al.Computing and processing correspondences with functional maps[M]//SIGGRAPH ASIA 2016 Courses. 2016: 1-60. DOI:10.1145/2988458.2988494.
[30] Sharma A, Ovsjanikov M.Weakly supervised deep functional maps for shape matching[J]. Advances in Neural Information Processing Systems 33:, 2020, 19264-19275.
[31] Hirshberg D A, Loper M, Rachlin E, et al.Evaluating the automated alignment of 3D human body scans[C]//2nd International Conference on 3D Body Scanning Technologies. Hometrica Consulting, 2011: 76-85. DOI:10.15221/11.076.
[32] Hirshberg D A, Loper M, Rachlin E, et al.Coregistration: Simultaneous alignment and modeling of articulated 3D shape[C]//Computer Vision-ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part VI 12. Springer Berlin Heidelberg, 2012: 242-255. DOI:10.1007/978-3-642-33783-3\_18.
[33] Goodfellow I, Pouget-Abadie J, Mirza M, et al.Generative adversarial nets[J]. Advances in neural information processing systems 27,2014:2672-2680.
[34] Abrevaya V F, Boukhayma A, Wuhrer S, et al.A decoupled 3d facial shape model by adversarial training[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2019: 9419-9428. DOI:10.1109/ICCV.2019.00951.
[35] Kanazawa A, Black M J, Jacobs D W, et al.End-to-end recovery of human shape and pose[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 7122-7131. DOI:10.1109/CVPR.2018.00744.
[36] Ma Q L, Yang J L, Ranjan A, et al.Learning to dress 3D people in generative clothing[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle, WA, USA. IEEE, 2020: 6468-6477. DOI: 10.1109/CVPR42600.2020.00650.
[37] Charles R Q, Hao S, Mo K C, et al.PointNet: Deep learning on point sets for 3D classification and segmentation[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI, USA. IEEE, 2017: 77-85. DOI: 10.1109/CVPR.2017.16.
[38] Isola P, Zhu J Y, Zhou T H, et al.Image-to-image translation with conditional adversarial networks[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI, USA. IEEE, 2017: 5967-5976. DOI: 10.1109/CVPR.2017.632.
[39] Mao X D, Li Q, Xie H R, et al.Least squares generative adversarial networks[C]//2017 IEEE International Conference on Computer Vision (ICCV). Venice, Italy. IEEE, 2017: 2813-2821. DOI: 10.1109/ICCV.2017.304.

基于形状和边判别的三维模型无监督配准^*

Unsupervised 3D registration of articulated shapes by shape and edge regularization

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 7

编辑推荐

Metrics

本文评价

访问统计

联系我们

[1]	陈汉钦, 秦进, 赵彤, 阎瑶. 面向识别的人脸三维信息估计[J]. 中国科学院大学学报, 2023, 40(2): 268-279.
[2]	王正晟, 李亚林, 张洪群. 基于GAN序列的卫星数据接收系统故障定位[J]. 中国科学院大学学报, 2023, 40(1): 109-118.
[3]	胡兵兵, 唐华, 吴幼龙. 基于互信息约束的生成对抗网络分类模型[J]. 中国科学院大学学报, 2022, 39(4): 551-560.
[4]	赵心驰, 姜策, 何为. 基于注意力生成对抗网络的图像强光去除[J]. 中国科学院大学学报, 2022, 39(4): 524-531.
[5]	陈国炜, 刘磊, 郭嘉逸, 潘宗序, 胡文龙. 基于生成对抗网络的半监督遥感图像飞机检测[J]. 中国科学院大学学报, 2020, 37(4): 539-546.
[6]	毛明旭, 叶佳敏, 王海刚, 张娇龙. 三维电容层析成像多电极激励模式的稳定性分析[J]. 中国科学院大学学报, 2017, 34(2): 186-190.
[7]	胡继伟, 洪峻. 基于三维重建的分布式卫星SAR干涉测高模型及误差分析[J]. 中国科学院大学学报, 2011, 28(4): 489-497.