Unsupervised 3D registration of articulated shapes by shape and edge regularization

doi:10.7523/j.ucas.2024.056

Abstract

Abstract:

Registration of non-rigid 3D models has a wide range of applications in different application areas such as animation driving， texture transfer， semantic understanding， etc. and is an important research topic in computer graphics.However， non-rigid objects based on skeletal deformation， such as human bodies and animals， often induce overall shape deformations in accordance with the movements of the joints. The non-rigid deformations and complex interactions of such objects thus pose difficulties and challenges to the 3D alignment task. Although recent studies using deep learning techniques can achieve good results in reconstructing 3D shapes with skeleton structures， these methods tend to rely heavily on large datasets with annotations. In the unsupervised case， they are still prone to generating overstretched and partially crossed model meshes.The paper proposes an unsupervised GAN-based network framework for the registration of 3D models， using a 3D generator to generate model meshes that fit the scanned point clouds， and two novel discriminators to impose regularization constraints on the deformed meshes， which achieves good results on public datasets.

Key words: 3D reconstruction, 3D registration, adversarial networks, non-rigid objects

CLC Number:

TP183

Yutao JIANG, Junqi DIAO, Jun XIAO, Ying WANG. Unsupervised 3D registration of articulated shapes by shape and edge regularization[J]. Journal of University of Chinese Academy of Sciences, 2026, 43(5): 657-666.

Figures/Tables 9

Fig.1 Errors in 3D human model registration

Fig.2 Design details of the proposed method

Fig.3 Error distribution of 3D-CODED results

Fig.4 Structure of the discriminator

Table 1 Evaluation of reconstruction and registration results

方法	FAUST	SMAL	FAUST-inter	SMAL-Reg
3D-CODED	0.961	0.19	10.46	3.23
Deep-Cage	2.160	*	21.65	*
本文方法	0.844	0.16	7.64	2.66

Fig.5 Reconstruction model results

Fig.6 3D animal reconstruction

Table 2 Ablation studies on two discriminators

方法	FAUST-chamfer	FAUST-inter
Baseline	0.961	10.46
仅含形状判别	0.904	8.27
本文方法	0.844	7.64

Fig.7 Impact of two discriminators on reconstruction results

References 39

[1]	Romero J， Tzionas D， Black M J. Embodied hands： modeling and capturing hands and bodies together［J］. ACM Trans Graph，2017，36（6）： 1-17. DOI：10.1145/3130800.3130883 .
[2]	Loper M， Mahmood N， Romero J， et al. SMPL： a skinned multi-person linear model［M］//Seminal Graphics Papers： Pushing the Boundaries， Vol 2. 2023： 851-866. DOI：10.1145/3596711.3596800 .
[3]	Yang H T， Zhu H， Wang Y R， et al. FaceScape： a large-scale high quality 3D face dataset and detailed riggable 3D face prediction［C］//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. Seattle， WA， USA. IEEE， 2020： 598-607. DOI： 10.1109/CVPR42600.2020.00068 .
[4]	Sumner R W， Popović J. Deformation transfer for triangle meshes［J］. ACM Transactions on Graphics， 2004， 23（3）： 399-405. DOI： 10.1145/1015706.1015736 .
[5]	Amberg B， Romdhani S， Vetter T. Optimal step nonrigid ICP algorithms for surface registration［C］//2007 IEEE Conference on Computer Vision and Pattern Recognition. Minneapolis， MN， USA. IEEE， 2007： 1-8. DOI： 10.1109/CVPR.2007.383165 .
[6]	Feng A W， Casas D， Shapiro A. Avatar reshaping and automatic rigging using a deformable model［C］//Proceedings of the 8th ACM SIGGRAPH Conference on Motion in Games. 2015： 57-64. DOI：10.1145/2822013.2822017 .
[7]	Zuffi S， Black M J. The stitched puppet： a graphical model of 3d human shape and pose［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015： 3537-3546. DOI：10.1109/cvpr.2015.7298976 .
[8]	Halimi O， Imanuel I， Litany O， et al. The whole is greater than the sum of its nonrigid parts［EB/OL］. arXiv 2020： 2001.09650.（2020-01-27）［2024-05-09］..
[9]	Deprelle T， Groueix T， Fisher M， et al. Learning elementary structures for 3D shape generation and matching［J］. Advances in Neural Information Processing Systems 32， 2019： 7433-7443.
[10]	Zhou B， Franco J S， Bogo F， et al. Reconstructing human body mesh from point clouds by adversarial gp network［C］//Proceedings of the Asian Conference on Computer Vision. 2020. DOI：10.1007/978-3-030-69525-5_8 .
[11]	Groueix T， Fisher M， Kim V G， et al. 3d-coded： 3d correspondences by deep deformation［C］//Proceedings of the european conference on computer vision （ECCV）. 2018： 230-246. DOI：10.1007/978-3-030-01216-8_15 .
[12]	Li C L， Simon T， Saragih J， et al. Lbs autoencoder： self-supervised fitting of articulated meshes to point clouds［C］//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019： 11967-11976. DOI：10.1109/cvpr.2019.01224 .
[13]	Wang Y F， Aigerman N， Kim V G， et al. Neural cages for detail-preserving 3D deformations［C］//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. Seattle， WA， USA. IEEE， 2020： 72-80. DOI： 10.1109/CVPR42600.2020.00015 .
[14]	Bhatnagar B L， Sminchisescu C， Theobalt C， et al. Combining implicit function learning and parametric models for 3D human reconstruction［C］//Computer Vision–ECCV 2020： 16th European Conference， Glasgow， UK， August 23–28， 2020， Proceedings， Part II 16. Springer International Publishing， 2020： 311-329. DOI：10.1007/978-3-030-58536-5_19 .
[15]	Ginzburg D， Raviv D. Cyclic functional mapping： self-supervised correspondence between non-isometric deformable shapes［C］//Computer Vision–ECCV 2020： 16th European Conference， Glasgow， UK， August 23–28， 2020， Proceedings， Part V 16. Springer International Publishing， 2020： 36-52. DOI：10.1007/978-3-030-58558-7_3 .
[16]	Eisenberger M， Lahner Z， Cremers D. Smooth shells： multi-scale shape registration with functional maps［C］//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020： 12265-12274. DOI：10.1109/CVPR42600.2020.01228 .
[17]	Varol G， Romero J， Martin X， et al. Learning from synthetic humans［C］//Proceedings of the IEEE conference on computer vision and pattern recognition. 2017： 109-117. DOI：10.1109/CVPR.2017.492 .
[18]	Bogo F， Romero J， Loper M， et al. FAUST： dataset and evaluation for 3D mesh registration［C］//Proceedings of the IEEE conference on computer vision and pattern recognition. 2014： 3794-3801. DOI：10.1109/CVPR.2014.491 .
[19]	Zuffi S， Kanazawa A， Jacobs D W， et al. 3D menagerie： modeling the 3D shape and pose of animals［C］//Proceedings of the IEEE conference on computer vision and pattern recognition. 2017： 6365-6373. DOI：10.1109/CVPR.2017.586 .
[20]	Varol G， Ceylan D， Russell B， et al. Bodynet： volumetric inference of 3D human body shapes［C］//Proceedings of the European conference on computer vision （ECCV）. 2018： 20-38. DOI：10.1007/978-3-030-01234-2_2 .
[21]	Rakotosaona M J， Ovsjanikov M. Intrinsic point cloud interpolation via dual latent space navigation［C］//European Conference on Computer Vision. Cham： Springer， 2020： 655-672.DOI：10.1007/978-3-030-58536-5_39 .
[22]	Ranjan A， Bolkart T， Sanyal S， et al. Generating 3D faces using convolutional mesh autoencoders［C］//Proceedings of the European conference on computer vision （ECCV）. 2018： 704-720. DOI：10.1007/978-3-030-01219-9_43 .
[23]	Kanazawa A， Tulsiani S， Efros A A， et al. Learning category-specific mesh reconstruction from image collections［C］//Proceedings of the European Conference on Computer Vision （ECCV）. 2018： 386-402. DOI：10.1007/978-3-030-01267-0_23 .
[24]	Xu H Y， Bazavan E G， Zanfir A， et al. GHUM & GHUML： generative 3D human shape and articulated pose models［C］//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. Seattle， WA， USA. IEEE， 2020： 6183-6192. DOI： 10.1109/CVPR42600.2020.00622 .
[25]	Chibane J， Alldieck T， Pons-Moll G. Implicit functions in feature space for 3D shape reconstruction and completion［C］//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020： 6970-6981. DOI：10.1109/CVPR42600.2020.00700 .
[26]	Saito S， Huang Z， Natsume R， et al. Pifu： pixel-aligned implicit function for high-resolution clothed human digitization［C］//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019： 2304-2314. DOI：10.1109/ICCV.2019.00239 .
[27]	Sun J， Ovsjanikov M， Guibas L. A concise and provably informative multi‐scale signature based on heat diffusion［C］//Computer graphics forum. Oxford， UK： Blackwell Publishing Ltd， 2009， 28（5）： 1383-1392. DOI：10.1111/J.1467-8659.2009.01515.X .
[28]	Marin R， Melzi S， Rodol􀅣 E， et al. FARM： functional automatic registration method for 3D human bodies［J］. Computer Graphics Forum， 2020， 39（1）： 160-173. DOI： 10.1111/cgf.13751 .
[29]	Ovsjanikov M， Corman E， Bronstein M， et al. Computing and processing correspondences with functional maps［M］//SIGGRAPH ASIA 2016 Courses. 2016： 1-60. DOI：10.1145/2988458.2988494 .
[30]	Sharma A， Ovsjanikov M. Weakly supervised deep functional maps for shape matching［C］//Proceedings of the 34th International Conference on Neural Information Processing Systems， ACM. 2020： 19264-19275.
[31]	Hirshberg D A， Loper M， Rachlin E， et al. Evaluating the automated alignment of 3D human body scans［C］//2nd International Conference on 3D Body Scanning Technologies. Hometrica Consulting， 2011： 76-85. DOI：10.15221/11.076 .
[32]	Hirshberg D A， Loper M， Rachlin E， et al. Coregistration： simultaneous alignment and modeling of articulated 3D shape［C］//Computer Vision–ECCV 2012： 12th European Conference on Computer Vision， Florence， Italy， October 7-13， 2012， Proceedings， Part VI 12. Springer Berlin Heidelberg， 2012： 242-255. DOI：10.1007/978-3-642-33783-3_18 .
[33]	Goodfellow I J， Pouget-Abadie J， Mirza M， et al. Generative adversarial nets［EB/OL］. arXiv 2014：1406.2661.（2014-06-10）［2024-05-09］. .
[34]	Abrevaya V F， Boukhayma A， Wuhrer S， et al. A decoupled 3d facial shape model by adversarial training［C］//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019： 9419-9428. DOI：10.1109/ICCV.2019.00951 .
[35]	Kanazawa A， Black M J， Jacobs D W， et al. End-to-end recovery of human shape and pose［C］//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018： 7122-7131. DOI：10.1109/CVPR.20 18.00744 .
[36]	Ma Q L， Yang J L， Ranjan A， et al. Learning to dress 3D people in generative clothing［C］//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. Seattle， WA， USA. IEEE， 2020： 6468-6477. DOI： 10.1109/CVPR42600.2020.00650 .
[37]	Charles R Q， Hao S， Mo K C， et al. PointNet： deep learning on point sets for 3D classification and segmentation［C］//2017 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. Honolulu， HI， USA. IEEE， 2017： 77-85. DOI： 10.1109/CVPR.2017.16 .
[38]	Isola P， Zhu J Y， Zhou T H， et al. Image-to-image translation with conditional adversarial networks［C］//2017 IEEE Conference on Computer Vision and Pattern Recognition （CVPR）. Honolulu， HI， USA. IEEE， 2017： 5967-5976. DOI： 10.1109/CVPR.2017.632 .
[39]	Mao X D， Li Q， Xie H R， et al. Least squares generative adversarial networks［C］//2017 IEEE International Conference on Computer Vision （ICCV）. Venice， Italy. IEEE， 2017： 2813-2821. DOI： 10.1109/ICCV.2017.304 .