欢迎访问中国科学院大学学报,今天是
电子信息与计算机科学

融合扩散模型和流形梯度约束的无人机图像拼接方法

  • 王杰 ,
  • 罗永曦 ,
  • 陈俊 ,
  • 吴业炜
展开
  • 1.广州大学电子与通信工程学院,广州 510006
    2.中国科学院空天信息创新研究院,北京 100094
    3.中国科学院大学计算机科学与技术学院,北京 100049
    4.中国科学院计算机网络信息中心,北京 100083

收稿日期: 2024-04-18

  修回日期: 2024-06-04

  网络出版日期: 2024-06-24

基金资助

中国科学院青年促进会(E0331804)

UAV image stitching method based on diffusion model and manifold gradient constraint

  • Jie WANG ,
  • Yongxi LUO ,
  • Jun CHEN ,
  • Yewei WU
Expand
  • 1.School of Electronics and Communication Engineering,Guangzhou University,Guangzhou 510006,China
    2.Aerospace Information Research Institute,Chinese Academy of Sciences,Beijing 100094,China
    3.School of Computer Science and Technology,University of Chinese Academy of Sciences,Beijing 100049,China
    4.Computer Network Information Center,Chinese Academy of Sciences,Beijing 100083,China

Received date: 2024-04-18

  Revised date: 2024-06-04

  Online published: 2024-06-24

摘要

无人机图像拼接是无人机遥感应用的必要前置步骤,但拼接结果中通常会存在大块不规则边界和多个拼接缝,影响后续分析和应用。现有方法通常不能同时解决这2个问题,需要对图像拼接结果进行多次处理,造成多次误差传递。针对这一问题,提出一种基于去噪离散扩散模型(DDPM)的无人机图像拼接补全方法,该方法对不规则边界和拼接缝统一设计掩码,利用带有流形梯度先验约束的扩散模型补全掩码区域的图像,可同时消除不规则边界与拼接缝,改进拼接结果的质量。利用不同场景数据开展对比实验,结果表明该方法有效消除了拼接结果中的不规则边界与拼接缝,修复后从局部到全局图像质量评价(PaQ-2-PiQ)与多尺度图像质量转换评价模型(MUSIQ)得分分别提升4.36%和15.37%,不规则边界处的结构相似性(SSIM)和峰值信噪比(PSNR)分别提升20.22%与33.69%。与SOTA方法和其他经典的图像拼接算法的对比实验结果表明,该方法在主观和客观2种质量评价方式下都优于其他方法,具有良好的鲁棒性和泛化性,可广泛应用于无人机图像拼接场景。

本文引用格式

王杰 , 罗永曦 , 陈俊 , 吴业炜 . 融合扩散模型和流形梯度约束的无人机图像拼接方法[J]. 中国科学院大学学报, 2026 , 43(2) : 252 -264 . DOI: 10.7523/j.ucas.2024.061

Abstract

Image stitching is a crucial prerequisite step for unmanned aerial vehicle (UAV) remote sensing applications, while the stitched images using most of the current image stitching methods often suffer from large irregular boundaries and multiple stitching seams, which can seriously affect subsequent analysis and applications. Existing improved methods typically cannot simultaneously address these two issues, and integrating the two types of methods in sequence is a straightforward way to solve the two problems, while this often can not obtain satisfactory performance because of the inevitable error propagation problem. This paper proposes an inpainting method for the UAV image stitching task based on the denoising diffusion probability model (DDPM). The method uniformly designs masks for irregular boundaries and stitching seams, and a diffusion model is then utilized with manifold gradient prior constraints to complete the masked regions. By doing so, both irregular boundaries and stitching seams are simultaneously eliminated, thereby improving the quality of the stitching results. Comparative experiments are conducted using four datasets established for different scenarios. The experimental results demonstrated the efficacy of the proposed method in effectively eliminating irregular boundaries and seams in the image stitching. Moreover, from patches to pictures quality (PaQ-2-PiQ) and multi-scale image quality (MUSIQ) scores increased by 4.36% and 15.37%, respectively. Furthermore, at the locations of irregular boundaries, the structural similarity (SSIM) and peak signal-to-noise ratio (PSNR) values improved by 20.22% and 33.69%, respectively. Compared with state-of-the-art methods and other conventional image stitching algorithms, the proposed method performs better in both subjective and objective quality metric scores, has good robustness and generalization, and can be widely applied to UAV image stitching scenarios.

参考文献

[1] He K M, Chang H W, Sun J. Rectangling panoramic images via warping[J]. ACM Transactions on Graphics201332(4):79. DOI: 10.1145/2461912.2462004 .
[2] Li D P, He K M, Sun J, et al. A geodesic-preserving method for image warping[C]//2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, MA. IEEE, 2015:213-221.DOI: 10.1109/CVPR.2015.7298617 .
[3] Zhang Y, Lai Y K, Zhang F L. Content-preserving image stitching with piecewise rectangular boundary constraints[J]. IEEE Transactions on Visualization and Computer Graphics202127(7): 3198-3212. DOI: 10.1109/TVCG.2020.2965097 .
[4] Nie L, Lin C Y, Liao K, et al. Deep rectangling for image stitching: a learning baseline[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New Orleans, LA, USA. IEEE, 2022: 5730-5738. DOI: 10.1109/CVPR52688.2022.00565 .
[5] Kwatra V, Sch?dl A, Essa I, et al. Graphcut textures: image and video synthesis using graph cuts[J]. ACM Transactions on Graphics200322(3): 277-286. DOI: 10.1145/882262.882264 .
[6] Eden A, Uyttendaele M, Szeliski R. Seamless image stitching of scenes with large motions and exposure differences[C]//2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06). New York, NY, USA. IEEE, 2006: 2498-2505. DOI: 10.1109/CVPR.2006.268 .
[7] Li N, Liao T L, Wang C. Perception-based seam cutting for image stitching[J]. Signal, Image and Video Processing201812(5): 967-974. DOI: 10.1007/s11760-018-1241-9 .
[8] Gao J, Li Y, Chin T J,et al.Seam-driven image stitching[C]//Eurographics (Short Papers),2013.DOI:10.2312/CONF/EG2013/SHORT/045-048 .
[9] Zhang F, Liu F. Parallax-tolerant image stitching[C]//2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus, OH, USA. IEEE, 2014: 3262-3269. DOI: 10.1109/CVPR.2014.423 .
[10] Lin K M, Jiang N J, Cheong L F, et al. SEAGULL: seam-guided local alignment for parallax-tolerant image stitching[C]//European Conference on Computer Vision. Cham: Springer, 2016: 370-385.10.1007/978-3-319-46487-9_23.
[11] Dai Q Y, Fang F M, Li J C, et al. Edge-guided composition network for image stitching[J]. Pattern Recognition2021118: 108019. DOI: 10.1016/j.patcog.2021.108019 .
[12] Nie L, Lin C Y, Liao K, et al. Parallax-tolerant unsupervised deep image stitching[C]//2023 IEEE/CVF International Conference on Computer Vision (ICCV). Paris, France. IEEE, 2023: 7365-7374. DOI: 10.1109/ICCV51070.2023.00680 .
[13] Sohl-Dickstein J, Weiss E A, Maheswaranathan N, et al. Deep unsupervised learning using nonequilibrium thermodynamics[C]//Proceedings of the 32nd International Conference on International Conference on Machine Learning-Volume 37. July 6 - 11, 2015, Lille, France. ACM, 2015: 2256-2265. DOI: 10.5555/3045118.3045358 .
[14] Ho J, Jain A, Abbeel P. Denoising diffusion probabilistic models[C]//Proceedings of the 34th International Conference on Neural Information Processing Systems. December 6 - 12, 2020,Vancouver, BC, Canada. ACM, 2020: 6840-6851. DOI: 10.5555/3495724.3496298 .
[15] Nichol A, Dhariwal P. Improved denoising diffusion probabilistic models[EB/OL]. arXiv 2021:2102.09672. (2021-02-18)[2024-05-20]. DOI: 10.48550/arXiv.2102.09672 .
[16] Zhu Y Z, Zhang K, Liang J Y, et al. Denoising diffusion models for plug-and-play image restoration[C]//2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Vancouver, BC, Canada. IEEE, 2023: 1219-1229. DOI: 10.1109/CVPRW59228.2023.00129 .
[17] Lugmayr A, Danelljan M, Romero A, et al. RePaint: inpainting using denoising diffusion probabilistic models[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New Orleans, LA, USA. IEEE, 2022: 11451-11461. DOI: 10.1109/CVPR52688.2022.01117 .
[18] Chung H, Sim B, Ryu D, et al. Improving diffusion models for inverse problems using manifold constraints[C]//Proceedings of the 36th International Conference on Neural Information Processing Systems. 28 November 2022, New Orleans, LA, USA. ACM, 2022: 25683-25696. DOI: 10.5555/3600270.3602132 .
[19] Dhariwal P, Nichol A. Diffusion models beat GANs on image synthesis[EB/OL]. arXiv 2021: 2105.05233.(2021-06-01)[2024-05-20]. .
[20] Lowe D G. Object recognition from local scale-invariant features[C]//Proceedings of the Seventh IEEE International Conference on Computer Vision. Kerkyra, Greece. IEEE, 1999: 1150-1157. DOI: 10.1109/ICCV.1999.790410 .
[21] Fischler M A, Bolles R C. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography[M]//Readings in Computer Vision. Amsterdam: Elsevier, 1987: 726-740. DOI: 10.1016/b978-0-08-051581-6.50070-2 .
[22] Chen Y B, Liu S F, Wang X L. Learning continuous image representation with local implicit image function[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Nashville, TN, USA. IEEE, 2021: 8624-8634. DOI: 10.1109/CVPR46437.2021.00852 .
[23] Lim B, Son S, Kim H, et al. Enhanced deep residual networks for single image super-resolution[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Honolulu, HI, USA. IEEE, 2017: 1132-1140. DOI: 10.1109/CVPRW.2017.151 .
[24] Ke J J, Wang Q F, Wang Y L, et al. MUSIQ: multi-scale image quality transformer[C]//2021 IEEE/CVF International Conference on Computer Vision (ICCV). Montreal, QC, Canada. IEEE, 2021: 5128-5137. DOI: 10.1109/ICCV48922.2021.00510 .
[25] Ying Z Q, Niu H R, Gupta P, et al. From patches to pictures (PaQ-2-PiQ): mapping the perceptual space of picture quality[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle, WA, USA. IEEE, 2020: 3572-3582. DOI: 10.1109/CVPR42600.2020.00363 .
[26] Jia Q, Li Z J, Fan X, et al. Leveraging line-point consistence to preserve structures for wide parallax image stitching[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Nashville, TN, USA. IEEE, 2021: 12181-12190. DOI: 10.1109/CVPR46437.2021.01201 .
[27] Liao T L, Li N. Single-perspective warps in natural image stitching[J]. IEEE Transactions on Image Processing201929:724-735. DOI: 10.1109/TIP.2019.2934344 .
[28] Lin C C, Pankanti S U, Ramamurthy K N, et al. Adaptive as-natural-as-possible image stitching[C]//2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, MA, USA. IEEE, 2015: 1155-1163. DOI: 10.1109/CVPR.2015.7298719 .
[29] Zaragoza J, Chin T J, Brown M S, et al. As-projective-as-possible image stitching with moving DLT[C]//2013 IEEE Conference on Computer Vision and Pattern Recognition. Portland, OR, USA. IEEE, 2013: 2339-2346. DOI: 10.1109/CVPR.2013.303 .
文章导航

/