基于双时相特征的SAR生成光学影像方法

doi:10.7523/j.ucas.2024.007

摘要/Abstract

摘要： 稳定连续的遥感光学影像时间序列应用价值巨大,但很多地区受云雨影响难以获取到这样的时序光学影像。利用合成孔径雷达(SAR)全天时、全天候成像能力,由SAR影像生成光学影像,能有效解决光学影像缺失问题,但目前复杂场景下生成质量与简单场景相比差距显著。本文基于哨兵卫星影像构建不同场景的双时相数据集,并改进了条件对抗网络生成器,以提取和融合双时相SAR特征和辅助光学特征,同时采用一种策略来平衡SAR和光学特征权重。与其他方法相比,所提出的模型FID和PSNR指标为最佳,复杂场景和简单场景下生成质量差距明显缩小;消融实验表明,所提出的模型与基准模型相比,FID下降46,PSNR提升6.6,SSIM提升0.44。该方法有效提升了不同场景下光学影像生成质量。

关键词: 可见光影像生成, 合成孔径雷达, 生成对抗网络

Abstract: The robust optical image time series are of great value in many applications of remote sensing. However, due to the effects of weather conditions like clouds and rains, it is very difficult to obtain such robust time series of optical images in many regions. Using the all-weather imaging capability of synthetic aperture radar (SAR) to generate optical images from SAR images is an effective solution to the missing data of optical images. But there is still a problem that the quality of generated images in complicated scenarios is much worse than those in simple scenarios. In this paper, we build bi-temporal datasets of different scenarios based on Sentinel imagery and propose an improved generator of conditional generation adversarial network. The encoder-decoder based generator learns to extract and fuse the bi-temporal polarized SAR features and the additional optical features from the source time phase. In addition, a strategy to balance the weights of SAR and optical features is adopted. Comparison experiments show that our method is the best on FID and PSNR among all evaluated methods. The proposed method significantly reduces the gap in the quality of generated images between simple scenario and complicated scenario. The ablation study shows that our method outperforms the baseline model by 46 in FID, 6.6 in PSNR and 0.44 in SSIM. Our method efficiently improves the quality of generated images in different scenarios.

Key words: optical image synthesis, SAR, generative adversarial network

中图分类号:

TP751

翁永椿, 马勇, 陈甫, 尚二萍, 姚武韬, 仉淑艳, 杨进, 刘建波. 基于双时相特征的SAR生成光学影像方法[J]. 中国科学院大学学报, DOI: 10.7523/j.ucas.2024.007.

WENG Yongchun, MA Yong, ChEN Fu, SHANG Erping, YAO Wutao, ZHANG Shuyan, YANG Jin, LIU Jianbo. A method for SAR-to-optical image synthesis based on bi-temporal features[J]. Journal of University of Chinese Academy of Sciences, DOI: 10.7523/j.ucas.2024.007.

参考文献

[1] Fuentes Reyes M, Auer S, Merkle N, et al.SAR-to-optical image translation based on conditional generative adversarial networks—Optimization, opportunities and limits[J]. Remote Sensing, 2019, 11(17): 2067. DOI:10.3390/rs11172067.
[2] Isola P, Zhu J Y, Zhou T H, et al.Image-to-image translation with conditional adversarial networks[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). July 21-26, 2017, Honolulu, HI, USA. IEEE, 2017: 5967-5976. DOI: 10.1109/CVPR.2017.632.
[3] Bermudez J D, Happ P N, Oliveira D A B, et al. SAR to optical image synthesis for cloud removal with generative adversarial networks[J]. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2018, IV-1: 5-11. DOI:10.5194/isprs-annals-iv-1-5-2018.
[4] Grohnfeldt C, Schmitt M, Zhu X X.A conditional generative adversarial network to fuse SAR and multispectral optical data for cloud removal from Sentinel-2 images[C]//IGARSS 2018-2018 IEEE International Geoscience and Remote Sensing Symposium. July 22-27, 2018, Valencia, Spain. IEEE, 2018: 1726-1729. DOI:10.1109/IGARSS.2018.8519215.
[5] Ebel P, Schmitt M, Zhu X X.Cloud removal in unpaired Sentinel-2 imagery using cycle-consistent GAN and SAR-optical data fusion[C]//IGARSS 2020-2020 IEEE International Geoscience and Remote Sensing Symposium. September 26 - October 2, 2020, Waikoloa, HI, USA. IEEE, 2020: 2065-2068. DOI:10.1109/IGARSS39084.2020.9324060.
[6] Gao J H, Yuan Q Q, Li J E, et al.Cloud removal with fusion of high resolution optical and SAR images using generative adversarial networks[J]. Remote Sensing, 2020, 12(1): 191. DOI:10.3390/rs12010191.
[7] Zuo Z C, Li Y X.A SAR-to-optical image translation method based on PIX2PIX[C]//2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS. July 11-16, 2021, Brussels, Belgium. IEEE, 2021: 3026-3029. DOI:10.1109/IGARSS47720.2021.9555111.
[8] 王磊. 深度学习框架下的极化SAR影像信息表达与分类研究[D]. 武汉: 武汉大学,2020. DOI:10.27379/d.cnki.gwhdu.2020.000008.
[9] Baier G, Deschemps A, Schmitt M, et al.Synthesizing optical and SAR imagery from land cover maps and auxiliary raster data[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 1-12. DOI:10.1109/TGRS.2021.3068532.
[10] Yang X, Wang Z H, Zhao J Y, et al.FG-GAN: A fine-grained generative adversarial network for unsupervised SAR-to-optical image translation[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 1-11. DOI:10.1109/TGRS.2022.3165371.
[11] Mao X D, Li Q, Xie H R, et al.Least squares generative adversarial networks[C]//2017 IEEE International Conference on Computer Vision (ICCV). October 22-29, 2017, Venice, Italy. IEEE, 2017: 2813-2821. DOI:10.1109/ICCV.2017.304.
[12] Arjovsky M, Chintala S, Bottou L.Wasserstein generative adversarial networks[C]// Proceedings of the 34th International Conference on Machine Learning (PMLR) - Volume 70. August 6 - 11, 2017, Sydney, NSW, Australia. New York: ACM, 2017: 214-223.DOI: 10.5555/3305381.3305404
[13] Karras T, Laine S, Aila T.A style-based generator architecture for generative adversarial networks[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach, CA, USA. IEEE, 2019: 4396-4405. DOI: 10.1109/CVPR.2019.00453.
[14] Wang T C, Liu M Y, Zhu J Y, et al.High-resolution image synthesis and semantic manipulation with conditional GANs[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. June 18-23, 2018, Salt Lake City, UT, USA. IEEE, 2018: 8798-8807. DOI:10.1109/CVPR.2018.00917.
[15] Zhang H, Goodfellow I, Metaxas D, et al.Self-attention generative adversarial networks[C]// Proceedings of the 36th International Conference on Machine Learning (PMLR) - Volume 97. June 9 - 15, 2019, Long Beach, California, USA. 2019: 7354-7363. DOI:10.48550/arXiv.1805.08318
[16] Schmitt M, Hughes L H, Zhu X X. The sen1-2 dataset for deep learning in sar-optical data fusion[J]. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2018, IV-1: 141-146. DOI: 10.5194/isprs-annals-iv-1-141-2018.
[17] Uziel R, Ronen M, Freifeld O.Bayesian adaptive superpixel segmentation[C]//2019 IEEE/CVF International Conference on Computer Vision (ICCV). October 27 - November 2, 2019, Seoul, Korea (South). IEEE, 2020:8469-8478.DOI:10.1109/ICCV.2019.00856.
[18] Johnson J, Alahi A, Li F F.Perceptual losses for real-time style transfer and super-resolution[C]//ECCV 2016: 14th European Conference on Computer Vision, October 11-14, 2016, Amsterdam, The Netherlands, Proceedings, Part II 14. Springer International Publishing, 2016: 694-711. DOI:10.1007/978-3-319-46475-6_43.
[19] Louis, J. Sentinel 2 MSI—Level 2A Product Definition: Issue 4.4[EB/OL]. (2016.04.01)[2022.09.21]. https://sentinel.esa.int/documents/247904/1848117/Sentinel-2-Level-2A-Product-Definition-Document.pdf.
[20] Kingma D P, Ba J. Adam: A method for stochastic optimization[EB/OL].2014: arXiv: 1412.6980.(2014-12-22)[2023-02-12] https://arxiv.org/abs/1412.6980.pdf.
[21] Heusel M, Ramsauer H, Unterthiner T, et al.GANs trained by a two time-scale update rule converge to a local nash equilibrium[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. December 4 - 9, 2017, Long Beach, California, USA. New York: ACM, 2017: 6629-6640. DOI:10.48550/arXiv.1706.08500.
[22] Salimans T, Goodfellow I, Zaremba W, et al.Improved techniques for training GANs[C]//Proceedings of the 30th International Conference on Neural Information Processing Systems. December 5 - 10, 2016, Barcelona, Spain. New York: ACM, 2016: 2234-2242. DOI:10.48550/arXiv.1606.03498.
[23] Szegedy C, Vanhoucke V, Ioffe S, et al.Rethinking the inception architecture for computer vision[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVRP). June 27-30, 2016, Las Vegas, NV, USA. IEEE, 2016: 2818-2826.DOI:10.1109/CVPR.2016.308.
[24] Wang Y J, Li J H, Lu Y, et al.Image quality evaluation based on image weighted separating block peak signal to noise ratio[C]//International Conference on Neural Networks and Signal Processing, 2003. Proceedings of the 2003. December 14-17, 2003, Nanjing, China. IEEE, 2004: 994-997. DOI:10.1109/ICNNSP.2003.1281036.
[25] Sara U, Akter M, Uddin M S.Image quality assessment through FSIM, SSIM, MSE and PSNR—a comparative study[J]. Journal of Computer and Communications, 2019, 7(3): 8-18. DOI:10.4236/jcc.2019.73002.
[26] Richardson E, Alaluf Y, Patashnik O, et al.Encoding in style: a StyleGAN encoder for image-to-image translation[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). June 20-25, 2021, Nashville, TN, USA. IEEE, 2021: 2287-2296. DOI: 10.1109/CVPR46437.2021.00232.
[27] Tang H, Xu D, Sebe N, et al.Multi-channel attention selection gan with cascaded semantic guidance for cross-view image translation[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). June 15-20, 2019, Long Beach, CA, USA. IEEE, 2019: 2417-2426. DOI:10.1109/CVPR.2019.00252.
[28] Gao F, Xu X X, Yu J, et al.Complementary, heterogeneous and adversarial networks for image-to-image translation[J]. IEEE Transactions on Image Processing, 2021, 30: 3487-3498. DOI:10.1109/TIP.2021.3061286.
[29] Esser P, Rombach R, Ommer B.Taming transformers for high-resolution image synthesis[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). June 20-25, 2021, Nashville, TN, USA. IEEE, 2021: 12868-12878. DOI:10.1109/CVPR46437.2021.01268.