A method for SAR-to-optical image synthesis based on bi-temporal features

doi:10.7523/j.ucas.2024.007

Abstract

Abstract: The robust optical image time series are of great value in many applications of remote sensing. However, due to the effects of weather conditions like clouds and rains, it is very difficult to obtain such robust time series of optical images in many regions. Using the all-weather imaging capability of synthetic aperture radar (SAR) to generate optical images from SAR images is an effective solution to the missing data of optical images. But there is still a problem that the quality of generated images in complicated scenarios is much worse than those in simple scenarios. In this paper, we build bi-temporal datasets of different scenarios based on Sentinel imagery and propose an improved generator of conditional generation adversarial network. The encoder-decoder based generator learns to extract and fuse the bi-temporal polarized SAR features and the additional optical features from the source time phase. In addition, a strategy to balance the weights of SAR and optical features is adopted. Comparison experiments show that our method is the best on FID and PSNR among all evaluated methods. The proposed method significantly reduces the gap in the quality of generated images between simple scenario and complicated scenario. The ablation study shows that our method outperforms the baseline model by 46 in FID, 6.6 in PSNR and 0.44 in SSIM. Our method efficiently improves the quality of generated images in different scenarios.

Key words: optical image synthesis, SAR, generative adversarial network

CLC Number:

TP751

WENG Yongchun, MA Yong, ChEN Fu, SHANG Erping, YAO Wutao, ZHANG Shuyan, YANG Jin, LIU Jianbo. A method for SAR-to-optical image synthesis based on bi-temporal features[J]. Journal of University of Chinese Academy of Sciences, DOI: 10.7523/j.ucas.2024.007.

References

[1] Fuentes Reyes M, Auer S, Merkle N, et al.SAR-to-optical image translation based on conditional generative adversarial networks—Optimization, opportunities and limits[J]. Remote Sensing, 2019, 11(17): 2067. DOI:10.3390/rs11172067.
[2] Isola P, Zhu J Y, Zhou T H, et al.Image-to-image translation with conditional adversarial networks[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). July 21-26, 2017, Honolulu, HI, USA. IEEE, 2017: 5967-5976. DOI: 10.1109/CVPR.2017.632.
[3] Bermudez J D, Happ P N, Oliveira D A B, et al. SAR to optical image synthesis for cloud removal with generative adversarial networks[J]. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2018, IV-1: 5-11. DOI:10.5194/isprs-annals-iv-1-5-2018.
[4] Grohnfeldt C, Schmitt M, Zhu X X.A conditional generative adversarial network to fuse SAR and multispectral optical data for cloud removal from Sentinel-2 images[C]//IGARSS 2018-2018 IEEE International Geoscience and Remote Sensing Symposium. July 22-27, 2018, Valencia, Spain. IEEE, 2018: 1726-1729. DOI:10.1109/IGARSS.2018.8519215.
[5] Ebel P, Schmitt M, Zhu X X.Cloud removal in unpaired Sentinel-2 imagery using cycle-consistent GAN and SAR-optical data fusion[C]//IGARSS 2020-2020 IEEE International Geoscience and Remote Sensing Symposium. September 26 - October 2, 2020, Waikoloa, HI, USA. IEEE, 2020: 2065-2068. DOI:10.1109/IGARSS39084.2020.9324060.
[6] Gao J H, Yuan Q Q, Li J E, et al.Cloud removal with fusion of high resolution optical and SAR images using generative adversarial networks[J]. Remote Sensing, 2020, 12(1): 191. DOI:10.3390/rs12010191.
[7] Zuo Z C, Li Y X.A SAR-to-optical image translation method based on PIX2PIX[C]//2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS. July 11-16, 2021, Brussels, Belgium. IEEE, 2021: 3026-3029. DOI:10.1109/IGARSS47720.2021.9555111.
[8] 王磊. 深度学习框架下的极化SAR影像信息表达与分类研究[D]. 武汉: 武汉大学,2020. DOI:10.27379/d.cnki.gwhdu.2020.000008.
[9] Baier G, Deschemps A, Schmitt M, et al.Synthesizing optical and SAR imagery from land cover maps and auxiliary raster data[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 1-12. DOI:10.1109/TGRS.2021.3068532.
[10] Yang X, Wang Z H, Zhao J Y, et al.FG-GAN: A fine-grained generative adversarial network for unsupervised SAR-to-optical image translation[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 1-11. DOI:10.1109/TGRS.2022.3165371.
[11] Mao X D, Li Q, Xie H R, et al.Least squares generative adversarial networks[C]//2017 IEEE International Conference on Computer Vision (ICCV). October 22-29, 2017, Venice, Italy. IEEE, 2017: 2813-2821. DOI:10.1109/ICCV.2017.304.
[12] Arjovsky M, Chintala S, Bottou L.Wasserstein generative adversarial networks[C]// Proceedings of the 34th International Conference on Machine Learning (PMLR) - Volume 70. August 6 - 11, 2017, Sydney, NSW, Australia. New York: ACM, 2017: 214-223.DOI: 10.5555/3305381.3305404
[13] Karras T, Laine S, Aila T.A style-based generator architecture for generative adversarial networks[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach, CA, USA. IEEE, 2019: 4396-4405. DOI: 10.1109/CVPR.2019.00453.
[14] Wang T C, Liu M Y, Zhu J Y, et al.High-resolution image synthesis and semantic manipulation with conditional GANs[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. June 18-23, 2018, Salt Lake City, UT, USA. IEEE, 2018: 8798-8807. DOI:10.1109/CVPR.2018.00917.
[15] Zhang H, Goodfellow I, Metaxas D, et al.Self-attention generative adversarial networks[C]// Proceedings of the 36th International Conference on Machine Learning (PMLR) - Volume 97. June 9 - 15, 2019, Long Beach, California, USA. 2019: 7354-7363. DOI:10.48550/arXiv.1805.08318
[16] Schmitt M, Hughes L H, Zhu X X. The sen1-2 dataset for deep learning in sar-optical data fusion[J]. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2018, IV-1: 141-146. DOI: 10.5194/isprs-annals-iv-1-141-2018.
[17] Uziel R, Ronen M, Freifeld O.Bayesian adaptive superpixel segmentation[C]//2019 IEEE/CVF International Conference on Computer Vision (ICCV). October 27 - November 2, 2019, Seoul, Korea (South). IEEE, 2020:8469-8478.DOI:10.1109/ICCV.2019.00856.
[18] Johnson J, Alahi A, Li F F.Perceptual losses for real-time style transfer and super-resolution[C]//ECCV 2016: 14th European Conference on Computer Vision, October 11-14, 2016, Amsterdam, The Netherlands, Proceedings, Part II 14. Springer International Publishing, 2016: 694-711. DOI:10.1007/978-3-319-46475-6_43.
[19] Louis, J. Sentinel 2 MSI—Level 2A Product Definition: Issue 4.4[EB/OL]. (2016.04.01)[2022.09.21]. https://sentinel.esa.int/documents/247904/1848117/Sentinel-2-Level-2A-Product-Definition-Document.pdf.
[20] Kingma D P, Ba J. Adam: A method for stochastic optimization[EB/OL].2014: arXiv: 1412.6980.(2014-12-22)[2023-02-12] https://arxiv.org/abs/1412.6980.pdf.
[21] Heusel M, Ramsauer H, Unterthiner T, et al.GANs trained by a two time-scale update rule converge to a local nash equilibrium[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. December 4 - 9, 2017, Long Beach, California, USA. New York: ACM, 2017: 6629-6640. DOI:10.48550/arXiv.1706.08500.
[22] Salimans T, Goodfellow I, Zaremba W, et al.Improved techniques for training GANs[C]//Proceedings of the 30th International Conference on Neural Information Processing Systems. December 5 - 10, 2016, Barcelona, Spain. New York: ACM, 2016: 2234-2242. DOI:10.48550/arXiv.1606.03498.
[23] Szegedy C, Vanhoucke V, Ioffe S, et al.Rethinking the inception architecture for computer vision[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVRP). June 27-30, 2016, Las Vegas, NV, USA. IEEE, 2016: 2818-2826.DOI:10.1109/CVPR.2016.308.
[24] Wang Y J, Li J H, Lu Y, et al.Image quality evaluation based on image weighted separating block peak signal to noise ratio[C]//International Conference on Neural Networks and Signal Processing, 2003. Proceedings of the 2003. December 14-17, 2003, Nanjing, China. IEEE, 2004: 994-997. DOI:10.1109/ICNNSP.2003.1281036.
[25] Sara U, Akter M, Uddin M S.Image quality assessment through FSIM, SSIM, MSE and PSNR—a comparative study[J]. Journal of Computer and Communications, 2019, 7(3): 8-18. DOI:10.4236/jcc.2019.73002.
[26] Richardson E, Alaluf Y, Patashnik O, et al.Encoding in style: a StyleGAN encoder for image-to-image translation[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). June 20-25, 2021, Nashville, TN, USA. IEEE, 2021: 2287-2296. DOI: 10.1109/CVPR46437.2021.00232.
[27] Tang H, Xu D, Sebe N, et al.Multi-channel attention selection gan with cascaded semantic guidance for cross-view image translation[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). June 15-20, 2019, Long Beach, CA, USA. IEEE, 2019: 2417-2426. DOI:10.1109/CVPR.2019.00252.
[28] Gao F, Xu X X, Yu J, et al.Complementary, heterogeneous and adversarial networks for image-to-image translation[J]. IEEE Transactions on Image Processing, 2021, 30: 3487-3498. DOI:10.1109/TIP.2021.3061286.
[29] Esser P, Rombach R, Ommer B.Taming transformers for high-resolution image synthesis[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). June 20-25, 2021, Nashville, TN, USA. IEEE, 2021: 12868-12878. DOI:10.1109/CVPR46437.2021.01268.