Optical remote sensing imaging is highly susceptible to clouds, which leads to partial information degradation or loss and significantly limits data availability. In contrast, synthetic aperture radar (SAR) is capable of all-weather and all-day imaging, providing stable structural information unaffected by cloud cover. To address the demand for high-quality utilization of multimodal remote sensing data in complex cloud-covered scenarios, this paper investigates an optical-SAR complementary information fusion approach for cloud removal. An attention-guided multimodal fusion framework is proposed, in which gated convolutional structures and multi-level attention mechanisms are jointly employed to enable multi-scale feature extraction and global cross-modal dependency modeling, thereby enhancing feature alignment and information interaction between optical and SAR modalities. Specifically, a cross-attention mechanism is introduced to guide SAR features in compensating for the information missing in cloud-covered optical regions. Furthermore, a multimodal cloud removal unit is designed to integrate deep features and map them back to the image space, effectively suppressing cloud artifacts while strengthening ground object representation to reconstruct cloud-free optical images. Experimental results demonstrate that the proposed method achieves notable improvements in cloud removal accuracy, detail preservation, and structural consistency compared with existing methods, and validates the effectiveness of multimodal complementary fusion for optical remote sensing image cloud removal.
[1] Wang N, Li W, Tao R, et al.Graph-based block-level urban change detection using Sentinel-2 time series[J]. Remote Sensing of Environment, 2022, 274: 112993. DOI:10.1016/j.rse.2022.112993.
[2] Huang B, Li Y, Han X Y, et al.Cloud removal from optical satellite imagery with SAR imagery using sparse representation[J]. IEEE Geoscience and Remote Sensing Letters, 2015, 12(5): 1046-1050. DOI:10.1109/LGRS.2014.2377476.
[3] Chen H, Wu C, Du B,et al.Change detection in multisource VHR images via deep Siamese convolutional multiple-layers recurrent neural network[J].IEEE Transactions on Geoscience and Remote Sensing, 2020, 58(4): 2848-2864. DOI:10.1109/TGRS.2019.2956756.
[4] Zhu C, Zhao Z Q, Zhu X Z, et al.Cloud removal for optical images using SAR structure data[C]//2016 IEEE 13th International Conference on Signal Processing (ICSP). November 6-10, 2016, Chengdu, China. IEEE, 2017: 1872-1875. DOI:10.1109/ICSP.2016.7878153.
[5] Xu F, Shi Y L, Ebel P, et al.GLF-CR: SAR-enhanced cloud removal with global-local fusion[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2022, 192: 268-278. DOI:10.1016/j.isprsjprs.2022.08.002.
[6] Liu L, Lei B.Can SAR images and optical images transfer with each other?[C]//IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium. July 22-27, 2018. Valencia. IEEE, 2018: 7019-7022. DOI:10.1109/igarss.2018.8518921.
[7] Bermudez J D, Happ P N, Oliveira D A B, et al. Sar to optical image synthesis for cloud removal with generative adversarial networks[J]. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2018, IV-1: 5-11. DOI:10.5194/isprs-annals-iv-1-5-2018.
[8] Grohnfeldt C, Schmitt M, Zhu X X.A conditional generative adversarial network to fuse sar and multispectral optical data for cloud removal from sentinel-2 images[C]//IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium. July 22-27, 2018, Valencia, Spain. IEEE, 2018: 1726-1729. DOI:10.1109/IGARSS.2018.8519215.
[9] Darbaghshahi F N, Mohammadi M R, Soryani M.Cloud removal in remote sensing images using generative adversarial networks and SAR-to-optical image translation[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 4105309. DOI:10.1109/TGRS.2021.3131035.
[10] Li Y, Fu R D, Meng X C, et al.A SAR-to-optical image translation method based on conditional generation adversarial network (cGAN)[J]. IEEE Access, 2020, 8: 60338-60343. DOI:10.1109/ACCESS.2020.2977103.
[11] Karras T, Laine S, Aila T.A style-based generator architecture for generative adversarial networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43(12): 4217-4228. DOI:10.1109/TPAMI.2020.2970919.
[12] Karras T, Laine S, Aittala M, et al.Analyzing and improving the image quality of StyleGAN[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). June 13-19, 2020. Seattle, WA, USA. IEEE, 2020: 8110-8119. DOI:10.1109/CVPR42600.2020.00813.
[13] Karras T, Aittala M, Laine S, et al.Alias-free generative adversarial networks[C]//Proceedings of the 35th International Conference on Neural Information Processing Systems. ACM, 2021: 852-863. DOI:10.5555/3540261.3540327
[14] Meraner A, Ebel P, Zhu X X, et al.Cloud removal in Sentinel-2 imagery using a deep residual neural network and SAR-optical data fusion[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2020, 166: 333-346. DOI:10.1016/j.isprsjprs.2020.05.013.
[15] He K M, Zhang X Y, Ren S Q, et al.Deep residual learning for image recognition[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). June 27-30, 2016, Las Vegas, NV, USA. IEEE, 2016: 770-778. DOI:10.1109/CVPR.2016.90.
[16] Zhang Q, Yuan Q, Li J, et al.Thick cloud and cloud shadow removal in multitemporal imagery using progressively spatio-temporal patch group deep learning[J].ISPRS Journal of Photogrammetry and Remote Sensing, 2020, 162:148-160.DOI:10.1016/j.isprsjprs.2020.02.008.
[17] Gao J H, Yi Y, Wei T, et al.Sentinel-2 cloud removal considering ground changes by fusing multitemporal SAR and optical images[J]. Remote Sensing, 2021, 13(19):3998.DOI:10.3390/rs13193998.
[18] Chen H, Yokoya N, Wu C, et al.Unsupervised multimodal change detection based on structural relationship graph representation learning[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 1-18, 5635318. DOI: 10.1109/TGRS.2022.3229027.
[19] Wang Y X, Zhang B, Zhang W J, et al.Cloud removal with SAR-optical data fusion using a unified spatial-spectral residual network[J]. IEEE Transactions on Geoscience and Remote Sensing, 2024, 62: 5600820. DOI:10.1109/TGRS.2023.3339210.
[20] Zhang S, Li X D, Zhou X Y, et al.Cloud removal using SAR and optical images via attention mechanism-based GAN[J]. Pattern Recognition Letters, 2023, 175(C): 8-15. DOI:10.1016/j.patrec.2023.09.014.
[21] Wang P, Chen Y K, Huang B, et al.MT_GAN: A SAR-to-optical image translation method for cloud removal[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2025, 225: 180-195. DOI:10.1016/j.isprsjprs.2025.04.011.
[22] Woo S, Park J, Lee J Y, et al.CBAM: Convolutional block attention module[C]//Computer Vision - ECCV 2018. Cham: Springer, 2018: 3-19. DOI:10.1007/978-3-030-01234-2_1.
[23] Ebel P, Meraner A, Schmitt M, et al.Multisensor data fusion for cloud removal in global and all-season sentinel-2 imagery[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59(7): 5866-5878. DOI:10.1109/TGRS.2020.3024744.
[24] Li J J, Zhang J C, Yang C, et al.Comparative analysis of pixel-level fusion algorithms and a new high-resolution dataset for SAR and optical image fusion[J]. Remote Sensing, 2023, 15(23): 5514. DOI:10.3390/rs15235514.
