光学遥感成像容易受到云的干扰,使得遥感图像中部分信息改变或丢失,降低数据的可用性。但合成孔径雷达（SAR）可以全天时全天候成像,围绕多模态遥感图像在复杂场景下的高质量利用需求,本文开展光学与SAR互补信息融合的去云方法研究。构建一种注意力引导的多模态融合模型,通过门控卷积结构与多层注意力机制协同建模,实现多尺度特征提取与跨模态全局依赖关系刻画,增强特征对齐与信息交互能力。利用交叉注意力引导SAR模态弥补云遮挡缺失的信息,并引入多模态去云单元整合深度特征,有效抑制云并强化地物表达,重建无云光学图像。实验结果表明,本文方法在去云精度、图像细节保持及结构一致性方面均取得明显提升,验证了多模态互补融合在遥感图像去云任务中的有效性。
Optical remote sensing imaging is highly susceptible to clouds, which leads to partial information degradation or loss and significantly limits data availability. In contrast, synthetic aperture radar (SAR) is capable of all-weather and all-day imaging, providing stable structural information unaffected by cloud cover. To address the demand for high-quality utilization of multimodal remote sensing data in complex cloud-covered scenarios, this paper investigates an optical-SAR complementary information fusion approach for cloud removal. An attention-guided multimodal fusion framework is proposed, in which gated convolutional structures and multi-level attention mechanisms are jointly employed to enable multi-scale feature extraction and global cross-modal dependency modeling, thereby enhancing feature alignment and information interaction between optical and SAR modalities. Specifically, a cross-attention mechanism is introduced to guide SAR features in compensating for the information missing in cloud-covered optical regions. Furthermore, a multimodal cloud removal unit is designed to integrate deep features and map them back to the image space, effectively suppressing cloud artifacts while strengthening ground object representation to reconstruct cloud-free optical images. Experimental results demonstrate that the proposed method achieves notable improvements in cloud removal accuracy, detail preservation, and structural consistency compared with existing methods, and validates the effectiveness of multimodal complementary fusion for optical remote sensing image cloud removal.
[1] Wang N, Li W, Tao R, et al.Graph-based block-level urban change detection using Sentinel-2 time series[J]. Remote Sensing of Environment, 2022, 274: 112993. DOI:10.1016/j.rse.2022.112993.
[2] Huang B, Li Y, Han X Y, et al.Cloud removal from optical satellite imagery with SAR imagery using sparse representation[J]. IEEE Geoscience and Remote Sensing Letters, 2015, 12(5): 1046-1050. DOI:10.1109/LGRS.2014.2377476.
[3] Chen H, Wu C, Du B,et al.Change detection in multisource VHR images via deep Siamese convolutional multiple-layers recurrent neural network[J].IEEE Transactions on Geoscience and Remote Sensing, 2020, 58(4): 2848-2864. DOI:10.1109/TGRS.2019.2956756.
[4] Zhu C, Zhao Z Q, Zhu X Z, et al.Cloud removal for optical images using SAR structure data[C]//2016 IEEE 13th International Conference on Signal Processing (ICSP). November 6-10, 2016, Chengdu, China. IEEE, 2017: 1872-1875. DOI:10.1109/ICSP.2016.7878153.
[5] Xu F, Shi Y L, Ebel P, et al.GLF-CR: SAR-enhanced cloud removal with global-local fusion[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2022, 192: 268-278. DOI:10.1016/j.isprsjprs.2022.08.002.
[6] Liu L, Lei B.Can SAR images and optical images transfer with each other?[C]//IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium. July 22-27, 2018. Valencia. IEEE, 2018: 7019-7022. DOI:10.1109/igarss.2018.8518921.
[7] Bermudez J D, Happ P N, Oliveira D A B, et al. Sar to optical image synthesis for cloud removal with generative adversarial networks[J]. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2018, IV-1: 5-11. DOI:10.5194/isprs-annals-iv-1-5-2018.
[8] Grohnfeldt C, Schmitt M, Zhu X X.A conditional generative adversarial network to fuse sar and multispectral optical data for cloud removal from sentinel-2 images[C]//IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium. July 22-27, 2018, Valencia, Spain. IEEE, 2018: 1726-1729. DOI:10.1109/IGARSS.2018.8519215.
[9] Darbaghshahi F N, Mohammadi M R, Soryani M.Cloud removal in remote sensing images using generative adversarial networks and SAR-to-optical image translation[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 4105309. DOI:10.1109/TGRS.2021.3131035.
[10] Li Y, Fu R D, Meng X C, et al.A SAR-to-optical image translation method based on conditional generation adversarial network (cGAN)[J]. IEEE Access, 2020, 8: 60338-60343. DOI:10.1109/ACCESS.2020.2977103.
[11] Karras T, Laine S, Aila T.A style-based generator architecture for generative adversarial networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43(12): 4217-4228. DOI:10.1109/TPAMI.2020.2970919.
[12] Karras T, Laine S, Aittala M, et al.Analyzing and improving the image quality of StyleGAN[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). June 13-19, 2020. Seattle, WA, USA. IEEE, 2020: 8110-8119. DOI:10.1109/CVPR42600.2020.00813.
[13] Karras T, Aittala M, Laine S, et al.Alias-free generative adversarial networks[C]//Proceedings of the 35th International Conference on Neural Information Processing Systems. ACM, 2021: 852-863. DOI:10.5555/3540261.3540327
[14] Meraner A, Ebel P, Zhu X X, et al.Cloud removal in Sentinel-2 imagery using a deep residual neural network and SAR-optical data fusion[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2020, 166: 333-346. DOI:10.1016/j.isprsjprs.2020.05.013.
[15] He K M, Zhang X Y, Ren S Q, et al.Deep residual learning for image recognition[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). June 27-30, 2016, Las Vegas, NV, USA. IEEE, 2016: 770-778. DOI:10.1109/CVPR.2016.90.
[16] Zhang Q, Yuan Q, Li J, et al.Thick cloud and cloud shadow removal in multitemporal imagery using progressively spatio-temporal patch group deep learning[J].ISPRS Journal of Photogrammetry and Remote Sensing, 2020, 162:148-160.DOI:10.1016/j.isprsjprs.2020.02.008.
[17] Gao J H, Yi Y, Wei T, et al.Sentinel-2 cloud removal considering ground changes by fusing multitemporal SAR and optical images[J]. Remote Sensing, 2021, 13(19):3998.DOI:10.3390/rs13193998.
[18] Chen H, Yokoya N, Wu C, et al.Unsupervised multimodal change detection based on structural relationship graph representation learning[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 1-18, 5635318. DOI: 10.1109/TGRS.2022.3229027.
[19] Wang Y X, Zhang B, Zhang W J, et al.Cloud removal with SAR-optical data fusion using a unified spatial-spectral residual network[J]. IEEE Transactions on Geoscience and Remote Sensing, 2024, 62: 5600820. DOI:10.1109/TGRS.2023.3339210.
[20] Zhang S, Li X D, Zhou X Y, et al.Cloud removal using SAR and optical images via attention mechanism-based GAN[J]. Pattern Recognition Letters, 2023, 175(C): 8-15. DOI:10.1016/j.patrec.2023.09.014.
[21] Wang P, Chen Y K, Huang B, et al.MT_GAN: A SAR-to-optical image translation method for cloud removal[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2025, 225: 180-195. DOI:10.1016/j.isprsjprs.2025.04.011.
[22] Woo S, Park J, Lee J Y, et al.CBAM: Convolutional block attention module[C]//Computer Vision - ECCV 2018. Cham: Springer, 2018: 3-19. DOI:10.1007/978-3-030-01234-2_1.
[23] Ebel P, Meraner A, Schmitt M, et al.Multisensor data fusion for cloud removal in global and all-season sentinel-2 imagery[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59(7): 5866-5878. DOI:10.1109/TGRS.2020.3024744.
[24] Li J J, Zhang J C, Yang C, et al.Comparative analysis of pixel-level fusion algorithms and a new high-resolution dataset for SAR and optical image fusion[J]. Remote Sensing, 2023, 15(23): 5514. DOI:10.3390/rs15235514.
