基于MFF-SFE的遥感图文跨模态检索方法

doi:10.7523/j.ucas.2024.025

中国科学院大学学报 ›› 2025, Vol. 42 ›› Issue (2): 236-247.DOI: 10.7523/j.ucas.2024.025

• 电子信息与计算机科学 • 上一篇

基于MFF-SFE的遥感图文跨模态检索方法

钟金彦^1,2, 陈俊^1,3,4, 李宇¹, 吴业炜¹, 葛小青¹

1. 中国科学院空天信息创新研究院, 北京 100094;
2. 中国科学院大学电子电气与通信工程学院, 北京 100049;
3. 中国科学院计算机网络信息中心, 北京 100083;
4. 中国科学院大学计算机科学与技术学院, 北京 100049

收稿日期:2024-01-18 修回日期:2024-04-17 发布日期:2024-05-22
通讯作者: 李宇,E-mail:liyu202615@aircas.ac.cn
基金资助:
中国科学院青年促进会(E0331804)资助

Cross-modal retrieval method based on MFF-SFE for remote sensing image-text

ZHONG Jinyan^1,2, CHEN Jun^1,3,4, LI Yu¹, WU Yewei¹, GE Xiaoqing¹

1. Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, China;
2. School of Electronic, Electrical and Communication Engineering, University of Chinese Academy of Sciences, Beijing 100049, China;
3. Computer Network Information Center, Chinese Academy of Sciences, Beijing 100083, China;
4. School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing 100049, China

Received:2024-01-18 Revised:2024-04-17 Published:2024-05-22

摘要/Abstract

摘要： 遥感图文跨模态检索技术能够从海量的遥感数据中快速获取有价值的信息，但现有遥感图文检索方法对遥感图像中的多尺度信息利用不足、目标信息识别效果不佳，检索精度相对较低。为此，提出一种新的遥感图文跨模态检索方法。该方法主要包括一个多尺度特征融合模块和一个显著特征增强模块，分别用于融合遥感图像的多尺度信息、加强对遥感图像目标信息的表达能力，从而提高遥感图文跨模态检索精度。在2个公开的遥感图像文本数据集上进行实验验证，结果表明，在遥感图文跨模态检索任务中，该方法在大部分评价指标上都优于其他方法，具有最佳的总体检索性能。

关键词: 跨模态检索, 遥感图像, 深度学习, 多尺度特征

Abstract: Remote sensing image-text cross-modal retrieval technology can quickly obtain valuable information from massive remote sensing data. However, existing remote sensing image-text retrieval methods have limitations in utilizing multi-scale information within remote sensing images, and the weak recognition of target information leads to relatively low retrieval accuracy. To address these issues, this paper proposes a new method for remote sensing image-text cross-modal retrieval. This method mainly comprises a multi-scale feature fusion module and a salient feature enhancement module, which are designed to integrate multi-scale information of remote sensing images and enhance the expression of target information in remote sensing images, so as to improve the precision of remote sensing image-text cross-modal retrieval. Experimental validation was conducted on two publicly available remote sensing image-text datasets. The results demonstrate that the proposed method outperforms other methods across most evaluation metrics in the remote sensing image-text cross-modal retrieval task and exhibits the best overall retrieval performance.

Key words: cross-modal retrieval, remote sensing images, deep learning, multi-scale feature

中图分类号:

TP751.1

钟金彦, 陈俊, 李宇, 吴业炜, 葛小青. 基于MFF-SFE的遥感图文跨模态检索方法[J]. 中国科学院大学学报, 2025, 42(2): 236-247.

ZHONG Jinyan, CHEN Jun, LI Yu, WU Yewei, GE Xiaoqing. Cross-modal retrieval method based on MFF-SFE for remote sensing image-text[J]. Journal of University of Chinese Academy of Sciences, 2025, 42(2): 236-247.

参考文献

[1] Chi M M, Plaza A, Benediktsson J A, et al. Big data for remote sensing: challenges and opportunities[J]. Proceedings of the IEEE, 2016, 104(11): 2207-2219. DOI: 10.1109/JPROC.2016.2598228.
[2] Kaur P, Pannu H S, Malhi A K. Comparative analysis on cross-modal information retrieval: a review[J]. Computer Science Review, 2021, 39: 100336. DOI: 10.1016/j.cosrev.2020.100336.
[3] Chen C, Zou H X, Shao N Y, et al. Deep semantic hashing retrieval of remotec sensing images[C]//IGARSS 2018-2018 IEEE International Geoscience and Remote Sensing Symposium. Valencia, Spain. IEEE, 2018: 1124-1127. DOI: 10.1109/IGARSS.2018.8519276.
[4] Ye F M, Luo W, Dong M, et al. SAR image retrieval based on unsupervised domain adaptation and clustering[J]. IEEE Geoscience and Remote Sensing Letters, 2019, 16(9): 1482-1486. DOI: 10.1109/LGRS.2019.2896948.
[5] Guo M, Zhou C H, Liu J H. Jointly learning of visual and auditory: a new approach for RS image and audio cross-modal retrieval[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019, 12(11): 4644-4654. DOI: 10.1109/JSTARS.2019.2949220.
[6] Shi Z W, Zou Z X. Can a machine generate humanlike language descriptions for a remote sensing image?[J]. IEEE Transactions on Geoscience and Remote Sensing, 2017, 55(6): 3623-3634. DOI: 10.1109/TGRS.2017.2677464.
[7] Wang G A, Hu Q H, Cheng J, et al. Semi-supervised generative adversarial hashing for image retrieval[C]//European Conference on Computer Vision. Cham: Springer, 2018: 491-507.10.1007/978-3-030-01267-0_29.
[8] Lu J S, Batra D, Parikh D, et al. ViLBERT: pretraining task-agnostic visiolinguistic representations for vision-and-language tasks[EB/OL]. 2019.arXiv: 1908.02265. http://arxiv.org/abs/1908.02265.
[9] Radford A, Kim J W, Hallacy C, et al. Learning transferable visual models from natural language supervision[EB/OL]. 2021. arXiv: 2103.00020. (2021-02-26)[2024-04-01]. http://arxiv.org/abs/2103.00020.
[10] Abdullah T, Bazi Y, Al Rahhal M M, et al. TextRS: deep bidirectional triplet network for matching text to remote sensing images[J]. Remote Sensing, 2020, 12(3): 405. DOI: 10.3390/rs12030405.
[11] Lv Y F, Xiong W, Zhang X H, et al. Fusion-based correlation learning model for cross-modal remote sensing image retrieval[J]. IEEE Geoscience and Remote Sensing Letters, 2021, 19: 6503205. DOI: 10.1109/LGRS.2021.3131592.
[12] Mikriukov G, Ravanbakhsh M, Demir B. Deep unsupervised contrastive hashing for large-scale cross-modal text-image retrieval in remote sensing[EB/OL]. 2022. arXiv: 2201.08125. http://arxiv.org/abs/2201.08125.
[13] Yuan Z Q, Zhang W K, Tian C Y, et al. Remote sensing cross-modal text-image retrieval based on global and local information[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 5620616. DOI: 10.1109/TGRS.2022.3163706.
[14] Cheng Q M, Zhou Y Z, Fu P, et al. A deep semantic alignment network for the cross-modal image-text retrieval in remote sensing[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2021, 14: 4284-4297. DOI: 10.1109/JSTARS.2021.3070872.
[15] Zheng F Z, Li W P, Wang X, et al. A cross-attention mechanism based on regional-level semantic features of images for cross-modal text-image retrieval in remote sensing[J]. Applied Sciences, 2022, 12(23): 12221. DOI: 10.3390/app122312221.
[16] Tang X, Wang Y J, Ma J J, et al. Interacting-enhancing feature transformer for cross-modal remote-sensing image and text retrieval[J]. IEEE Transactions on Geoscience and Remote Sensing, 2023, 61: 5611715. DOI: 10.1109/TGRS.2023.3280546.
[17] Yuan Z Q, Zhang W K, Fu K, et al. Exploring a fine-grained multiscale method for cross-modal remote sensing image retrieval[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 4404119. DOI: 10.1109/TGRS.2021.3078451.
[18] Wang Y J, Ma J J, Li M T, et al. Multi-scale interactive transformer for remote sensing cross-modal image-text retrieval[C]//IGARSS 2022-2022 IEEE International Geoscience and Remote Sensing Symposium. Kuala Lumpur, Malaysia. IEEE, 2022: 839-842. DOI: 10.1109/IGARSS46834.2022.9883252.
[19] 张若愚, 聂婕, 宋宁, 等. 基于布局化-语义联合表征遥感图文检索方法[J]. 北京航空航天大学学报, 2024, 50(2): 671-683. DOI:10.13700/j.bh.1001-5965.2022.0527.
[20] Szegedy C, Ioffe S, Vanhoucke V, et al. Inception-v4, inception-ResNet and the impact of residual connections on learning[C]//Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence. February 4-9, 2017, San Francisco, California, USA. ACM, 2017: 4278-4284. DOI: 10.5555/3298023.3298188.
[21] Devlin J, Chang M W, Lee K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[EB/OL]. 2018. arXiv: 1810.04805. (2018-10-11)[2024-04-01]. http://arxiv.org/abs/1810.04805.pdf.
[22] Qu B, Li X L, Tao D C, et al. Deep semantic understanding of high resolution remote sensing image[C]//2016 International Conference on Computer, Information and Telecommunication Systems (CITS). Kunming, China. IEEE, 2016: 1-5. DOI: 10.1109/CITS.2016.7546397.
[23] Faghri F, Fleet D J, Kiros J R, et al. VSE++: improving visual-semantic embeddings with hard negatives[EB/OL]. 2017. arXiv: 1707.05612. (2017-07-18)[2024-04-01].http://arxiv.org/abs/1707.05612.
[24] Lee K H, Chen X, Hua G, et al. Stacked cross attention for image-text matching[C]//European Conference on Computer Vision. Cham: Springer, 2018: 212-228.10.1007/978-3-030-01225-0_13.
[25] Ren S Q, He K M, Girshick R, et al. Faster R-CNN: towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6): 1137-1149. DOI: 10.1109/TPAMI.2016.2577031.
[26] Wang Z H, Liu X H, Li H S, et al. CAMP: cross-modal adaptive message passing for text-image retrieval[C]//2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul, Korea (South). IEEE, 2019: 5763-5772. DOI: 10.1109/ICCV.2019.00586.
[27] Wang T, Xu X, Yang Y, et al. Matching images and text with multi-modal tensor fusion and re-ranking[C]//Proceedings of the 27th ACM International Conference on Multimedia. October 21-25, 2019, Nice, France. ACM, 2019: 12-20. DOI: 10.1145/3343031.3350875.

基于MFF-SFE的遥感图文跨模态检索方法

Cross-modal retrieval method based on MFF-SFE for remote sensing image-text

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

访问统计

联系我们

[1]	周文雪, 张华春. 一种面向SAR图像快速舰船检测的轻量化网络[J]. 中国科学院大学学报, 2024, 41(6): 776-785.
[2]	喻永生, 罗铁坚. 基于GAN反演的无缝图像补全技术[J]. 中国科学院大学学报, 2024, 41(5): 705-714.
[3]	陈经纬, 李宇, 陈俊, 张洪群. 基于MFF-Deeplabv3+网络的高分辨率遥感影像建筑物提取方法[J]. 中国科学院大学学报, 2024, 41(5): 654-664.
[4]	黄玉林, 梁磊, 李卫军, 习晓环. 基于多尺度特征和注意力机制的深度学习点云压缩[J]. 中国科学院大学学报, 2024, 41(5): 687-694.
[5]	林昱彤, 王红, 柴团耀. 基于卷积神经网络多尺度特征的大豆基因组表型预测[J]. 中国科学院大学学报, 2024, 41(4): 468-476.
[6]	肖俊, 石光田. 三维点云去噪技术[J]. 中国科学院大学学报, 2023, 40(5): 577-595.
[7]	盛晓光, 王颖, 张迎伟, 项若曦, 付红萍. WILS：面向学业预警的非均衡增量式学习方法[J]. 中国科学院大学学报, 2023, 40(3): 422-432.
[8]	姚沐风, 昝露洋, 李柏鹏, 李庆亭, 陈正超. 基于CAR-Siamese网络的高分辨率遥感图像建筑物变化检测[J]. 中国科学院大学学报, 2023, 40(3): 380-387.
[9]	陈若男, 彭玲, 刘玉菲, 卫志超, 吕蓓茹, 陈德跃. 引入空间距离信息的城郊山区道路提取与应用[J]. 中国科学院大学学报, 2022, 39(5): 658-667.
[10]	古煜民, 阎福礼. 基于不同骨架UNet++网络的建筑物提取[J]. 中国科学院大学学报, 2022, 39(4): 512-523.
[11]	林大权, 范睿, 张良峰. 基于高维特征的图像对抗攻击算法[J]. 中国科学院大学学报, 2022, 39(3): 421-431.
[12]	赵敏钧, 赵亚伟, 赵雅捷, 罗刚. 一种新的基于深度学习的重叠关系联合抽取模型（英文）[J]. 中国科学院大学学报, 2022, 39(2): 240-251.
[13]	席彦新, 计璐艳, 耿修瑞. 一种多时相遥感图像多目标检测算法[J]. 中国科学院大学学报, 2021, 38(4): 503-510.
[14]	周龙, 王伟强, 吕科. FACR：一种快速且准确的车辆识别器[J]. 中国科学院大学学报, 2021, 38(1): 130-136.
[15]	李森, 彭玲, 胡媛, 池天河. 基于FD-RCF的高分辨率遥感影像耕地边缘检测[J]. 中国科学院大学学报, 2020, 37(4): 483-489.