针对传统的遥感图像锐化方法通常会导致锐化图像光谱失真的问题与目前基于深度学习的锐化方法利用网络层之间信息不充分的问题，结合深度卷积神经网络和残差网络的特性，提出一种基于多重残差网络的多光谱遥感图像锐化方法。该方法利用深度卷积网络和残差网络，设计深度残差模块，通过堆叠深度残差模块提取图像深层次的空间和光谱特征，同时利用残差建立子块与子块之间的跳跃连接，将梯度信息传递到更深的网络，避免梯度爆炸问题，使网络更加高效。实验基于WorldView-2的多光谱图像和全色图像进行模拟实验与真实实验，将实验结果与传统方法和现有深度学习方法进行比较。结果表明，该方法改善了传统方法存在的光谱失真现象；相较于现有的深度学习方法，能够学习到更深层次的图像特征，更好地保留图像的空间与光谱信息；全局相对光谱损失、光谱角映射、空间相关系数、整体质量评价指标和全局融合质量评价指标分别比深度卷积锐化网络方法提高24.4%、26.7%、6.2%、4.7%和6.3%。主观视觉评价、客观定量评价和光谱曲线表明，相比于传统的锐化算法以及常用的深度学习锐化算法，该方法在空间分辨率和光谱分辨率上都有显著提升，特别是对于复杂地物环境条件下的遥感图像。
This paper proposes a multi-spectral remote sensing image sharpening method based on a deep convolutional neural network and residual network. The method addresses the problems of spectral distortion in traditional remote sensing image sharpening methods and insufficient information utilization between network layers in current deep learning-based methods. The proposed method uses the depth convolution and residual network to design the depth residual module to extract the spatial and spectral features of the deep image. Additionally, residual connections between sub-blocks are established to transmit gradient information to deeper networks and avoid gradient explosion problems, making the network more efficient. Experiments are conducted on simulated and real-world multi-spectral images from WorldView-2, and the results are compared with traditional and existing deep learning-based methods. The proposed method improves the spectral distortion phenomenon and learns deeper image features to better preserve the spatial and spectral information of the image. The proposed method outperforms the deep convolutional sharpening network method in terms of various evaluation metrics, including ERGAS, SAM, SCC, UIQI, and the global fusion quality evaluation index. The proposed method improves these metrics by 24.4%, 26.7%,6.2%,4.7%, and 6.3%, respectively. Subjective and objective evaluations and spectral curve also indicate that the proposed method significantly improves the spatial and spectral resolution of remote sensing images, especially under complex environmental conditions.
[1] 孙洪泉, 窦闻, 易文斌. 遥感图像融合的研究现状、困境及发展趋势探讨[J]. 遥感信息, 2011, 26(1): 104-108. DOI: 10.3969/j.issn.1000-3177.2011.01.021.
[2] Dadrass Javan F, Samadzadegan F, Mehravar S, et al. A review of image fusion techniques for pan-sharpening of high-resolution satellite imagery[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2021, 171: 101-117. DOI: 10.1016/j.isprsjprs.2020.11.001.
[3] 王华伟. 基于IHS变换的多光谱和全色图像融合算法研究[D]. 成都: 西南交通大学, 2016.
[4] Ranchin T, Wald L. Fusion of high spatial and spectral resolution images: the ARSIS concept and its implementation[J]. Photogrammetric Engineering and Remote Sensing, 2000, 66: 49-61.
[5] Aharon M, Elad M, Bruckstein A. K-SVD: an algorithm for designing overcomplete dictionaries for sparse representation[J]. IEEE Transactions on Signal Processing, 2006, 54(11): 4311-4322. DOI: 10.1109/TSP.2006.881199.
[6] Nasr A, Darwis A，Shaheen S. Use of intensity-hue-saturation (IHS) transformations in change detection of multitemporal remotely sensed data[C]//Conference on Image and Signal Processing for Remote Sensing Ⅶ，Sep 18-21, 2001, Toulouse, France.2001.DOI:10.1117/12.454166.
[7] Laben C A, Brower B V. Process for enhancing the spatial resolution of multispectral imagery using pan-sharpening: US6011875[P]. 2000-01-04.
[8] Tu T M, Su S C, Shyu H C, et al. A new look at IHS-like image fusion methods[J]. Information Fusion, 2001, 2(3): 177-186. DOI: 10.1016/S1566-2535(01)00036-7.
[9] Abdolahpoor A, Kabiri P. New texture-based pansharpening method using wavelet packet transform and PCA[J]. International Journal of Wavelets, Multiresolution and Information Processing, 2020, 18(4): 2050025. DOI: 10.1142/s0219691320500253.
[10] Vivone G. Robust band-dependent spatial-detail approaches for panchromatic sharpening[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(9): 6421-6433. DOI: 10.1109/TGRS.2019.2906073.
[11] Aiazzi B, Baronti S, Selva M. Improving component substitution pansharpening through multivariate regression of MS+Pan data[J]. IEEE Transactions on Geoscience and Remote Sensing, 2007, 45(10): 3230-3239. DOI: 10.1109/TGRS.2007.901007.
[12] Choi J, Yu K, Kim Y. A new adaptive component-substitution-based satellite image fusion by using partial replacement[J]. IEEE Transactions on Geoscience and Remote Sensing, 2011, 49(1): 295-309. DOI: 10.1109/TGRS.2010.2051674.
[13] Vivone G, Alparone L, Chanussot J, et al. A critical comparison among pansharpening algorithms[J]. IEEE Transactions on Geoscience and Remote Sensing,2015,53(5): 2565-2586.DOI: 10.1109/TGRS.2014.2361734.
[14] Kaplan N H, Erer I. Bilateral filtering-based enhanced pansharpening of multispectral satellite images[J]. IEEE Geoscience and Remote Sensing Letters, 2014, 11(11): 1941-1945. DOI: 10.1109/LGRS.2014.2314389.
[15] Liu J M, Liang S L. Pan-sharpening using a guided filter[J]. International Journal of Remote Sensing, 2016, 37(8): 1777-1800. DOI: 10.1080/01431161.2016.1163749.
[16] Fang F M, Li F, Shen C M, et al. A variational approach for pan-sharpening[J]. IEEE Transactions on Image Processing, 2013, 22(7): 2822-2834. DOI: 10.1109/TIP.2013.2258355.
[17] Azarang A, Ghassemian H. A new pansharpening method using multi resolution analysis framework and deep neural networks[C]//The 3rd International Conference on Pattern Recognition and Image Analysis (IPRIA). April 19-20, 2017, Shahrekord, Iran. IEEE, 2017: 1-6. DOI: 10.1109/PRIA.2017.7983017.
[18] Zhong J Y, Yang B, Huang G Y, et al. Remote sensing image fusion with convolutional neural network[J]. Sensing and Imaging, 2016, 17(1): 10. DOI: 10.1007/s11220-016-0135-6.
[19] Masi G, Cozzolino D, Verdoliva L, et al. Pansharpening by convolutional neural networks[J]. Remote Sensing, 2016, 8(7): 594. DOI: 10.3390/rs8070594.
[20] Rao Y Z, He L, Zhu J W. A residual convolutional neural network for pan-shaprening[C]//2017 International Workshop on Remote Sensing with Intelligent Processing (RSIP). May 18-21, 2017, Shanghai, China. IEEE, 2017: 1-4. DOI: 10.1109/RSIP.2017.7958807.
[21] Wei Y C, Yuan Q Q, Shen H F, et al. Boosting the accuracy of multispectral image pansharpening by learning a deep residual network[J]. IEEE Geoscience and Remote Sensing Letters, 2017, 14(10): 1795-1799. DOI: 10.1109/LGRS.2017.2736020.
[22] Yang J F, Fu X Y, Hu Y W, et al. PanNet: a deep network architecture for pan-sharpening[C]//2017 IEEE International Conference on Computer Vision (ICCV). October 22-29, 2017, Venice, Italy. IEEE, 2017: 1753-1761. DOI: 10.1109/ICCV.2017.193.
[23] Liu X Y, Wang Y H, Liu Q J. Remote sensing image fusion based on two-stream fusion network[C]//International Conference on Multimedia Modeling. Cham: Springer, 2018: 428-439.10.1007/978-3-319-73603-7_35.
[24] He K M, Sun J. Convolutional neural networks at constrained time cost[C]//2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). June 7-12, 2015, Boston, MA, USA. IEEE, 2015: 5353-5360. DOI: 10.1109/CVPR.2015.7299173.
[25] Stephan M, Hoffman Matthew D, Blei David M. Stochastic gradient descent as approximate Bayesian inference[J]. Journal of Machine Learning Research, 2017, 18: 4873-4907.
[26] Wald L, Ranchin T, Mangolini M. Fusion of satellite images of different spatial resolutions: assessing the quality of resulting images[J]. Photogrammetric Engineering and Remote Sensing, 1997, 63(6): 691-699.
