This paper proposes a multi-spectral remote sensing image sharpening method based on a deep convolutional neural network and residual network. The method addresses the problems of spectral distortion in traditional remote sensing image sharpening methods and insufficient information utilization between network layers in current deep learning-based methods. The proposed method uses the depth convolution and residual network to design the depth residual module to extract the spatial and spectral features of the deep image. Additionally, residual connections between sub-blocks are established to transmit gradient information to deeper networks and avoid gradient explosion problems, making the network more efficient. Experiments are conducted on simulated and real-world multi-spectral images from WorldView-2, and the results are compared with traditional and existing deep learning-based methods. The proposed method improves the spectral distortion phenomenon and learns deeper image features to better preserve the spatial and spectral information of the image. The proposed method outperforms the deep convolutional sharpening network method in terms of various evaluation metrics, including ERGAS, SAM, SCC, UIQI, and the global fusion quality evaluation index. The proposed method improves these metrics by 24.4%, 26.7%,6.2%,4.7%, and 6.3%, respectively. Subjective and objective evaluations and spectral curve also indicate that the proposed method significantly improves the spatial and spectral resolution of remote sensing images, especially under complex environmental conditions.
ZHOU Qingze
,
GUO Qing
. Multispectral remote sensing image pan-sharpening method based on multi-residual network[J]. Journal of University of Chinese Academy of Sciences, 2025
, 42(4)
: 565
-575
.
DOI: 10.7523/j.ucas.2023.086
[1] 孙洪泉, 窦闻, 易文斌. 遥感图像融合的研究现状、困境及发展趋势探讨[J]. 遥感信息, 2011, 26(1): 104-108. DOI: 10.3969/j.issn.1000-3177.2011.01.021.
[2] Dadrass Javan F, Samadzadegan F, Mehravar S, et al. A review of image fusion techniques for pan-sharpening of high-resolution satellite imagery[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2021, 171: 101-117. DOI: 10.1016/j.isprsjprs.2020.11.001.
[3] 王华伟. 基于IHS变换的多光谱和全色图像融合算法研究[D]. 成都: 西南交通大学, 2016.
[4] Ranchin T, Wald L. Fusion of high spatial and spectral resolution images: the ARSIS concept and its implementation[J]. Photogrammetric Engineering and Remote Sensing, 2000, 66: 49-61.
[5] Aharon M, Elad M, Bruckstein A. K-SVD: an algorithm for designing overcomplete dictionaries for sparse representation[J]. IEEE Transactions on Signal Processing, 2006, 54(11): 4311-4322. DOI: 10.1109/TSP.2006.881199.
[6] Nasr A, Darwis A，Shaheen S. Use of intensity-hue-saturation (IHS) transformations in change detection of multitemporal remotely sensed data[C]//Conference on Image and Signal Processing for Remote Sensing Ⅶ，Sep 18-21, 2001, Toulouse, France.2001.DOI:10.1117/12.454166.
[7] Laben C A, Brower B V. Process for enhancing the spatial resolution of multispectral imagery using pan-sharpening: US6011875[P]. 2000-01-04.
[8] Tu T M, Su S C, Shyu H C, et al. A new look at IHS-like image fusion methods[J]. Information Fusion, 2001, 2(3): 177-186. DOI: 10.1016/S1566-2535(01)00036-7.
[9] Abdolahpoor A, Kabiri P. New texture-based pansharpening method using wavelet packet transform and PCA[J]. International Journal of Wavelets, Multiresolution and Information Processing, 2020, 18(4): 2050025. DOI: 10.1142/s0219691320500253.
[10] Vivone G. Robust band-dependent spatial-detail approaches for panchromatic sharpening[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(9): 6421-6433. DOI: 10.1109/TGRS.2019.2906073.
[11] Aiazzi B, Baronti S, Selva M. Improving component substitution pansharpening through multivariate regression of MS+Pan data[J]. IEEE Transactions on Geoscience and Remote Sensing, 2007, 45(10): 3230-3239. DOI: 10.1109/TGRS.2007.901007.
[12] Choi J, Yu K, Kim Y. A new adaptive component-substitution-based satellite image fusion by using partial replacement[J]. IEEE Transactions on Geoscience and Remote Sensing, 2011, 49(1): 295-309. DOI: 10.1109/TGRS.2010.2051674.
[13] Vivone G, Alparone L, Chanussot J, et al. A critical comparison among pansharpening algorithms[J]. IEEE Transactions on Geoscience and Remote Sensing,2015,53(5): 2565-2586.DOI: 10.1109/TGRS.2014.2361734.
[14] Kaplan N H, Erer I. Bilateral filtering-based enhanced pansharpening of multispectral satellite images[J]. IEEE Geoscience and Remote Sensing Letters, 2014, 11(11): 1941-1945. DOI: 10.1109/LGRS.2014.2314389.
[15] Liu J M, Liang S L. Pan-sharpening using a guided filter[J]. International Journal of Remote Sensing, 2016, 37(8): 1777-1800. DOI: 10.1080/01431161.2016.1163749.
[16] Fang F M, Li F, Shen C M, et al. A variational approach for pan-sharpening[J]. IEEE Transactions on Image Processing, 2013, 22(7): 2822-2834. DOI: 10.1109/TIP.2013.2258355.
[17] Azarang A, Ghassemian H. A new pansharpening method using multi resolution analysis framework and deep neural networks[C]//The 3rd International Conference on Pattern Recognition and Image Analysis (IPRIA). April 19-20, 2017, Shahrekord, Iran. IEEE, 2017: 1-6. DOI: 10.1109/PRIA.2017.7983017.
[18] Zhong J Y, Yang B, Huang G Y, et al. Remote sensing image fusion with convolutional neural network[J]. Sensing and Imaging, 2016, 17(1): 10. DOI: 10.1007/s11220-016-0135-6.
[19] Masi G, Cozzolino D, Verdoliva L, et al. Pansharpening by convolutional neural networks[J]. Remote Sensing, 2016, 8(7): 594. DOI: 10.3390/rs8070594.
[20] Rao Y Z, He L, Zhu J W. A residual convolutional neural network for pan-shaprening[C]//2017 International Workshop on Remote Sensing with Intelligent Processing (RSIP). May 18-21, 2017, Shanghai, China. IEEE, 2017: 1-4. DOI: 10.1109/RSIP.2017.7958807.
[21] Wei Y C, Yuan Q Q, Shen H F, et al. Boosting the accuracy of multispectral image pansharpening by learning a deep residual network[J]. IEEE Geoscience and Remote Sensing Letters, 2017, 14(10): 1795-1799. DOI: 10.1109/LGRS.2017.2736020.
[22] Yang J F, Fu X Y, Hu Y W, et al. PanNet: a deep network architecture for pan-sharpening[C]//2017 IEEE International Conference on Computer Vision (ICCV). October 22-29, 2017, Venice, Italy. IEEE, 2017: 1753-1761. DOI: 10.1109/ICCV.2017.193.
[23] Liu X Y, Wang Y H, Liu Q J. Remote sensing image fusion based on two-stream fusion network[C]//International Conference on Multimedia Modeling. Cham: Springer, 2018: 428-439.10.1007/978-3-319-73603-7_35.
[24] He K M, Sun J. Convolutional neural networks at constrained time cost[C]//2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). June 7-12, 2015, Boston, MA, USA. IEEE, 2015: 5353-5360. DOI: 10.1109/CVPR.2015.7299173.
[25] Stephan M, Hoffman Matthew D, Blei David M. Stochastic gradient descent as approximate Bayesian inference[J]. Journal of Machine Learning Research, 2017, 18: 4873-4907.
[26] Wald L, Ranchin T, Mangolini M. Fusion of satellite images of different spatial resolutions: assessing the quality of resulting images[J]. Photogrammetric Engineering and Remote Sensing, 1997, 63(6): 691-699.
