Analysis of the effect of patch size on super-resolution of large scene remote sensing images

doi:10.7523/j.ucas.2024.019

Abstract

Abstract: Single image Super-resolution (SISR) can improve the resolution of remote sensing images (RSIs), thereby improving the application value of data. At present, the number of pixels of RSIs generally reaches hundreds of millions, and it is usually necessary to divide the image into patches when performing SISR. However, there is a lack of relevant research on how to effectively determine the patch size and whether different sizes affect the results. In this paper, taking a large-scale high-resolution RSIs as the experiment data, 3 typical SISR models are selected, 9 groups of SR experiments under different patch sizes are carried out, and the super resolution (SR) results for the whole of the large-scaled RSI are analyzed comprehensively both qualitatively and quantitatively. The results show that: (1) Cutting of the patches results in stitching seams at the stitching place. In particular, when the patch size is small, a large number of stitching seams show a block effect and the inconsistency is more obvious. (2) With the increase of the patch size, the SR accuracy of the three models is improved, and the overall computational efficiency is also improved. When the test patches are larger than the training patches, the elapsed time and accuracy stabilize. (3) The feasibility and accuracy of the whole RSI input are closely related to the model. The ESPCN model has the best accuracy when inputting the whole RSI, the RDBPN model may cause the accuracy to decrease due to the non-square matrix of the RSI, and the HSENET model has high requirements for computing power and cannot calculate the whole RSI. In conclusion, this paper provides an experimental basis for the selection of patch size for RSI SR engineering applications.

Key words: deep learning super-resolution, large scene remote sensing images, spatial resolution, input size, patch size

CLC Number:

TP751

SUN Ruiqi, ZHANG Wenjuan, LI Zhen, MA Xuesong, MEI Junlin. Analysis of the effect of patch size on super-resolution of large scene remote sensing images[J]. Journal of University of Chinese Academy of Sciences, DOI: 10.7523/j.ucas.2024.019.

References

[1] 张兵. 当代遥感科技发展的现状与未来展望[J].中国科学院院刊,2017,32(7):774-784.DOI:10.16418/j.issn.1000-3045.2017.07.012.
[2] 黄昕, 张良培, 李平湘. 高空间分辨率遥感图像分类的SSMC方法[J]. 中国图象图形学报, 2006, 11(4): 529-534, 插4. DOI: 10.3969/j.issn.1006-8961.2006.04.014.
[3] Martins V S, Kaleita A L, Gelder B K, et al.Exploring multiscale object-based convolutional neural network (multi-OCNN) for remote sensing image classification at high spatial resolution[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2020, 168: 56-73. DOI: 10.1016/j.isprsjprs.2020.08.004.
[4] 张鑫龙, 陈秀万, 李飞, 等. 高分辨率遥感影像的深度学习变化检测方法[J]. 测绘学报, 2017, 46(8): 999-1008. DOI: 10.11947/j.AGCS.2017.20170036.
[5] Fytsilis A L, Prokos A, Koutroumbas K D, et al.A methodology for near real-time change detection between Unmanned Aerial Vehicle and wide area satellite images[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2016, 119: 165-186. DOI: 10.1016/j.isprsjprs.2016.06.001.
[6] 林祥国, 张继贤. 面向对象的形态学建筑物指数及其高分辨率遥感影像建筑物提取应用[J]. 测绘学报, 2017, 46(6): 724-733. DOI: 10.11947/j.AGCS.2017.20170068.
[7] Sun X, Wang P J, Yan Z Y, et al.FAIR1M: a benchmark dataset for fine-grained object recognition in high-resolution remote sensing imagery[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2022, 184: 116-130. DOI: 10.1016/j.isprsjprs.2021.12.004.
[8] 郑凯旋, 林兴稳, 闻建光, 等. 集合卡尔曼滤波方法的高时空分辨率山区地表反照率反演[J]. 遥感学报, 2022, 26(12): 2568-2581. DOI: 10.11834/jrs.20210322.
[9] Huang L, Han X Y, Wang X L, et al.Coupling with high-resolution remote sensing data to evaluate urban non-point source pollution in Tongzhou, China[J]. The Science of the Total Environment, 2022, 831: 154632. DOI: 10.1016/j.scitotenv.2022.154632.
[10] 左超, 陈钱. 计算光学成像: 何来, 何处, 何去, 何从?[J]. 红外与激光工程, 2022, 51(2): 150-330. DOI: 10.3788/IRLA20220110.
[11] 左超, 陈钱. 分辨率, 超分辨率与空间带宽积拓展—从计算光学成像角度的一些思考[J]. 中国光学(中英文), 2022(6): 1105-1166. DOI: 10.37188/CO.2022-0105.
[12] 郑珂. 基于深度学习的高光谱图像超分辨率重建与分类方法研究[D]. 北京: 中国矿业大学 (北京), 2020. DOI: 10.27624/d.cnki.gzkbu.2020.000027.
[13] 李佳星, 赵勇先, 王京华. 基于深度学习的单幅图像超分辨率重建算法综述[J]. 自动化学报, 2021, 47(10): 2341-2363. DOI: 10.16383/j.aas.c190859.
[14] Keys R.Cubic convolution interpolation for digital image processing[J]. IEEE Transactions on Acoustics, Speech, and Signal Processing, 1981, 29(6): 1153-1160. DOI: 10.1109/TASSP.1981.1163711.
[15] Wang Q M, Shi W Z, Atkinson P M.Sub-pixel mapping of remote sensing images based on radial basis function interpolation[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2014, 92: 1-15. DOI: 10.1016/j.isprsjprs.2014.02.012.
[16] Ma J L, Chan J C W, Canters F. Robust locally weighted regression for superresolution enhancement of multi-angle remote sensing imagery[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2014, 7(4): 1357-1371. DOI: 10.1109/JSTARS.2014.2312887.
[17] Wang P, Yao H Y, Li C, et al.Multiresolution analysis based on dual-scale regression for pansharpening[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 60: 5406319. DOI: 10.1109/tgrs.2021.3131477.
[18] Pan Z X, Yu J, Huang H J, et al.Super-resolution based on compressive sensing and structural self-similarity for remote sensing images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2013, 51(9): 4864-4876. DOI: 10.1109/TGRS.2012.2230270.
[19] Hou B, Zhou K, Jiao L C.Adaptive super-resolution for remote sensing images based on sparse representation with global joint dictionary model[J]. IEEE Transactions on Geoscience and Remote Sensing, 2018, 56(4): 2312-2327. DOI: 10.1109/TGRS.2017.2778191.
[20] Jiang K, Wang Z Y, Yi P, et al.Edge-enhanced GAN for remote sensing image superresolution[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(8): 5799-5812. DOI: 10.1109/TGRS.2019.2902431.
[21] Xu M Z, Ma J, Zhu Y Y.Dual-Diffusion: dual conditional denoising diffusion probabilistic models for blind super-resolution reconstruction in RSIs[J]. IEEE Geoscience and Remote Sensing Letters, 2023, 20: 6008505. DOI: 10.1109/LGRS.2023.3304418.
[22] Dong C, Loy C C, He K M, et al.Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(2): 295-307. DOI: 10.1109/TPAMI.2015.2439281.
[23] Shi W Z, Caballero J, Huszár F, et al.Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV, USA. IEEE, 2016: 1874-1883. DOI: 10.1109/CVPR.2016.207.
[24] Pan Z X, Ma W, Guo J Y, et al.Super-resolution of single remote sensing image based on residual dense backprojection networks[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(10): 7918-7933. DOI: 10.1109/TGRS.2019.2917427.
[25] Lei S, Shi Z W.Hybrid-scale self-similarity exploitation for remote sensing image super-resolution[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 60: 5401410. DOI: 10.1109/TGRS.2021.3069889.
[26] Ji S P, Wei S Q, Lu M.Fully convolutional networks for multisource building extraction from an open aerial and satellite imagery data set[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(1): 574-586. DOI: 10.1109/TGRS.2018.2858817.
[27] Odena A, Dumoulin V, Olah C.Deconvolution and checkerboard artifacts[J]. Distill, 2016, 1(10): e3. DOI: 10.23915/distill.00003.