Removing highlights from single image via an attention-auxiliary generative adversarial network

doi:10.7523/j.ucas.2020.0018

Abstract

Abstract: The highlights in the image will degrade the image quality to some extent. In this paper, we focus on visually removing the highlights from degraded images and generating clean images. In order to solve this problem, we present an attention-auxiliary generative adversarial networks. It mainly consists of the convolutional long short term memory network with squeeze-and-excitation (SE) block and the map-auxiliary module. Map-auxiliary can instruct the autoencoder to generate clean images. The injection of SE block and map-auxiliary module to the generator is the main contribution of this paper. And our proposed deep learning-based approach can be easily ported to handle other similar image recovery problems. Experiments prove that the network architecture is effective and makes a lot of sense.

Key words: GAN (generative adversarial networks), attention map-auxiliary, squeeze-andexcitation block, image restoration, highlights-removal

CLC Number:

TP391.4

ZHAO Xinchi, JIANG Ce, HE Wei. Removing highlights from single image via an attention-auxiliary generative adversarial network[J]. Journal of University of Chinese Academy of Sciences, 2022, 39(4): 524-531.

References

[1] Tan P, Lin S, Quan L, et al. Highlight removal by illumination-constrained inpainting//Proceedings of the 9th IEEE International Conference on Computer Vision. October 13-16, 2003, Nice, France. IEEE, 2003:164-169. DOI:10.1109/ICCV.2003.1238333.
[2] Yang Q X, Wang S N, Ahuja N. Real-time specular highlight removal using bilateral filtering//European Conference on Computer Vision-ECCV 2010. Springer, Berlin, Heidelberg, 2010:87-100. DOI:10.1007/978-3-642-15561-1_7.
[3] Kurihata H, Takahashi T, Ide I, et al. Rainy weather recognition from in-vehicle camera images for driver assistance//IEEE Proceedings of Intelligent Vehicles Symposium, June 6-8, 2005, Las Vegas, NV, USA. IEEE, 2005:205-210. DOI:10.1109/IVS.2005.1505103.
[4] Roser M, Geiger A. Video-based raindrop detection for improved image registration//2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops. September 27-October 4, 2009, Kyoto, Japan. IEEE, 2009:570-577. DOI:10.1109/ICCVW.2009.5457650.
[5] Roser M, Kurz J, Geiger A. Realistic modeling of water droplets for monocular adherent raindrop recognition using bézier curves//Asian Conference on Computer Vision-ACCV 2010. Springer, Berlin, Heidelberg, 2010:235-244. DOI:10.1007/978-3-642-22819-3_24.
[6] Tanaka Y, Yamashita A, Kaneko T, et al. Removal of adherent waterdrops from images acquired with a stereo camera system[J]. IEICE Transactions on Information and Systems, 2006, E89-D (7):2021-2027. DOI:10.1093/ietisy/e89-d.7.2021.
[7] Yamashita A, Fukuchi I, Kaneko T. Noises removal from image sequences acquired with moving camera by estimating camera motion from spatio-temporal information//2009 IEEE/RSJ International Conference on Intelligent Robots and Systems. October 10-15, 2009, St. Louis, MO, USA. IEEE, 2009:3794-3801. DOI:10.1109/IROS.2009.5354639.
[8] You S D, Tan R T, Kawakami R, et al. Adherent raindrop modeling, detection and removal in video[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 38(9):1721-1733. DOI:10.1109/TPAMI.2015.2491937.
[9] Hara T, Saito H, Kanade T. Removal of glare caused by water droplets//2009 Conference for Visual Media Production. November 12-13, 2009, London, UK. IEEE, 2009:144-151. DOI:10.1109/CVMP.2009.17.
[10] Qian R, Tan R T, Yang W H, et al. Attentive generative adversarial network for raindrop removal from a single image//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. June 18-23, 2018, Salt Lake City, UT, USA. 2018:2482-2491. DOI:10.1109/CVPR.2018.00263.
[11] Goodfellow I, Pouget-Abadie J, Mirza M, et al. Generative adversarial nets//Advances in Neural Information Processing Systems, 2014:2672-2680.
[12] Hu J, Shen L, Sun G. Squeeze-and-excitation networks//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. June 18-23, 2018, Salt Lake City, UT, USA. 2018:7132-7141. DOI:10.1109/CVPR.2018.00745.
[13] He K M, Zhang X Y, Ren S Q, et al. Deep residual learning for image recognition//2016 IEEE Conference on Computer Vision and Pattern Recognition. June 27-30, 2016, Las Vegas, NV, USA. 2016:770-778. DOI:10.1109/CVPR.2016.90.
[14] Shi X J, Chen Z R, Wang H, et al. Convolutional LSTM network:a machine learning approach for precipitation nowcasting//Advances in Neural Information Processing Systems, 2015:802-810.
[15] Nair V, Hinton G E. Rectified linear units improve restricted boltzmann machines//Proceedings of the 27th International Conference on Machine Learning (ICML-10), 2010:807-814.
[16] Hochreiter S, Schmidhuber J. Long short-term memory[J]. Neural Computation, 1997, 9(8):1735-1780. DOI:10.1162/neco.1997.9.8.1735.
[17] Johnson J, Alahi A, Li F F. Perceptual losses for real-time style transfer and super-resolution//European Conference on Computer Vision-ECCV 2016. Springer, Cham, 2016:694-711. DOI:10.1007/978-3-319-46475-6_43.
[18] Zhu J Y, Park T, Isola P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks//2017 IEEE International Conference on Computer Vision. October 22-29, 2017, Venice, Italy. IEEE, 2017:2242-2251. DOI:10.1109/ICCV.2017.244.