基于原型学习改进的伪标签半监督学习算法

doi:10.7523/j.issn.2095-6134.2021.06.015

摘要/Abstract

摘要： 近年来，基于图像增广和一致性正则化的半监督学习（semi-supervised learning， SSL）方法被广泛应用并取得了很大的成功。然而，由于伪标签算法存在"认知偏误"问题，即模型的错误通过伪标签累积从而难以改正，因此很少有人关注基于伪标签（pseudo-labeling， PL）的半监督学习方法。提出一种特征图的原型图注意力特征修正模型（prototype attention layer， PAL）：即在神经网络映射的特征空间上学习一个图注意力模型，将此模型应用于特征空间中，可以充分利用原型的信息来修正特征，将修正后的特征所产生的伪标签与原型分配产生的伪标签随机线性组合，从而得到新的伪标签。将这一模型应用到2种伪标签半监督学习框架上所得到的算法（prototype attention improved pseudo-labeling，PAIPL），在CIFAR-10和CIFAR-100的多个半监督分类问题上进行测试，分类准确率都得到了显著提升。特别地，将提出的修正模型应用于伪标签半监督学习PLCB框架时，又提出相互混合的监督技术，从而取得了更好的效果。还将提出的模型应用到其他多个伪标签半监督学习框架上，并在多个数据集上进行实验，验证了所提出的模型作为一个附加模块是普适且有效的。

关键词: 半监督学习, 伪标签, MixUp, 图注意力模型, 原型学习

Abstract: In recent years, semi-supervised learning (SSL) methods based on image augmentation and consistency regularization have been widely used and have achieved great success. However, little attention has been paid to pseudo-labeling (PL)-based semi-supervised learning methods because of the "confirmation bias" problem, i.e., errors in the model are accumulated by wrong pseudo-labels and thus difficult to be corrected. In this paper, we propose a feature refinement model based on the feature space graph. The model learns a graph attention model on the feature space mapped by the neural network. We apply this model to the feature space to make use of the information of the prototypes to refine the features. The pseudo-labels generated by the refined features are randomly and linearly combined with the pseudo-labels generated by the prototypes assignment to obtain new pseudo-labels. In this paper, we apply this module to two pseudo-labeling semi-supervised learning frameworks and achieve significant accuracy improvements in several CIFAR-10 and CIFAR-100 semi-supervised classification problems. In particular, we apply our feature refinement model to the pseudo-labeling semi-supervised learning framework PLCB and add the proposed mutual mix supervision techniques to achieve good results on this framework. By applying the proposed feature refinement module to several pseudo-labeling semi-supervised learning frameworks and conducting experiments on several datasets, the proposed algorithm is demonstrated to be universal and effective as an add-on module.

Key words: semi-supervised learning, pseudo-labeling, MixUp, graph attention model, prototype learning

中图分类号:

TP391

杨雨龙, 郭田德, 韩丛英. 基于原型学习改进的伪标签半监督学习算法[J]. 中国科学院大学学报, 2021, 38(6): 841-851.

YANG Yulong, GUO Tiande, HAN Congying. Improving pseudo-labeling semi-supervised learning based on prototype learning[J]. Journal of University of Chinese Academy of Sciences, 2021, 38(6): 841-851.

参考文献

[1] Ouali Y, Hudelot C,Tami M. An overview of deep semi-supervised learning[EB/OL]. arXiv:2006.05278. (2020-07-06)[2021-04-15]. https://arxiv.org/abs/2006.05278.
[2] Oliver A, Odena A, Raffel C, et al. Realistic evaluation of deep semi-supervised learning algorithms[EB/OL]. arXiv:1804.09170. (2019-06-17)[2021-04-15]. https://arxiv.org/abs/1804.09170.
[3] Chen T, Kornblith S, Swersky K, et al. Big self-supervised models are strong semi-supervised learners[EB/OL]. arXiv:2006.10029. (2020-10-26)[2021-04-15]. https://arxiv.org/abs/2006.10029.
[4] Xie Q Z, Luong M T, Hovy E, et al. Self-training with noisy student improves ImageNet classification[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). June 13-19, 2020, Seattle, WA, USA. IEEE, 2020:10684-10695.
[5] Ibrahim M S, Vahdat A, Ranjbar M, et al. Semi-supervised semantic image segmentation with self-correcting networks[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). June 13-19, 2020, Seattle, WA, USA. IEEE, 2020:12712-12722.
[6] Ouali Y, Hudelot C, Tami M. Semi-supervised semantic segmentation with cross-consistency training[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). June 13-19, 2020, Seattle, WA, USA. IEEE, 2020:12671-12681.
[7] He J X, Gu J T, Shen J J, et al. Revisiting self-training for neural sequence generation[EB/OL]. arXiv:1909.13788. (2020-10-18)[2021-04-15]. https://arxiv.org/abs/1909.13788.
[8] Chen L X, Ruan W T, Liu X Y, et al. SeqVAT:virtual adversarial training for semi-supervised sequence labeling[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Online. Stroudsburg, PA, USA:Association for Computational Linguistics, 2020:8801-8811.
[9] Li Y T, Liu L, Tan R T. Decoupled certainty-driven consi-stency loss for semi-supervised learning[EB/OL]. arXiv:1901.05657. (2020-07-31)[2021-04-15]. https://arxiv.org/abs/1901.05657.
[10] Tarvainen A, Valpola H. Mean teachers are better role models:weight-averaged consistency targets improve semi-supervised deep learning results[EB/OL]. arXiv:1703.01780. (2018-04-16)[2021-04-15]. https://arxiv.org/abs/1703.01780.
[11] Arazo E, Ortego D, Albert P, et al. Pseudo-labeling and confirmation bias in deep semi-supervised learning[C]//2020 International Joint Conference on Neural Networks (IJCNN). July 19-24, 2020, Glasgow, UK. IEEE, 2020:1-8.
[12] Zhang H, Cisse M, Dauphin Y N. Mixup:beyond empirical risk minimization[EB/OL]. arXiv:1710.09412. (2018-04-27)[2021-04-15]. https://arxiv.org/abs/1710.09412.
[13] Veličković P, Cucurull G, Casanova A, et al. Graph attention networks[EB/OL]. arXiv:1710.10903. (2018-02-04)[2021-04-15]. https://arxiv.org/abs/1710.10903.
[14] Riloff E, Wiebe J. Learning extraction patterns for subjective expressions[C]//Proceedings of the 2003 Conference on Empirical Methods in Natural Language Processing. July 11-12, 2003, Sapporo, Japan. Association for Computational Linguistics, 2003:105-112.
[15] Laine S, Aila T. Temporal ensembling for semi-supervised learning[EB/OL]. arXiv:1610.02242. (2017-03-15)[2021-04-15]. https://arxiv.org/abs/1610.02242.
[16] Ke Z H, Wang D Y, Yan Q, et al. Dual student:breaking the limits of the teacher in semi-supervised learning[C]//2019 IEEE/CVF International Conference on Computer Vision (ICCV). October 27-November 2, 2019, Seoul, South Korea. IEEE, 2019:6728-6736.
[17] Miyato T, Maeda S I, Koyama M, et al. Virtual adversarial training:a regularization method for supervised and semi-supervised learning[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 41(8):1979-1993.
[18] Verma V, Lamb A, Kannala J, et al. Interpolation consi-stency training for semi-supervised learning[C]//Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence. August 10-16, 2019. Macao, China. California:International Joint Conferences on Artificial Intelligence Organization, 2019:3635-3641.
[19] Berthelot D, Carlini N, Goodfellow I J, et al. MixMatch:a holistic approach to semi-supervised learning[EB/OL]. arXiv:1905.02249. (2019-10-23)[2021-04-15]. https://arxiv.org/abs/1905.02249.
[20] Xie Q Z, Dai Z H, Hovy E, et al. Unsupervised data augmentation for consistency training[EB/OL]. arXiv:1904.12848. (2020-11-5)[2021-04-15]. https://arxiv.org/abs/1904.12848.
[21] Lee D H. Pseudo-label:the simple and efficient semi-super-vised learning method for deep neural networks[EB/OL]. (2013-07)[2021-04-15]. https://www.researchgate.net/publication/280581078_Pseudo-Label_The_Simple_and_Efficient_Semi-Supervised_Learning_Method_for_Deep_Neural_Networks.
[22] Zhuang C X, Zhai A, Yamins D. Local aggregation for unsu-pervised learning of visual embeddings[C]//2019 IEEE/CVF International Conference on Computer Vision (ICCV). October 27-November 2, 2019, Seoul, South Korea. IEEE, 2019:6001-6011.
[23] Kuo C W, Ma C Y, Huang J B, et al. Manifold graph with learned prototypes for semi-supervised image classification[EB/OL]. arXiv:1906.05202. (2019-06-13)[2021-04-15]. https://arxiv.org/abs/1906.05202.
[24] Srivastava N, Hinton G E, Krizhevsky A, et al. Dropout:a simple way to prevent neural networks from overfitting[J]. Journal of Machine Learning Research, 2014, 15:1929-1958.
[25] Salimans T, Kingma D P. Weight normalization:a simple reparameterization to accelerate training of deep neural networks[EB/OL]. arXiv:1602.07868(2016-06-04)[2021-04-15]. https://arxiv.org/abs/1602.07868.
[26] Tanaka D, Ikami D, Yamasaki T, et al. Joint optimization framework for learning with noisy labels[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. June 18-23, 2018, Salt Lake City, UT, USA. IEEE, 2018:5552-5560.
[27] Grandvalet Y, Bengio Y. Semi-supervised learning by entropy minimization[C]//NIPS'04:Proceedings of the 17th International Conference on Neural Information Processing Systems. December 13-18, 2004, Vancouver, British Columbia, Canada. MIT Press, 2004:529-536.
[28] Iscen A, Tolias G, Avrithis Y, et al. Label propagation for deep semi-supervised learning[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). June 15-20, 2019, Long Beach, CA, USA. IEEE, 2019:5065-5074.
[29] Chen T, Kornblith S, Norouzi M, et al. A simple framework for contrastive learning of visual representations[EB/OL]. arXiv:2002.05709. (2020-07-01)[2021-04-15]. https://arxiv.org/abs/2002.05709.
[30] Zhang Y, Xiang T, Hospedales T M, et al. Deep mutual learning[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. June 18-23, 2018, Salt Lake City, UT, USA. IEEE, 2018:4320-4328.
[31] Athiwaratkun B, Finzi M, Izmailov P, et al. There are many consistent explanations of unlabeled data:why you should average[EB/OL]. arXiv:1806.05594. (2019-02-21)[2021-04-15]. https://arxiv.org/abs/1806.05594.
[32] Van der Maaten L, Hinton G. Visualizing data using t-SNE[J]. Journal of Machine Learning Research, 2008, 9(86):2579-2605.