带有谱解耦正则的交叉熵损失的解

doi:10.7523/j.ucas.2023.071

中国科学院大学学报 ›› 2025, Vol. 42 ›› Issue (2): 268-275.DOI: 10.7523/j.ucas.2023.071

• 电子信息与计算机科学 • 上一篇

带有谱解耦正则的交叉熵损失的解

扈崟汉, 郭田德, 韩丛英

中国科学院大学数学科学学院, 北京 100049

收稿日期:2022-12-29 修回日期:2023-09-01 发布日期:2023-09-01
通讯作者: 韩丛英,E-mail:hancy@ucas.ac.cn
基金资助:
国家自然科学基金重点项目(11991022，U19B2040)和中央高校基本科研业务费专项资助

Solutions of cross-entropy loss with spectral decoupling regularization

HU Yinhan, GUO Tiande, HAN Congying

School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing 100049, China

Received:2022-12-29 Revised:2023-09-01 Published:2023-09-01

摘要/Abstract

摘要： 研究在过参数化线性模型下，不同强度的谱解耦正则对模型的影响。在没有权重衰减的情况下，证明不同强度的谱解耦正则得到的模型是等价的。在存在较小权重衰减时，用目标函数的二阶泰勒展开得到一个近似解，分析该近似解发现减小谱解耦正则有增强权重衰减的作用，并且在二分类问题中直接等价于增大权重衰减的系数。最后，通过实验验证该结论。

关键词: 交叉熵损失, 谱解耦正则, 权重衰减, 梯度饥饿, 神经网络

Abstract: In this paper, we study the effect of spectral decoupling with different strengths on over-parameterized models. In the absence of weight decay, we show that the models obtained by spectral decoupling of different strengths are equivalent. When there is a small weight decay, we use the second-order Taylor expansion of the objective function to obtain an approximate solution. Analyzing the approximate solution, we find that reducing the spectral decoupling has the effect of enhancing the weight decay, which is directly equivalent in the binary classification problem. Finally, we verify our analytical conclusions through experiments.

Key words: cross-entropy loss, spectral decoupling regularization, weight decay, gradient starvation, neural network

中图分类号:

TP391

扈崟汉, 郭田德, 韩丛英. 带有谱解耦正则的交叉熵损失的解[J]. 中国科学院大学学报, 2025, 42(2): 268-275.

HU Yinhan, GUO Tiande, HAN Congying. Solutions of cross-entropy loss with spectral decoupling regularization[J]. Journal of University of Chinese Academy of Sciences, 2025, 42(2): 268-275.

参考文献

[1] Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep convolutional neural networks[J]. Communications of the ACM, 2017, 60(6): 84-90. DOI: 10.1145/3065386.
[2] Hendrycks D, Mu N, Cubuk E D, et al. AugMix: a simple data processing method to improve robustness and uncertainty[EB/OL]. 2019. arXiv: 1912.02781.(2020-02-17)[2023-07-15]. https://arxiv.org/abs/1912.02781.
[3] Madry A, Makelov A, Schmidt L, et al. Towards deep learning models resistant to adversarial attacks[EB/OL]. 2017. arXiv: 1706.06083.(2019-09-04)[2023-07-15]. https://arxiv.org/abs/1706.06083.
[4] Jacot A, Gabriel F, Hongler C. Neural tangent kernel: convergence and generalization in neural networks[C]//Proceedings of the 32nd International Conference on Neural Information Processing Systems. December 3-8, 2018, Montréal, Canada. New York: ACM, 2018: 8580-8589. DOI: 10.5555/3327757.3327948.
[5] Daniel S, Elad H, Shpigel N M, et al. The implicit bias of gradient descent on separable data[J]. Journal of Machine Learning Research, 2018, 19(1): 2822-2878. DOI: 10.5555/3291125.3309632.
[6] Pezeshki M, Kaba S O, Bengio Y, et al. Gradient starvation: a learning proclivity in neural networks[EB/OL]. 2020. arXiv: 2011.09468. (2021-11-24)[2023-07-15]. https://arxiv.org/abs/2011.09468.
[7] Lewkowycz A, Gur-Ari G. On the training dynamics of deep networks with L₂ regularization[C]//Proceedings of the 34th International Conference on Neural Information Processing Systems. December 6-12, 2020, Vancouver, BC, Canada. New York: ACM, 2020: 4790-4799. DOI: 10.5555/3495724.3496126.
[8] Geirhos R, Jacobsen J H, Michaelis C, et al. Shortcut learning in deep neural networks[J]. Nature Machine Intelligence, 2020, 2(11): 665-673. DOI: 10.1038/s42256-020-00257-z.
[9] Shah H, Tamuly K, Raghunathan A, et al. The pitfalls of simplicity bias in neural networks[EB/OL]. 2020. arXiv: 2006.07710. (2020-10-28)[2023-07-15]. https://arxiv.org/abs/2006.07710.
[10] Xu Z Q J, Zhang Y Y, Luo T, et al. Frequency principle: Fourier analysis sheds light on deep neural networks[EB/OL]. 2019. arXiv: 1901.06523.(2019-09-20)[2023-07-15]. https://arxiv.org/abs/1901.06523.
[11] Bartlett P L, Long P M, Lugosi G, et al. Benign overfitting in linear regression[J]. Proceedings of the National Academy of Sciences of the United States of America, 2020, 117(48): 30063-30070. DOI: 10.1073/pnas.1907378117.
[12] Hastie T, Montanari A, Rosset S, et al. Surprises in high-dimensional ridgeless least squares interpolation[J]. Annals of Statistics, 2022, 50(2): 949-986. DOI: 10.1214/21-aos2133.
[13] Shamir O. The implicit bias of benign overfitting[EB/OL]. 2022. arXiv: 2201.11489. (2022-05-29)[2023-07-15]. https://arxiv.org/abs/2201.11489.
[14] Hsu D, Muthukumar V, Xu J. On the proliferation of support vectors in high dimensions[J]. Journal of Statistical Mechanics Theory and Experiment, 2022, 2022(11): 114011. DOI: 10.1088/1742-5468/ac98a9.
[15] Vidya M, Adhyyan N, Vignesh S, et al. Classification vs regression in overparameterized regimes: does the loss function matter?[J]. Journal of Machine Learning Research, 2021, 22(1):10104-10172. DOI: 10.5555/3546258.3546480.

带有谱解耦正则的交叉熵损失的解

Solutions of cross-entropy loss with spectral decoupling regularization

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

访问统计

联系我们

[1]	王艺璇, 张怀, 石耀霖, 程术. 基于LSTM神经网络的南加州中期地震预测[J]. 中国科学院大学学报, 2025, 42(2): 199-208.
[2]	汪展鹏, 王丽瑾. 一类随机Lotka-Volterra系统参数的神经网络学习[J]. 中国科学院大学学报, 2025, 42(1): 20-25.
[3]	袁鸣, 赵彤. 基于自监督异质图神经网络的图分类框架[J]. 中国科学院大学学报, 2024, 41(6): 830-841.
[4]	李雪源, 韩丛英. Actor-critic框架下的二次指派问题求解方法[J]. 中国科学院大学学报, 2024, 41(2): 275-284.
[5]	罗中凯, 张立波. 学习路径规划方法[J]. 中国科学院大学学报, 2024, 41(1): 11-27.
[6]	李元, 张栖, 朱建明, 焦建彬. 基于多层次深度模型的社交网络核心谣言传播节点识别[J]. 中国科学院大学学报, 2024, 41(1): 136-144.
[7]	董文豪, 张怀. 基于迁移学习的岩屑岩性识别[J]. 中国科学院大学学报, 2023, 40(6): 743-750.
[8]	刘子健, 姜泉江, 刘会杰. 基于神经网络的低轨卫星多波束系统峰均比自适应限幅法[J]. 中国科学院大学学报, 2023, 40(5): 670-676.
[9]	张萌, 潘志刚. 基于分层模糊聚类和小波卷积神经网络的SAR图像变化检测算法[J]. 中国科学院大学学报, 2023, 40(5): 637-646.
[10]	连会强, 刘兵, 李朋远, 于华. 销量约束下基于切片递归神经网络模型的成品油价格推荐算法[J]. 中国科学院大学学报, 2023, 40(4): 566-576.
[11]	姚沐风, 昝露洋, 李柏鹏, 李庆亭, 陈正超. 基于CAR-Siamese网络的高分辨率遥感图像建筑物变化检测[J]. 中国科学院大学学报, 2023, 40(3): 380-387.
[12]	庄子俊, 袁晓兵, 裴俊, 王国辉, 刘建坡. 基于无监督表达学习的森林地貌特征建模及林火易发性评估[J]. 中国科学院大学学报, 2023, 40(2): 227-239.
[13]	霍鑫怡, 李焱磊, 陈龙永, 张福博, 孙巍. 基于卷积注意力和胶囊网络的 SAR少样本目标识别方法[J]. 中国科学院大学学报, 2022, 39(6): 783-792.
[14]	曹哲, 冯珊珊, 孙显, 洪文. 基于图像分割和EM算法的PolSAR地物分类方法[J]. 中国科学院大学学报, 2022, 39(5): 639-647.
[15]	王树立, 唐海蓉, 计璐艳. 勺型网络：用于Landsat遥感图像云检测的新型网络[J]. 中国科学院大学学报, 2022, 39(5): 712-720.