Welcome to Journal of University of Chinese Academy of Sciences,Today is
Review Article

Survey on angle-based classification

  • FU Sheng ,
  • XUE Yuan ,
  • ZHANG Sanguo
Expand
  • School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing 100049, China

Received date: 2018-01-02

  Revised date: 2018-01-02

  Online published: 2019-05-15

Supported by

Supported by the Special Fund of University of Chinese Academy of Sciences for Scientific Research Cooperation (Y652022Y00)

Abstract

Statistical classification problems are widely encountered in many applications, e.g., face recognition, fraud detection, and hand-written character recognition. In this article we make a comprehensive analysis on statistical methods for supervised classification problems. Specifically, we introduce the angle-based classification structure, which combines binary and multicategory problems in a unified framework. Several new variants of the angle-based classifiers are also discussed, such as robust learning and weighted learning. Furthermore, we show some theoretical results about Fisher consistency for these angle-based classifiers.

Cite this article

FU Sheng , XUE Yuan , ZHANG Sanguo . Survey on angle-based classification[J]. Journal of University of Chinese Academy of Sciences, 2019 , 36(3) : 289 -298 . DOI: 10.7523/j.issn.2095-6134.2019.03.001

References

[1] Mitchell T M. Machine learning[M]. New York:McGraw-Hill Inc, 1997.
[2] Duda R O, Hart P E, Stork D G. Pattern classification[M]. 2nd ed. New York:Wiley-Interscience, 2000.
[3] Bishop C M. Pattern recognition and machine learning[M]. New York:Springer, 2006.
[4] Hastie T, Tibshirani R, Friedman J. The elements of statistical learning:data mining, inference and prediction[M]. 2nd ed. New York:Springer, 2009.
[5] Murphy K P. Machine learning:a probabilistic perspective[M]. Cambridge, MA:MIT press, 2012.
[6] Wahba G. Support vector machines, reproducing kernel Hilbert spaces and the randomized GACV[M]//Advances in Kernel Methods-Support Vector Learning:vol 6. Cambridge, MA:MIT Press, 1999:69-88.
[7] Boser B E, Guyon I M, Vapnik V N. A training algorithm for optimal margin classifiers[C]//Proceedings of the fifth annual workshop on computational learning theory. New York:ACM, 1992:144-152.
[8] Cortes C, Vapnik V. Support-vector networks[J]. Machine Learning, 1995, 20(3):273-297.
[9] Rosenblatt F. The perceptron, a perceiving and recognizing automaton project para[M]. Buffalo, NY:Cornell Aeronautical Laboratory, 1957.
[10] Cox D R. The regression analysis of binary sequences[J]. Journal of the Royal Statistical Society. Series B, 1958, 20(2):215-242.
[11] Walker S H, Duncan D B. Estimation of the probability of an event as a function of several independent variables[J]. Biometrika, 1967, 54(1):167-179.
[12] Freund Y, Schapire R E. A decision-theoretic generalization of on-line learning and an application to boosting[J]. Journal of Computer and System Sciences, 1997, 55(1):119-139.
[13] Suykens J A, Vandewalle J. Least squares support vector machine classifiers[J]. Neural Processing Letters, 1999, 9(3):293-300.
[14] Chang K W, Hsieh C J, Lin C J. Coordinate descent method for large-scale l2-loss linear support vector machines[J]. Journal of Machine Learning Research, 2008, 9(Jul):1369-1398.
[15] Liu Y, Zhang H H, Wu Y. Hard or soft classification Large-margin unified machines[J]. Journal of the American Statistical Association, 2011, 106(493):166-177.
[16] Marron J S, Todd M J, Ahn J. Distance-weighted discrimination[J]. Journal of the American Statistical Association, 2007, 102(480):1267-1271.
[17] Marron J S. Distance-weighted discrimination[J]. Wiley Interdisciplinary Reviews:Computational Statistics, 2015, 7(2):109-114.
[18] Tibshirani R. Regression shrinkage and selection via the lasso[J]. Journal of the Royal Statistical Society. Series B, 1996, 58(1):267-288.
[19] Liu Y, Zhang H H, Park C, et al. Support vector machines with adaptive Lq penalty[J]. Computational Statistics & Data Analysis, 2007, 51(12):6380-6394.
[20] Zou H, Hastie T. Regularization and variable selection via the elastic net[J]. Journal of the Royal Statistical Society. Series B, 2005, 67(2):301-320.
[21] Yuan M, Lin Y. Model selection and estimation in regression with grouped variables[J]. Journal of the Royal Statistical Society. Series B, 2006, 68(1):49-67.
[22] Fan J, Li R. Variable selection via nonconcave penalized likelihood and its oracle properties[J]. Journal of the American Statistical Association, 2001, 96(456):1348-1360.
[23] Zhang C H. Nearly unbiased variable selection under minimax concave penalty[J]. The Annals of Statistics, 2010, 38(2):894-942.
[24] Boyd S, Vandenberghe L. Convex optimization[M]. New York:Cambridge University Press, 2004.
[25] Dietterich T G, Bakiri G. Solving multiclass learning problems via error-correcting output codes[J]. Journal of Artificial Intelligence Research, 1995, 2:263-286.
[26] Allwein E L, Schapire R E, Singer Y. Reducing multiclass to binary:a unifying approach for margin classifiers[J]. Journal of Machine Learning Research, 2001, 1(2):113-141.
[27] Lee Y, Lin Y, Wahba G. Multicategory support vector machines:Theory and application to the classification of microarray data and satellite radiance data[J]. Journal of the American Statistical Association, 2004, 99(465):67-81.
[28] Liu Y, Yuan M. Reinforced multicategory support vector machines[J]. Journal of Computational and Graphical Statistics, 2011, 20(4):901-919.
[29] Weston J, Watkins C. Support vector machines for multi-class pattern recognition[C]//Proc European Symposium on Artificial Neural Networks:vol 99. Bruges, Belgium:D-Facto public, 1999:219-224.
[30] Crammer K, Singer Y. On the algorithmic implementation of multiclass kernel-based vector machines[J]. Journal of Machine Learning Research, 2002, 2(2):265-292.
[31] Tang Y, Zhang H H. Multiclass proximal support vector machines[J]. Journal of Computational and Graphical Statistics, 2006, 15(2):339-355.
[32] Park S Y, Liu Y, Liu D, et al. Multicategory composite least squares classifiers[J]. Statistical Analysis & Data Mining, 2010, 3(4):272-286.
[33] Zhang C, Liu Y. Multicategory large-margin unified machines[J]. Journal of Machine Learning Research, 2013, 14(1):1349-1386.
[34] Shen X, Tseng G C, Zhang X, et al. On ψ-learning[J]. Journal of the American Statistical Association, 2003, 98(463):724-734.
[35] Liu Y, Shen X. Multicategory ψ-learning[J]. Journal of the American Statistical Association, 2006, 101(474):500-509.
[36] Wu Y, Liu Y. On multicategory truncated hinge loss support vector machines[M]//Prediction and Discovery:AMS-IMS-SIAM Joint Summer Research Conference, Machine and Statistical Learning:volume 443. Snowbird, Utah:American Mathematical Society, 2006:49-58.
[37] Wu Y, Liu Y. Robust truncated hinge loss support vector machines[J]. Journal of the American Statistical Association, 2007, 102(479):974-983.
[38] Wu Y, Liu Y. Adaptively weighted large margin classifiers[J]. Journal of Computational and Graphical Statistics, 2013, 22(2):416-432.
[39] Zhang C, Liu Y. Multicategory angle-based large-margin classification[J]. Biometrika, 2014, 101(3):625-640.
[40] Hill S I, Doucet A. A framework for kernel-based multi-category classification[J]. Journal of Artificial Intelligence Research, 2007, 30:525-564.
[41] Lange K, Wu T. An MM algorithm for multicategory vertex discriminant analysis[J]. Journal of Computational and Graphical Statistics, 2008, 17(3):527-544.
[42] Saberian M J, Vasconcelos N. Multiclass boosting:theory and algorithms[C]//Advances in Neural Information Processing Systems:vol 24. Granada, Spain:Curran Associates, Inc, 2011:2124-2132.
[43] Mroueh Y, Poggio T, Rosasco L, et al. Multiclass learning with simplex coding[C]//Advances in Neural Information Processing Systems:vol 25. Lake Tahoe, Nevada:Curran Associates, Inc, 2012:2789-2797.
[44] Zhang C, Liu Y, Wang J, et al. Reinforced angle-based multicategory support vector machines[J]. Journal of Computational and Graphical Statistics, 2016, 25(3):806-825.
[45] Fu S, Zhang S, Liu Y. Adaptively weighted large-margin angle-based classifiers[J]. Journal of Multivariate Analysis, 2018, 166:282-299.
[46] Zhang C, Pham M, Fu S, et al. Robust multicategory support vector machines using difference convex algorithm[J]. Mathematical Programming, 2017. doi:10.1007/s10107-017-1209-5.
[47] Vapnik V N. Statistical learning theory[M]. New York:Wiley, 1998.
[48] Zhang T.Statistical behavior and consistency of classification methods based on convex risk minimization[J]. The Annals of Statistics, 2004, 32(1):56-85.
[49] Zhang T. Statistical analysis of some multi-category large margin classification methods[J]. Journal of Machine Learning Research, 2004, 5(Oct):1225-1251.
[50] Bartlett P L, Jordan M I, McAuliffe J D. Convexity, classification, and risk bounds[J]. Journal of the American Statistical Association, 2006, 101(473):138-156.
[51] Liu Y. Fisher consistency of multicategory support vector machines[C]//Proceedings of the Eleventh International Conference on Artificial Intelligence and Statistics:volume 2. San Juan, Puerto Rico:PMLR, 2007:291-298.
[52] Zou H, Zhu J, Hastie T. New multicategory boosting algorithms based on multicategory Fisher-consistent losses[J]. The Annals of Applied Statistics, 2008, 2(4):1290.
[53] Zhang C, Lu X, Zhu Z, et al. REC:fast sparse regression-based multicategory classification[J]. Statistics and Its Interface, 2017, 10(2):175-185.
[54] Sun H, Craig B A, Zhang L. Angle-based multicategory distance-weighted SVM[J]. Journal of Machine Learning Research, 2017, 18(1):2981-3001.
Outlines

/