Survey on angle-based classification

doi:10.7523/j.issn.2095-6134.2019.03.001

Abstract

Abstract: Statistical classification problems are widely encountered in many applications, e.g., face recognition, fraud detection, and hand-written character recognition. In this article we make a comprehensive analysis on statistical methods for supervised classification problems. Specifically, we introduce the angle-based classification structure, which combines binary and multicategory problems in a unified framework. Several new variants of the angle-based classifiers are also discussed, such as robust learning and weighted learning. Furthermore, we show some theoretical results about Fisher consistency for these angle-based classifiers.

Key words: angle-based classification framework, Fisher consistency, robust learning, statistical classification, weighted learning

CLC Number:

FU Sheng, XUE Yuan, ZHANG Sanguo. Survey on angle-based classification[J]. , 2019, 36(3): 289-298.

References

[1] Mitchell T M. Machine learning[M]. New York:McGraw-Hill Inc, 1997.
[2] Duda R O, Hart P E, Stork D G. Pattern classification[M]. 2nd ed. New York:Wiley-Interscience, 2000.
[3] Bishop C M. Pattern recognition and machine learning[M]. New York:Springer, 2006.
[4] Hastie T, Tibshirani R, Friedman J. The elements of statistical learning:data mining, inference and prediction[M]. 2nd ed. New York:Springer, 2009.
[5] Murphy K P. Machine learning:a probabilistic perspective[M]. Cambridge, MA:MIT press, 2012.
[6] Wahba G. Support vector machines, reproducing kernel Hilbert spaces and the randomized GACV[M]//Advances in Kernel Methods-Support Vector Learning:vol 6. Cambridge, MA:MIT Press, 1999:69-88.
[7] Boser B E, Guyon I M, Vapnik V N. A training algorithm for optimal margin classifiers[C]//Proceedings of the fifth annual workshop on computational learning theory. New York:ACM, 1992:144-152.
[8] Cortes C, Vapnik V. Support-vector networks[J]. Machine Learning, 1995, 20(3):273-297.
[9] Rosenblatt F. The perceptron, a perceiving and recognizing automaton project para[M]. Buffalo, NY:Cornell Aeronautical Laboratory, 1957.
[10] Cox D R. The regression analysis of binary sequences[J]. Journal of the Royal Statistical Society. Series B, 1958, 20(2):215-242.
[11] Walker S H, Duncan D B. Estimation of the probability of an event as a function of several independent variables[J]. Biometrika, 1967, 54(1):167-179.
[12] Freund Y, Schapire R E. A decision-theoretic generalization of on-line learning and an application to boosting[J]. Journal of Computer and System Sciences, 1997, 55(1):119-139.
[13] Suykens J A, Vandewalle J. Least squares support vector machine classifiers[J]. Neural Processing Letters, 1999, 9(3):293-300.
[14] Chang K W, Hsieh C J, Lin C J. Coordinate descent method for large-scale l2-loss linear support vector machines[J]. Journal of Machine Learning Research, 2008, 9(Jul):1369-1398.
[15] Liu Y, Zhang H H, Wu Y. Hard or soft classification Large-margin unified machines[J]. Journal of the American Statistical Association, 2011, 106(493):166-177.
[16] Marron J S, Todd M J, Ahn J. Distance-weighted discrimination[J]. Journal of the American Statistical Association, 2007, 102(480):1267-1271.
[17] Marron J S. Distance-weighted discrimination[J]. Wiley Interdisciplinary Reviews:Computational Statistics, 2015, 7(2):109-114.
[18] Tibshirani R. Regression shrinkage and selection via the lasso[J]. Journal of the Royal Statistical Society. Series B, 1996, 58(1):267-288.
[19] Liu Y, Zhang H H, Park C, et al. Support vector machines with adaptive Lq penalty[J]. Computational Statistics & Data Analysis, 2007, 51(12):6380-6394.
[20] Zou H, Hastie T. Regularization and variable selection via the elastic net[J]. Journal of the Royal Statistical Society. Series B, 2005, 67(2):301-320.
[21] Yuan M, Lin Y. Model selection and estimation in regression with grouped variables[J]. Journal of the Royal Statistical Society. Series B, 2006, 68(1):49-67.
[22] Fan J, Li R. Variable selection via nonconcave penalized likelihood and its oracle properties[J]. Journal of the American Statistical Association, 2001, 96(456):1348-1360.
[23] Zhang C H. Nearly unbiased variable selection under minimax concave penalty[J]. The Annals of Statistics, 2010, 38(2):894-942.
[24] Boyd S, Vandenberghe L. Convex optimization[M]. New York:Cambridge University Press, 2004.
[25] Dietterich T G, Bakiri G. Solving multiclass learning problems via error-correcting output codes[J]. Journal of Artificial Intelligence Research, 1995, 2:263-286.
[26] Allwein E L, Schapire R E, Singer Y. Reducing multiclass to binary:a unifying approach for margin classifiers[J]. Journal of Machine Learning Research, 2001, 1(2):113-141.
[27] Lee Y, Lin Y, Wahba G. Multicategory support vector machines:Theory and application to the classification of microarray data and satellite radiance data[J]. Journal of the American Statistical Association, 2004, 99(465):67-81.
[28] Liu Y, Yuan M. Reinforced multicategory support vector machines[J]. Journal of Computational and Graphical Statistics, 2011, 20(4):901-919.
[29] Weston J, Watkins C. Support vector machines for multi-class pattern recognition[C]//Proc European Symposium on Artificial Neural Networks:vol 99. Bruges, Belgium:D-Facto public, 1999:219-224.
[30] Crammer K, Singer Y. On the algorithmic implementation of multiclass kernel-based vector machines[J]. Journal of Machine Learning Research, 2002, 2(2):265-292.
[31] Tang Y, Zhang H H. Multiclass proximal support vector machines[J]. Journal of Computational and Graphical Statistics, 2006, 15(2):339-355.
[32] Park S Y, Liu Y, Liu D, et al. Multicategory composite least squares classifiers[J]. Statistical Analysis & Data Mining, 2010, 3(4):272-286.
[33] Zhang C, Liu Y. Multicategory large-margin unified machines[J]. Journal of Machine Learning Research, 2013, 14(1):1349-1386.
[34] Shen X, Tseng G C, Zhang X, et al. On ψ-learning[J]. Journal of the American Statistical Association, 2003, 98(463):724-734.
[35] Liu Y, Shen X. Multicategory ψ-learning[J]. Journal of the American Statistical Association, 2006, 101(474):500-509.
[36] Wu Y, Liu Y. On multicategory truncated hinge loss support vector machines[M]//Prediction and Discovery:AMS-IMS-SIAM Joint Summer Research Conference, Machine and Statistical Learning:volume 443. Snowbird, Utah:American Mathematical Society, 2006:49-58.
[37] Wu Y, Liu Y. Robust truncated hinge loss support vector machines[J]. Journal of the American Statistical Association, 2007, 102(479):974-983.
[38] Wu Y, Liu Y. Adaptively weighted large margin classifiers[J]. Journal of Computational and Graphical Statistics, 2013, 22(2):416-432.
[39] Zhang C, Liu Y. Multicategory angle-based large-margin classification[J]. Biometrika, 2014, 101(3):625-640.
[40] Hill S I, Doucet A. A framework for kernel-based multi-category classification[J]. Journal of Artificial Intelligence Research, 2007, 30:525-564.
[41] Lange K, Wu T. An MM algorithm for multicategory vertex discriminant analysis[J]. Journal of Computational and Graphical Statistics, 2008, 17(3):527-544.
[42] Saberian M J, Vasconcelos N. Multiclass boosting:theory and algorithms[C]//Advances in Neural Information Processing Systems:vol 24. Granada, Spain:Curran Associates, Inc, 2011:2124-2132.
[43] Mroueh Y, Poggio T, Rosasco L, et al. Multiclass learning with simplex coding[C]//Advances in Neural Information Processing Systems:vol 25. Lake Tahoe, Nevada:Curran Associates, Inc, 2012:2789-2797.
[44] Zhang C, Liu Y, Wang J, et al. Reinforced angle-based multicategory support vector machines[J]. Journal of Computational and Graphical Statistics, 2016, 25(3):806-825.
[45] Fu S, Zhang S, Liu Y. Adaptively weighted large-margin angle-based classifiers[J]. Journal of Multivariate Analysis, 2018, 166:282-299.
[46] Zhang C, Pham M, Fu S, et al. Robust multicategory support vector machines using difference convex algorithm[J]. Mathematical Programming, 2017. doi:10.1007/s10107-017-1209-5.
[47] Vapnik V N. Statistical learning theory[M]. New York:Wiley, 1998.
[48] Zhang T.Statistical behavior and consistency of classification methods based on convex risk minimization[J]. The Annals of Statistics, 2004, 32(1):56-85.
[49] Zhang T. Statistical analysis of some multi-category large margin classification methods[J]. Journal of Machine Learning Research, 2004, 5(Oct):1225-1251.
[50] Bartlett P L, Jordan M I, McAuliffe J D. Convexity, classification, and risk bounds[J]. Journal of the American Statistical Association, 2006, 101(473):138-156.
[51] Liu Y. Fisher consistency of multicategory support vector machines[C]//Proceedings of the Eleventh International Conference on Artificial Intelligence and Statistics:volume 2. San Juan, Puerto Rico:PMLR, 2007:291-298.
[52] Zou H, Zhu J, Hastie T. New multicategory boosting algorithms based on multicategory Fisher-consistent losses[J]. The Annals of Applied Statistics, 2008, 2(4):1290.
[53] Zhang C, Lu X, Zhu Z, et al. REC:fast sparse regression-based multicategory classification[J]. Statistics and Its Interface, 2017, 10(2):175-185.
[54] Sun H, Craig B A, Zhang L. Angle-based multicategory distance-weighted SVM[J]. Journal of Machine Learning Research, 2017, 18(1):2981-3001.