Comparison of bias-variance structure of three classification algorithms:MCLP, LDA and C5.0

doi:10.7523/j.issn.2095-6134.2009.4.003

Abstract

Abstract:

Based on Domingos bias-variance decomposition framework, on three different data sets, we compared the bias-variance structure of the three classification methods: MCLP, LDA and C5.0. The experimental results showed that, generally speaking, C5.0 has low bias and high variance, LDA has high bias and low variance, and MCLP is in between them but near LDA. When the training set is small, bias and variance of MCLP is comparatively high. However, with the increasing of training set, bias and variance of MCLP obviously decrease and even are lower than those of C5.0 and LDA. This study established the basis for constructing the ensemble suited to MCLP.

Key words: multiple-criteria linear programming(MCLP), linear discrimant analysis(LDA), C5.0, bias, variance

CLC Number:

O29
N945

ZHU Mei-Hong, SHI Yong, LI Ai-Hua, ZHANG Dong-Ling. Comparison of bias-variance structure of three classification algorithms:MCLP, LDA and C5.0[J]. , 2009, 26(4): 443-450.

References

[1] Shi Y. Multiple criteria mult iple constraint-levels linear programming: concepts, techniques and application s
[M]. New Jersey, River Edge:World Scientific Publishing, 2001.

[2] Shi Y, Wise M, Luo M, et al. Data mining in credit card portfolio management: a multiple cri teria decision making approach
[C]//Koksalan M, Zionts S (Eds.). Mult iple Criteria Decision Making in the New Millennium. Berlin: Springer,2001:427 -436.

[3] Shi Y, Peng Y, Xu X, et al. Data mining via multiple crite ria linear programming: applications in credit card portfolio management
[J]. Information Technology and Decision Making, 2002, 1 (1): 145-166.

[4] Kou G, Liu X, Peng Y, et al. Multiple criteria linear prog ramm ing approach to data mining: models, algorithm designs and software development
[J]. Optimization Methods and Software, 2003, 18 (4): 453-473.

[5] He J, Shi Y, Xu W X. Classificati ons of credit cardholder behavior by using multiple mriteria non-linear program ming
[C]//Shi Y, Xu W X, Chen Z X(Eds.). CASDMKM 2004, LNAI 3327. 2004: 154 -163.

[6] Peng Y, Shi Y, Li X S, et al. E-business intelligence via MCMP -based data mining methods
[C]//Zhong N, et al(Eds.).WImBI2006,LNAI4845 . 2007: 443-453.

[7] Zhu M H, Li A H, Shi Y, et al. A bias-variance analysis o f mu ltiple criteria linear programming classification ensembles
[C]//2008IEEE /WIC/ACM International Conference on Web Intelligence and Intelligent Agent Tech nology. 2008:447-450.

[8] Shi Y, He J,Wang L,et al. Computer-based a lgorithms for mul tiple criteria and multipleconstraint level integer linear programming
[J]. Computersand Mathematics with Applications,
[WTBZ] 2005,49:903-921.

[9] Li A H, Shi Y. An integr ated classification method: combination of MCLP and LDA
[C]//Deng X, Y e Y(Eds.). WINE2005, LNCS3828.2005: 758-767.

[10] Peng Y, Kou G, Shi Y, et al. A new m ulti-criteria convex quadratic programming mode for credit data analysis
[J]. Decision Support Systems
[WTBZ], 2008.10.1016/j.dss.2007.12.001.

[11] Domingos P. A unified bias-variance decomposition and its applicati ons
[C]//Proceedings of the 17th International Conference on Machine Learning . Stanford, CA: Morgan Kaufmann, 2000: 231-238.

[12] Geman S, Bienenstock E, Doursat R. Neural networks and the bias/ variance dilemma
[J]. Neural Computation
[WTBZ], 1992, 4: 1-48.

[13] Kong E B, Dietterich T G. Error-correc ting output coding corrects bias and variance
[C]//Proceedings of the 12t h International Conference on Machine Learning.1995: 313-321 C.

[14] Kohavi R, Wolpert D H. Bias plus variance decomposition for zero-o ne loss functions
[C]//Proce edings of the 13th International Conference on Machine Learning, the 7th Interna tional Conference on Machine Learning. Bari, Italy: Morgan Kaufm ann, 1996: 275-283.

[15] Gareth M J. Variance and bias for general loss functions
[J]. Machine Learning, 2003, 51(2): 115-135.

[16] Valentini G, Dietterich T G. Bias-variance analysis of support vector machines for the development of SV M-based ensemble methods
[J]. Journal of Machine Learning Research,
[WTBZ] 2004,5: 725-775.

[17] Richard A J, Dean W W. Applied multivariate statistical analysis
[M]. 北京:清华大学出版社,2001.

[18] Glen J J. A comparison of standard and two-stage mathematical prog ramming discriminant analysis methods
[J]. European Journal of Operatio nal Research,
[WTBZ]2006,171:496-515.

[19] Han J W, Kamber M. 数据挖掘:概念与技术
[M].范明,孟小峰译.北京 :机械工业出版社,2007.

[20] UCI repository of machine learning databases
[R]. http://www.ics. uci.edu/~mlearn/ MLRepository.html.