Journal of University of Chinese Academy of Sciences >
Comparative study on text representation schemes in Chinese text classification
Received date: 2008-10-13
Revised date: 2008-11-07
Online published: 2009-05-15
We investigated the representation methods for text classification, proposed the framework of analyzing Chinese text representation algorithms, analyzed the influence of text representation, and obtained the influence of variable text representation factors on classification effect. Using Chinese characters can directly obtain better effect than expected; there is little difference on classification effect among splitting articles with smaller or huger dictionary or even by complicated splitting algorithm; and classification with only 01 to represent whether a feature is presented in a text or not can lead to not bad effect. We also found it can greatly improve classification effect to use reasonable vector value such as suitable formalization algorithm. These conclusions have provided instructions to contifurther applications.
Key words： Chinese text classification; text presentation; vectorization
ZHANG Ai-Hua , JING Ji-Wu , XIANG Ji . Comparative study on text representation schemes in Chinese text classification[J]. Journal of University of Chinese Academy of Sciences, 2009 , 26(3) : 400 -407 . DOI: 10.7523/j.issn.2095-6134.2009.3.015
[1] Sebastiani F. Machine learning in automated text categorization. ACM Computing Surveys,2002,34(1): 1~47
[2] Salton G, Wong A, Yang C. A vector space model for automatic indexing. Communication of the ACM,1975,18(11): 613~620
[3] Yang Y. A comparative study on feature selection in text categorization.In: Proceedings of the Fourteenth International Conference on Machine Learning (ICML'97). San Francisco: Morgan Kaufmann Publishers Inc, 1997. 412~420
[4] Su JS,Zhang BF,Xu X.Advances in machine learning based text categorization. Journal of Software,2006,17(9):1848~1859(in Chinese) 苏金树,张博锋,徐 昕. 基于机器学习的文本分类技术研究进展. 软件学报,2006,17(9):1848~1859
[5] Feng SC,Shan SW,Gong BH,et al.On the directory navigation service in Tianwang.Journal of Computer Research and Development,2004,41(4):653~659(in Chinese) 冯是聪,单松巍,龚笔宏,等. "天网"目录导航服务研究. 计算机研究与发展,2004,41(4):653~659
[6] Yang YM, Liu X. A re-examination of text categorization methods. In: Proceedings of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. New York: ACM Press, 1999. 42~49
[7] Luo K,Lin MG,Xi DM.Review of classification algorithms in data mining.Computer Engineering,2005,31(1):3~5,11(in Chinese) 罗 可,林睦纲,郗东妹. 数据挖掘中分类算法综述. 计算机工程,2005,31(1):3~5,11
[8] Li JY, Sun MS, Zhang X. A comparison and semi-quantitative analysis of words and character-bigrams as features in Chinese text categorization. In: Proceedings of the 21st International Conference on Computational Linguistics and the 44th Annual Meeting of the ACL. Morristown: Association for Computational Linguistics, 2006. 545~552
[9] Song FX, Liu SH, Yang JY. A comparative study on text representation schemes in text categorization. Pattern Analysis & Applications, 2005, 8(1):199~209
[10] Lang J, Lin F, Wang J. A comparative study on representing units in Chinese text clustering, Knowledge Science. In: Engineering and Management (KSEM2006). Heidelberg: Springer Berlin, 2006. 466~476
[11] Debole F, Sebastiani F.Supervised term weighting for automated text categorization.In: Proceedings of the 2003 ACM Symposium on Applied Computing. New York: ACM Press, 2003. 784~788
[12] 搜狗词典.http://www.sogou.com/labs/dl/w.html,
[13] 计算所ICTCLAS分词系统.http://www.ictclas.org/,
[14] 北大分类数据.http://www.infomall.cn/trainset-intro.pdf,
[15] 搜狐分类数据.http://www.sogou.com/labs/dl/c.html,
[16] 复旦分类数据.http://www.nlp.org.cn/docs/download.php?doc-id=281,
[17] Harmon DK. Overview of the third text retrieval conference (Trec-3). Gaithersburg: DIANE Publishing, 1995.69~80
/
| 〈 |
|
〉 |