中文文本分类中的文本表示因素比较

doi:10.7523/j.issn.2095-6134.2009.3.015

中国科学院大学学报 ›› 2009, Vol. 26 ›› Issue (3): 400-407.DOI: 10.7523/j.issn.2095-6134.2009.3.015

中文文本分类中的文本表示因素比较

张爱华^1,2, 荆继武², 向继²

1. 中国科学技术大学电子工程与信息科学系, 合肥 230027;
2. 中国科学院研究生院信息安全国家重点实验室, 北京 100049

收稿日期:2008-10-13 修回日期:2008-11-07 发布日期:2009-05-15
通讯作者: 张爱华
基金资助:
国家863研究计划(2006AA01Z454)项目资助 

Comparative study on text representation schemes in Chinese text classification

ZHANG Ai-Hua^1,2, JING Ji-Wu², XIANG Ji²

1. Department of Electronic Engineering and Information Science, University of Science and Technology of China, Hefei 230027, China;
2. State Key Laboratory of Information Security, Graduate University of the Chinese Academy of Sciences, Beijing 100049, China

Received:2008-10-13 Revised:2008-11-07 Published:2009-05-15

摘要/Abstract

摘要：

研究了中文文本分类中的文本表示方法,提出了对中文文本表示因素的分析框架,并通过对3个数据集实验结果的分析,确定了各种文本表示因素对分类效果的影响.直接使用汉字进行划分也可以获得较好的分类效果;简单的不使用很大词库的分词和使用大词库的分词,以及复杂的分词对分类效果影响不大;仅使用01表示特征是否出现也可以获得比较好的分类效果;采用综合了合理的向量取值(如使用合适的归一化算法)可以较大幅度地提高分类准确率等.这些结论为后续的应用提供了指导原则.

关键词: 中文文本分类, 文本表示, 向量化

Abstract:

We investigated the representation methods for text classification, proposed the framework of analyzing Chinese text representation algorithms, analyzed the influence of text representation, and obtained the influence of variable text representation factors on classification effect. Using Chinese characters can directly obtain better effect than expected; there is little difference on classification effect among splitting articles with smaller or huger dictionary or even by complicated splitting algorithm; and classification with only 01 to represent whether a feature is presented in a text or not can lead to not bad effect. We also found it can greatly improve classification effect to use reasonable vector value such as suitable formalization algorithm. These conclusions have provided instructions to contifurther applications.

Key words: Chinese text classification, text presentation, vectorization

中图分类号:

TP31

张爱华, 荆继武, 向继. 中文文本分类中的文本表示因素比较[J]. 中国科学院大学学报, 2009, 26(3): 400-407.

ZHANG Ai-Hua, JING Ji-Wu, XIANG Ji. Comparative study on text representation schemes in Chinese text classification[J]. , 2009, 26(3): 400-407.

参考文献

[1] Sebastiani F. Machine learning in automated text categorization. ACM Computing Surveys,2002,34(1): 1~47

[2] Salton G, Wong A, Yang C. A vector space model for automatic indexing. Communication of the ACM,1975,18(11): 613~620

[3] Yang Y. A comparative study on feature selection in text categorization.In: Proceedings of the Fourteenth International Conference on Machine Learning (ICML'97). San Francisco: Morgan Kaufmann Publishers Inc, 1997. 412~420

[4] Su JS,Zhang BF,Xu X.Advances in machine learning based text categorization. Journal of Software,2006,17(9):1848~1859(in Chinese) 苏金树,张博锋,徐昕. 基于机器学习的文本分类技术研究进展. 软件学报,2006,17(9):1848~1859

[5] Feng SC,Shan SW,Gong BH,et al.On the directory navigation service in Tianwang.Journal of Computer Research and Development,2004,41(4):653~659(in Chinese) 冯是聪,单松巍,龚笔宏,等. "天网"目录导航服务研究. 计算机研究与发展,2004,41(4):653~659

[6] Yang YM, Liu X. A re-examination of text categorization methods. In: Proceedings of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. New York: ACM Press, 1999. 42~49

[7] Luo K,Lin MG,Xi DM.Review of classification algorithms in data mining.Computer Engineering,2005,31(1):3~5,11(in Chinese) 罗可,林睦纲,郗东妹. 数据挖掘中分类算法综述. 计算机工程,2005,31(1):3~5,11

[8] Li JY, Sun MS, Zhang X. A comparison and semi-quantitative analysis of words and character-bigrams as features in Chinese text categorization. In: Proceedings of the 21st International Conference on Computational Linguistics and the 44th Annual Meeting of the ACL. Morristown: Association for Computational Linguistics, 2006. 545~552

[9] Song FX, Liu SH, Yang JY. A comparative study on text representation schemes in text categorization. Pattern Analysis & Applications, 2005, 8(1):199~209

[10] Lang J, Lin F, Wang J. A comparative study on representing units in Chinese text clustering, Knowledge Science. In: Engineering and Management (KSEM2006). Heidelberg: Springer Berlin, 2006. 466~476

[11] Debole F, Sebastiani F.Supervised term weighting for automated text categorization.In: Proceedings of the 2003 ACM Symposium on Applied Computing. New York: ACM Press, 2003. 784~788

[12] 搜狗词典.http://www.sogou.com/labs/dl/w.html,

[13] 计算所ICTCLAS分词系统.http://www.ictclas.org/,

[14] 北大分类数据.http://www.infomall.cn/trainset_-intro.pdf,

[15] 搜狐分类数据.http://www.sogou.com/labs/dl/c.html,

[16] 复旦分类数据.http://www.nlp.org.cn/docs/download.php?doc_-id=281,

[17] Harmon DK. Overview of the third text retrieval conference (Trec-3). Gaithersburg: DIANE Publishing, 1995.69~80

中文文本分类中的文本表示因素比较

Comparative study on text representation schemes in Chinese text classification

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 1

编辑推荐

Metrics

本文评价

访问统计

联系我们