摘要
经典的TF-IDF算法仅考虑了特征词频率和逆文档频率等,忽略了特征词的类间、类内分布信息.本文通过TF-IDF算法计算特征词在不同规模语料库中的权重,分析特征词的类信息对权重的影响,并进一步针对该影响提出一种新的衡量特征词的类间、类内分布信息的方法.本文通过增加两个新的权值,类间离散因子和类内离散因子,将其与经典的TF-IDF算法结合,提出了基于类信息的改进的TF-IDF-CI算法.本文通过朴素贝叶斯模型对改进后的算法的分类性能进行了验证.实验证明,改进后的权重算法在测试数据集上的表现,在准确率、召回率和F1值上均优于经典的TF-IDF算法.
The classical TF-IDF algorithm only considers the feature term frequency,inverse document frequency,etc.but overlooks the distribution information of feature terms between and inside categories.In this study,we calculate the weights of feature terms through the TF-IDF algorithm in the corpus with different scales and analyze the impact of category information on weights.Based on this,a new method is proposed to measure the distribution information of feature terms between and inside categories.Furthermore,an improved TF-IDF-DI algorithm based on category information is proposed by adding two new weights and discrete factors between and inside categories to the classic TFIDF algorithm.The Naive Bayes algorithm is used to validate the classification performance of the improved algorithm.Experiments show that the algorithm is superior to the classic TF-IDF algorithm in precision,recall,and F1 values.
作者
姚严志
李建良
YAO Yan-Zhi;LI Jian-Liang(School of Science,Nanjing University of Science and Technology,Nanjing 210094,China)
出处
《计算机系统应用》
2021年第9期237-241,共5页
Computer Systems & Applications
