自然语言处理（Natural Language Processing，简称NLP），是为各类企业及开发者提供的用于文本分析及挖掘的核心工具，旨在帮助用户高效的处理文本，已经广泛应用在电商、文娱、司法、公安、金融、医疗、电力等行业客户的多项业务中，取得了良好的效果。
1、项目背景
任何行业领域，用户对产品的评价都显得尤为重要。通过用户评论，可以对用户情感倾向进行判定。
例如，目前最为普遍的网购行为：对于用户来说，参考评论可以做出更优的购买决策；对于商家来说，对商品评论按照情感倾向进行分类，并通过文本聚类得到普遍提及的商品优缺点，可以进一步改良产品。
本案例主要讨论如何对商品评论进行情感倾向判定。下图为某电商平台上针对某款手机的部分评论：
2、数据集
这份某款手机的商品评论信息数据集，包含2个属性，共计8187个样本。
使用Pandas中的read_excel函数读取xls格式的数据集文件，注意文件的编码设置为gb18030，代码如下所示：
import pandas as pd
#读入数据集
data = pd.read_excel("data.xls", encoding='gb18030')
print(data.head())读取数据集效果（部分）如下所示：
查看数据集的相关信息，包括行列数，列名，以及各个类别的样本数，实现代码如下所示：
代码语言：javascript复制# 数据集的大小
print(data.shape)
# 数据集的列名
print(data.columns.values)
# 不同类别数据记录的统计
print(data['Class'].value_counts())效果如下所示
代码语言：javascript复制(8186, 2)
array([u'Comment', u'Class'], dtype=object)
1 3042
-1 2657
0 2487
Name: Class, dtype: int643、数据预处理
现在，我们要将Comment列的文本信息，转化成数值矩阵表示，也就是将文本映射到特征空间。
首先，通过jieba，使用HMM模型，对文本进行中文分词，实现代码如下所示：
# 导入中文分词库jieba
import jieba
import numpy as np接下来，对数据集的每个样本的文本进行中文分词，如遇到缺失值，使用“还行、一般吧”进行填充，实现代码如下所示：
代码语言：javascript复制cutted = []
for row in data.values:
try:
raw_words = (" ".join(jieba.cut(row[0])))
cutted.append(raw_words)
except AttributeError:
print row[0]
cutted.append(u"还行 一般吧")
cutted_array = np.array(cutted)
# 生成新数据文件，Comment字段为分词后的内容
data_cutted = pd.DataFrame({
'Comment': cutted_array,
'Class': data['Class']
})读取并查看预处理后的数据，实现代码如下所示：
代码语言：javascript复制print(data_cutted.head())数据集效果（部分）如下所示：
为了更直观地观察词频高的词语，我们使用第三方库wordcloud进行文本的可视化，导入库实现代码如下所示：
# 导入第三方库wordcloud
from wordcloud import WordCloud
import matplotlib.pyplot as plt针对好评，中评和差评的文本，建立WordCloud对象，绘制词云，好评词云可视化实现代码如下所示：
# 好评
wc = WordCloud(font_path='Courier.ttf')
wc.generate(''.join(data_cutted['Comment'][data_cutted['Class'] == 1]))
plt.axis('off')
plt.imshow(wc)
plt.show()好评词云效果如下所示：
中评词云可视化实现代码如下所示：
代码语言：javascript复制# 中评
wc = WordCloud(font_path='Courier.ttf')
wc.generate(''.join(data_cutted['Comment'][data_cutted['Class'] == 0]))
plt.axis('off')
plt.imshow(wc)
plt.show()中评词云效果如下所示：
差评词云可视化实现代码如下所示：
代码语言：javascript复制# 差评
wc = WordCloud(font_path='Courier.ttf')
wc.generate(''.join(data_cutted['Comment'][data_cutted['Class'] == -1]))
plt.axis('off')
plt.imshow(wc)
plt.show()差评词云效果如下所示：
从词云展现的词频统计图来看，"手机"，"就是"，"屏幕"，"收到"等词对于区分毫无帮助而且会造成偏差。因此，需要把这些对区分类没有意义的词语筛选出来，放到停用词文件stopwords.txt中。实现代码如下所示：
# 读入停用词文件
import codecs
with codecs.open('stopwords.txt', 'r', encoding='utf-8') as f:
stopwords = [item.strip() for item in f]
for item in stopwords[0:200]:
print(item,)输出停用词效果如下所示：
使用jieba库的extract_tags函数，统计好评，中评，差评文本中的TOP20关键词。
#设定停用词文件,在统计关键词的时候，过滤停用词
import jieba.analyse
jieba.analyse.set_stop_words('stopwords.txt') 好评关键词分析，实现代码如下所示：
代码语言：javascript复制# 好评关键词
keywords_pos = jieba.analyse.extract_tags(''.join(data_cutted['Comment'][data_cutted['Class'] == 1]), topK=20)
for item in keywords_pos:
print（item,）好评关键词TOP20如下所示：
不错 正品 赠品 五分 发货 东西 满意 机子 喜欢 收到 很漂亮 充电 好评 很快 卖家 速度 评价 流畅 快递 物流中评关键词分析，实现代码如下所示：
代码语言：javascript复制#中评关键词
keywords_med = jieba.analyse.extract_tags(''.join(data_cutted['Comment'][data_cutted['Class'] == 0]), topK=20)
for item in keywords_med:
print(item,)中评关键词TOP20如下所示：
充电 不错 发热 外观 感觉 电池 机子 问题 赠品 有点 无线 发烫 换货 软件 快递 安卓 内存 退货 知道 售后差评关键词分析，实现代码如下所示：
代码语言：javascript复制#差评关键词
keywords_neg = jieba.analyse.extract_tags(''.join(data_cutted['Comment'][data_cutted['Class'] == -1]), topK=20)
for item in keywords_neg:
print（item,）差评关键词TOP20如下所示：
差评 售后 垃圾 赠品 退货 问题 换货 充电 降价 发票 充电器 东西 刚买 发热 无线 机子 死机 收到 质量 15经过以上步骤的处理，整个数据集的预处理工作“告一段落”。在中文文本分析和情感分析的工作中，数据预处理的内容主要是分词。只有经过分词处理后的文本数据集才可以进行下一步的向量化操作，满足输入模型的条件。
4、基于SVM的情感分类模型
经过分词之后的文本数据集要先进行向量化之后才能输入到分类模型中进行运算。
我们使用sklearn库实现向量化方法，去掉停用词，并将其通过tf，tf-idf映射到特征空间。
其中，tftf为词频，即分词后每个词项在该条评论中出现的次数；dfdf为出现该词项评论数目；NN为评论总数，使用对数来适当抑制tftf和dfdf值的影响。
我们使用sklearn库中的函数直接实现SVM算法。在这里，我们选取以下形式的SVM模型参与运算。
为了方便，创建文本情感分析类CommentClassifier，来实现建模过程：
__init__为类的初始化函数，输入参数classifier_type和vector_type,分别代表分类模型的类型和向量化方法的类型。fit()函数，来实现向量化与模型建立的过程。
实现代码如下所示：
代码语言：javascript复制# 实现向量化方法
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.feature_extraction.text import CountVectorizer
#实现svm和贝叶斯模型
from sklearn.svm import SVC
from sklearn.svm import LinearSVC
from sklearn.linear_model import SGDClassifier
# 实现交叉验证
from sklearn.cross_validation import train_test_split
from sklearn.cross_validation import cross_val_score
# 实现评价指标
from sklearn import metrics
# 文本情感分类的类：CommentClassifier
class CommentClassifier:
def __init__(self, classifier_type, vector_type):
self.classifier_type = classifier_type #分类器类型：支持向量机或贝叶斯分类
self.vector_type = vector_type #文本向量化模型：01模型,TF模型,TF-IDF模型
def fit(self, train_x, train_y, max_df):
list_text = list(train_x)
#向量化方法：0 - 0/1,1 - TF,2 - TF-IDF
if self.vector_type == 0:
self.vectorizer = CountVectorizer(max_df, stop_words = stopwords, ngram_range=(1, 3)).fit(list_text)
elif self.vector_type == 1:
self.vectorizer = TfidfVectorizer(max_df, stop_words = stopwords, ngram_range=(1, 3), use_idf=False).fit(list_text)
else:
self.vectorizer = TfidfVectorizer(max_df, stop_words = stopwords, ngram_range=(1, 3)).fit(list_text)
self.array_trainx = self.vectorizer.transform(list_text)
self.array_trainy = train_y
#分类模型选择：1 - SVC,2 - LinearSVC,3 - SGDClassifier，三种SVM模型
if self.classifier_type == 1:
self.model = SVC(kernel='linear', gamma=10 ** -5, C=1).fit(self.array_trainx, self.array_trainy)
elif self.classifier_type == 2:
self.model = LinearSVC().fit(self.array_trainx, self.array_trainy)
else:
self.model = SGDClassifier().fit(self.array_trainx, self.array_trainy)
def predict_value(self, test_x):
list_text = list(test_x)
self.array_testx = self.vectorizer.transform(list_text)
array_predict = self.model.predict(self.array_testx)
return array_predict
def predict_proba(self, test_x):
list_text = list(test_x)
self.array_testx = self.vectorizer.transform(list_text)
array_score = self.model.predict_proba(self.array_testx)
return array_score - 使用
train_test_split()函数划分训练集和测试集。训练集：80%；测试集：20%。 - 建立
classifier_type和vector_type两个参数的取值列表，来表示选择的向量化方法以及分类模型 - 输出每种向量化方法和分类模型的组合所对应的分类评价结果，内容包括混淆矩阵以及含
Precision、Recall和F1-score三个指标的评分矩阵
实现代码如下所示：
代码语言：javascript复制#划分训练集，测试集
train_x, test_x, train_y, test_y = train_test_split(data_cutted['Comment'].ravel().astype('U'), data_cutted['Class'].ravel(),
test_size=0.2, random_state=4)
classifier_list = [1,2,3]
vector_list = [0,1,2]
for classifier_type in classifier_list:
for vector_type in vector_list:
commentCls = CommentClassifier(classifier_type, vector_type)
#max_df 设置为0.98
commentCls.fit(train_x, train_y, 0.98)
if classifier_type == 0:
value_result = commentCls.predict_value(test_x)
proba_result = commentCls.predict_proba(test_x)
print(classifier_type,vector_type)
print('classification report')
print(metrics.classification_report(test_y, value_result, labels=[-1, 0, 1]))
print('confusion matrix')
print(metrics.confusion_matrix(test_y, value_result, labels=[-1, 0, 1]))
else:
value_result = commentCls.predict_value(test_x)
print(classifier_type,vector_type)
print('classification report')
print(metrics.classification_report(test_y, value_result, labels=[-1, 0, 1]))
print('confusion matrix')
print(metrics.confusion_matrix(test_y, value_result, labels=[-1, 0, 1]))输出效果如下所示：
代码语言：javascript复制1 0
classification report
precision recall f1-score support
-1 0.68 0.62 0.65 519
0 0.55 0.49 0.52 485
1 0.75 0.86 0.80 634
avg / total 0.67 0.68 0.67 1638
confusion matrix
[[324 130 65]
[131 236 118]
[ 24 64 546]]
1 1
classification report
precision recall f1-score support
-1 0.71 0.74 0.72 519
0 0.58 0.54 0.56 485
1 0.84 0.85 0.85 634
avg / total 0.72 0.72 0.72 1638
confusion matrix
[[385 109 25]
[145 263 77]
[ 15 80 539]]
1 2
classification report
precision recall f1-score support
-1 0.70 0.74 0.72 519
0 0.58 0.52 0.55 485
1 0.84 0.86 0.85 634
avg / total 0.72 0.72 0.72 1638
confusion matrix
[[386 106 27]
[151 254 80]
[ 14 76 544]]
2 0
classification report
precision recall f1-score support
-1 0.70 0.62 0.66 519
0 0.56 0.51 0.54 485
1 0.76 0.88 0.82 634
avg / total 0.68 0.69 0.68 1638
confusion matrix
[[320 135 64]
[122 248 115]
[ 16 57 561]]
2 1
classification report
precision recall f1-score support
-1 0.69 0.73 0.71 519
0 0.61 0.48 0.54 485
1 0.81 0.91 0.86 634
avg / total 0.71 0.73 0.72 1638
confusion matrix
[[377 108 34]
[154 233 98]
[ 12 44 578]]
2 2
classification report
precision recall f1-score support
-1 0.70 0.74 0.72 519
0 0.61 0.50 0.55 485
1 0.83 0.91 0.87 634
avg / total 0.72 0.73 0.73 1638
confusion matrix
[[383 108 28]
[154 241 90]
[ 13 43 578]]
3 0
classification report
precision recall f1-score support
-1 0.69 0.69 0.69 519
0 0.58 0.47 0.52 485
1 0.79 0.90 0.84 634
avg / total 0.70 0.71 0.70 1638
confusion matrix
[[359 118 42]
[148 228 109]
[ 14 47 573]]
3 1
classification report
precision recall f1-score support
-1 0.70 0.74 0.72 519
0 0.60 0.49 0.54 485
1 0.81 0.88 0.84 634
avg / total 0.71 0.72 0.71 1638
confusion matrix
[[386 96 37]
[152 240 93]
[ 13 66 555]]
3 2
classification report
precision recall f1-score support
-1 0.65 0.75 0.69 519
0 0.63 0.49 0.55 485
1 0.83 0.86 0.85 634
avg / total 0.71 0.72 0.71 1638
confusion matrix
[[389 98 32]
[169 236 80]
[ 45 41 548]]从结果上来看，选择tfidf向量化方法，使用LinearSVC模型效果比较好，f1-socre为0.73
从混淆矩阵来看，我们会发现多数的错误分类都出现在中评和差评上。我们可以将原始数据集的中评删除。实现代码如下所示：
代码语言：javascript复制data_bi = data_cutted[data_cutted['Class'] != 0]
data_bi['Class'].value_counts()效果如下所示：
代码语言：javascript复制 1 3042
-1 2658
Name: Class, dtype: int64再次运行分类模型，查看分类结果，如下所示：
代码语言：javascript复制1 0
classification report
precision recall f1-score support
-1 0.90 0.79 0.84 537
1 0.83 0.92 0.87 603
avg / total 0.86 0.86 0.86 1140
confusion matrix
[[425 112]
[ 48 555]]
1 1
classification report
precision recall f1-score support
-1 0.87 0.92 0.90 537
1 0.93 0.88 0.90 603
avg / total 0.90 0.90 0.90 1140
confusion matrix
[[496 41]
[ 71 532]]
1 2
classification report
precision recall f1-score support
-1 0.88 0.93 0.90 537
1 0.93 0.88 0.91 603
avg / total 0.90 0.90 0.90 1140
confusion matrix
[[497 40]
[ 70 533]]
2 0
classification report
precision recall f1-score support
-1 0.90 0.80 0.85 537
1 0.84 0.92 0.88 603
avg / total 0.87 0.86 0.86 1140
confusion matrix
[[431 106]
[ 48 555]]
2 1
classification report
precision recall f1-score support
-1 0.92 0.91 0.91 537
1 0.92 0.93 0.92 603
avg / total 0.92 0.92 0.92 1140
confusion matrix
[[486 51]
[ 43 560]]
2 2
classification report
precision recall f1-score support
-1 0.93 0.91 0.92 537
1 0.92 0.94 0.93 603
avg / total 0.92 0.92 0.92 1140
confusion matrix
[[488 49]
[ 39 564]]
3 0
classification report
precision recall f1-score support
-1 0.92 0.82 0.87 537
1 0.86 0.94 0.90 603
avg / total 0.89 0.88 0.88 1140
confusion matrix
[[443 94]
[ 38 565]]
3 1
classification report
precision recall f1-score support
-1 0.92 0.91 0.91 537
1 0.92 0.93 0.92 603
avg / total 0.92 0.92 0.92 1140
confusion matrix
[[486 51]
[ 41 562]]
3 2
classification report
precision recall f1-score support
-1 0.88 0.93 0.90 537
1 0.93 0.89 0.91 603
avg / total 0.91 0.91 0.91 1140
confusion matrix
[[497 40]
[ 67 536]]删除差评之后，不同组合的分类模型效果均有显著提升。这也说明，分类模型能够有效地将好评区分出来。
数据集中存在标注不准确的问题，主要集中在中评。由于人在评论时，除非有问题否则一般都会打好评，如果打了中评说明对产品有不满意之处，在情感的表达上就会趋向于负向情感，同时评论具有很大主观性，很多中评会将其归为差评，但数据集中却认为是中评。因此，将一条评论分类为好评、中评、差评是不够客观，中评与差评之间的边界很模糊，因此识别率很难提高。
5、基于word2vec中doc2vec的无监督分类模型
开源文本向量化工具word2vec，可以为文本数据寻求更加深层次的特征表示。词语之间可以进行运算:
w2v(woman)-w2v(man) w2v(king)=w2v(queen)
基于word2vec的doc2vec，将每个文档表示为一个向量，并且通过余弦距离可以计算两个文档的相似程度，那么就可以计算一句话和一句极好的好评的距离，以及一句话到极差的差评的距离。
在本案例的数据集中：
- 好评：快 就是 手感 满意 也好 喜欢 也 流畅 很 服务态度 实用 超快 挺快 用着 速度 礼品 也不错 非常好 挺好 感觉 才来 还行 好看 也快 不错的 送了 非常不错 超级 赞 好多东西 很实用 各方面 挺好的 很多 漂亮 配件 还不错 也多 特意 慢 满分 好用 非常漂亮......
- 差评：不多说 上当 差差 刚用 服务差 一点也不 不要 简直 还是去 实体店 大家 保证 不肯 生气 开发票 磨损 后悔 印记 网 什么破 烂烂 左边 失效 太 骗 掉价 走下坡路 不说了 彻底 三星手机 自营 几次 真心 别的 看完 简单说 机会 这是 生气了 触动 缝隙 冲动了 失望......
我们使用第三方库gensim来实现doc2vec模型。
实现代码如下所示：
代码语言：javascript复制import pandas as pd
from gensim.models import Doc2Vec
from gensim.models.doc2vec import TaggedDocument
import logging
logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)
train_x = data_bi['Comment'].ravel()
train_y = data_bi['Class'].ravel()
#为train_x列贴上标签"TRAIN"
def labelizeReviews(reviews, label_type):
labelized = []
for i, v in enumerate(reviews):
label = '%s_%s' % (label_type, i)
labelized.append(TaggedDocument(v.split(" "), [label]))
return labelized
train_x = labelizeReviews(train_x, "TRAIN")
#建立Doc2Vec模型model
size = 300
all_data = []
all_data.extend(train_x)
model = Doc2Vec(min_count=1, window=8, size=size, sample=1e-4, negative=5, hs=0, iter=5, workers=8)
model.build_vocab(all_data)
# 设置迭代次数10
for epoch in range(10):
model.train(train_x)
#建立空列表pos和neg以对相似度计算结果进行存储，计算每个评论和极好评论之间的余弦距离，并存在pos列表中
#计算每个评论和极差评论之间的余弦距离，并存在neg列表中
pos = []
neg = []
for i in range(0,len(train_x)):
pos.append(model.docvecs.similarity("TRAIN_0","TRAIN_{}".format(i)))
neg.append(model.docvecs.similarity("TRAIN_1","TRAIN_{}".format(i)))
#将pos列表和neg列表更新到原始数据文件中，分别表示为字段PosSim和字段NegSim
data_bi[u'PosSim'] = pos
data_bi[u'NegSim'] = neg模型训练过程如下所示：
代码语言：javascript复制2017-05-27 14:30:28,393 : INFO : collecting all words and their counts
2017-05-27 14:30:28,394 : INFO : PROGRESS: at example #0, processed 0 words (0/s), 0 word types, 0 tags
2017-05-27 14:30:28,593 : INFO : collected 10545 word types and 5700 unique tags from a corpus of 5700 examples and 482148 words
2017-05-27 14:30:28,595 : INFO : Loading a fresh vocabulary
2017-05-27 14:30:28,649 : INFO : min_count=1 retains 10545 unique words (100% of original 10545, drops 0)
2017-05-27 14:30:28,650 : INFO : min_count=1 leaves 482148 word corpus (100% of original 482148, drops 0)
2017-05-27 14:30:28,705 : INFO : deleting the raw counts dictionary of 10545 items
2017-05-27 14:30:28,706 : INFO : sample=0.0001 downsamples 217 most-common words
2017-05-27 14:30:28,707 : INFO : downsampling leaves estimated 108356 word corpus (22.5% of prior 482148)
2017-05-27 14:30:28,709 : INFO : estimated required memory for 10545 words and 300 dimensions: 38560500 bytes
2017-05-27 14:30:28,784 : INFO : resetting layer weights
2017-05-27 14:30:29,120 : INFO : training model with 8 workers on 10545 vocabulary and 300 features, using sg=0 hs=0 sample=0.0001 negative=5 window=8
2017-05-27 14:30:29,121 : INFO : expecting 5700 sentences, matching count from corpus used for vocabulary survey
2017-05-27 14:30:30,176 : INFO : PROGRESS: at 10.24% examples, 72316 words/s, in_qsize 15, out_qsize 0
2017-05-27 14:30:31,211 : INFO : PROGRESS: at 29.96% examples, 91057 words/s, in_qsize 16, out_qsize 0
2017-05-27 14:30:32,218 : INFO : PROGRESS: at 66.30% examples, 126742 words/s, in_qsize 15, out_qsize 0
2017-05-27 14:30:33,231 : INFO : PROGRESS: at 86.00% examples, 122698 words/s, in_qsize 15, out_qsize 0
2017-05-27 14:30:33,571 : INFO : worker thread finished; awaiting finish of 7 more threads
2017-05-27 14:30:33,573 : INFO : worker thread finished; awaiting finish of 6 more threads
2017-05-27 14:30:33,605 : INFO : worker thread finished; awaiting finish of 5 more threads
2017-05-27 14:30:33,647 : INFO : worker thread finished; awaiting finish of 4 more threads
2017-05-27 14:30:33,678 : INFO : worker thread finished; awaiting finish of 3 more threads
2017-05-27 14:30:33,696 : INFO : worker thread finished; awaiting finish of 2 more threads
2017-05-27 14:30:33,711 : INFO : worker thread finished; awaiting finish of 1 more threads
2017-05-27 14:30:33,722 : INFO : worker thread finished; awaiting finish of 0 more threads
2017-05-27 14:30:33,724 : INFO : training on 2410740 raw words (570332 effective words) took 4.6s, 124032 effective words/s
2017-05-27 14:30:33,727 : INFO : training model with 8 workers on 10545 vocabulary and 300 features, using sg=0 hs=0 sample=0.0001 negative=5 window=8
2017-05-27 14:30:33,731 : INFO : expecting 5700 sentences, matching count from corpus used for vocabulary survey
2017-05-27 14:30:34,753 : INFO : PROGRESS: at 36.38% examples, 212225 words/s, in_qsize 15, out_qsize 0
2017-05-27 14:30:35,762 : INFO : PROGRESS: at 75.24% examples, 216859 words/s, in_qsize 16, out_qsize 0
2017-05-27 14:30:36,243 : INFO : worker thread finished; awaiting finish of 7 more threads
2017-05-27 14:30:36,244 : INFO : worker thread finished; awaiting finish of 6 more threads
2017-05-27 14:30:36,264 : INFO : worker thread finished; awaiting finish of 5 more threads
2017-05-27 14:30:36,306 : INFO : worker thread finished; awaiting finish of 4 more threads
2017-05-27 14:30:36,311 : INFO : worker thread finished; awaiting finish of 3 more threads
2017-05-27 14:30:36,320 : INFO : worker thread finished; awaiting finish of 2 more threads
2017-05-27 14:30:36,330 : INFO : worker thread finished; awaiting finish of 1 more threads
2017-05-27 14:30:36,336 : INFO : worker thread finished; awaiting finish of 0 more threads
2017-05-27 14:30:36,338 : INFO : training on 2410740 raw words (570008 effective words) took 2.6s, 219523 effective words/s
2017-05-27 14:30:36,339 : INFO : training model with 8 workers on 10545 vocabulary and 300 features, using sg=0 hs=0 sample=0.0001 negative=5 window=8
2017-05-27 14:30:36,341 : INFO : expecting 5700 sentences, matching count from corpus used for vocabulary survey
2017-05-27 14:30:37,353 : INFO : PROGRESS: at 28.23% examples, 177496 words/s, in_qsize 16, out_qsize 0
2017-05-27 14:30:38,372 : INFO : PROGRESS: at 66.30% examples, 193880 words/s, in_qsize 16, out_qsize 0
2017-05-27 14:30:39,061 : INFO : worker thread finished; awaiting finish of 7 more threads
2017-05-27 14:30:39,062 : INFO : worker thread finished; awaiting finish of 6 more threads
2017-05-27 14:30:39,074 : INFO : worker thread finished; awaiting finish of 5 more threads
2017-05-27 14:30:39,115 : INFO : worker thread finished; awaiting finish of 4 more threads
2017-05-27 14:30:39,122 : INFO : worker thread finished; awaiting finish of 3 more threads
2017-05-27 14:30:39,132 : INFO : worker thread finished; awaiting finish of 2 more threads
2017-05-27 14:30:39,147 : INFO : worker thread finished; awaiting finish of 1 more threads
2017-05-27 14:30:39,154 : INFO : worker thread finished; awaiting finish of 0 more threads
2017-05-27 14:30:39,155 : INFO : training on 2410740 raw words (570746 effective words) took 2.8s, 203312 effective words/s
2017-05-27 14:30:39,158 : INFO : training model with 8 workers on 10545 vocabulary and 300 features, using sg=0 hs=0 sample=0.0001 negative=5 window=8
2017-05-27 14:30:39,159 : INFO : expecting 5700 sentences, matching count from corpus used for vocabulary survey
2017-05-27 14:30:40,168 : INFO : PROGRESS: at 37.74% examples, 222816 words/s, in_qsize 16, out_qsize 0
2017-05-27 14:30:41,177 : INFO : PROGRESS: at 77.55% examples, 223202 words/s, in_qsize 16, out_qsize 0
2017-05-27 14:30:41,605 : INFO : worker thread finished; awaiting finish of 7 more threads
2017-05-27 14:30:41,610 : INFO : worker thread finished; awaiting finish of 6 more threads
2017-05-27 14:30:41,614 : INFO : worker thread finished; awaiting finish of 5 more threads
2017-05-27 14:30:41,645 : INFO : worker thread finished; awaiting finish of 4 more threads
2017-05-27 14:30:41,670 : INFO : worker thread finished; awaiting finish of 3 more threads
2017-05-27 14:30:41,674 : INFO : worker thread finished; awaiting finish of 2 more threads
2017-05-27 14:30:41,682 : INFO : worker thread finished; awaiting finish of 1 more threads
2017-05-27 14:30:41,690 : INFO : worker thread finished; awaiting finish of 0 more threads
2017-05-27 14:30:41,692 : INFO : training on 2410740 raw words (569889 effective words) took 2.5s, 225457 effective words/s
2017-05-27 14:30:41,694 : INFO : training model with 8 workers on 10545 vocabulary and 300 features, using sg=0 hs=0 sample=0.0001 negative=5 window=8
2017-05-27 14:30:41,696 : INFO : expecting 5700 sentences, matching count from corpus used for vocabulary survey
2017-05-27 14:30:42,712 : INFO : PROGRESS: at 29.16% examples, 183182 words/s, in_qsize 15, out_qsize 0
2017-05-27 14:30:43,754 : INFO : PROGRESS: at 69.96% examples, 203560 words/s, in_qsize 15, out_qsize 0
2017-05-27 14:30:44,804 : INFO : PROGRESS: at 91.97% examples, 173787 words/s, in_qsize 14, out_qsize 0
2017-05-27 14:30:44,973 : INFO : worker thread finished; awaiting finish of 7 more threads
2017-05-27 14:30:44,989 : INFO : worker thread finished; awaiting finish of 6 more threads
2017-05-27 14:30:45,028 : INFO : worker thread finished; awaiting finish of 5 more threads
2017-05-27 14:30:45,061 : INFO : worker thread finished; awaiting finish of 4 more threads
2017-05-27 14:30:45,097 : INFO : worker thread finished; awaiting finish of 3 more threads
2017-05-27 14:30:45,101 : INFO : worker thread finished; awaiting finish of 2 more threads
2017-05-27 14:30:45,121 : INFO : worker thread finished; awaiting finish of 1 more threads
2017-05-27 14:30:45,125 : INFO : worker thread finished; awaiting finish of 0 more threads
2017-05-27 14:30:45,128 : INFO : training on 2410740 raw words (569903 effective words) took 3.4s, 166370 effective words/s
2017-05-27 14:30:45,131 : INFO : training model with 8 workers on 10545 vocabulary and 300 features, using sg=0 hs=0 sample=0.0001 negative=5 window=8
2017-05-27 14:30:45,132 : INFO : expecting 5700 sentences, matching count from corpus used for vocabulary survey
2017-05-27 14:30:46,152 : INFO : PROGRESS: at 11.26% examples, 79348 words/s, in_qsize 16, out_qsize 0
2017-05-27 14:30:47,153 : INFO : PROGRESS: at 27.52% examples, 85992 words/s, in_qsize 16, out_qsize 0
2017-05-27 14:30:48,166 : INFO : PROGRESS: at 66.47% examples, 130273 words/s, in_qsize 15, out_qsize 0
2017-05-27 14:30:49,061 : INFO : worker thread finished; awaiting finish of 7 more threads
2017-05-27 14:30:49,076 : INFO : worker thread finished; awaiting finish of 6 more threads
2017-05-27 14:30:49,088 : INFO : worker thread finished; awaiting finish of 5 more threads
2017-05-27 14:30:49,123 : INFO : worker thread finished; awaiting finish of 4 more threads
2017-05-27 14:30:49,144 : INFO : worker thread finished; awaiting finish of 3 more threads
2017-05-27 14:30:49,147 : INFO : worker thread finished; awaiting finish of 2 more threads
2017-05-27 14:30:49,152 : INFO : worker thread finished; awaiting finish of 1 more threads
2017-05-27 14:30:49,159 : INFO : worker thread finished; awaiting finish of 0 more threads
2017-05-27 14:30:49,160 : INFO : training on 2410740 raw words (570333 effective words) took 4.0s, 141860 effective words/s
2017-05-27 14:30:49,161 : INFO : training model with 8 workers on 10545 vocabulary and 300 features, using sg=0 hs=0 sample=0.0001 negative=5 window=8
2017-05-27 14:30:49,163 : INFO : expecting 5700 sentences, matching count from corpus used for vocabulary survey
2017-05-27 14:30:50,185 : INFO : PROGRESS: at 31.78% examples, 193530 words/s, in_qsize 15, out_qsize 0
2017-05-27 14:30:51,244 : INFO : PROGRESS: at 48.51% examples, 141817 words/s, in_qsize 15, out_qsize 0
2017-05-27 14:30:52,278 : INFO : PROGRESS: at 69.96% examples, 134399 words/s, in_qsize 16, out_qsize 0
2017-05-27 14:30:52,918 : INFO : worker thread finished; awaiting finish of 7 more threads
2017-05-27 14:30:52,936 : INFO : worker thread finished; awaiting finish of 6 more threads
2017-05-27 14:30:52,945 : INFO : worker thread finished; awaiting finish of 5 more threads
2017-05-27 14:30:52,976 : INFO : worker thread finished; awaiting finish of 4 more threads
2017-05-27 14:30:52,979 : INFO : worker thread finished; awaiting finish of 3 more threads
2017-05-27 14:30:52,984 : INFO : worker thread finished; awaiting finish of 2 more threads
2017-05-27 14:30:52,995 : INFO : worker thread finished; awaiting finish of 1 more threads
2017-05-27 14:30:52,998 : INFO : worker thread finished; awaiting finish of 0 more threads
2017-05-27 14:30:52,999 : INFO : training on 2410740 raw words (570031 effective words) took 3.8s, 148864 effective words/s
2017-05-27 14:30:53,000 : INFO : training model with 8 workers on 10545 vocabulary and 300 features, using sg=0 hs=0 sample=0.0001 negative=5 window=8
2017-05-27 14:30:53,002 : INFO : expecting 5700 sentences, matching count from corpus used for vocabulary survey
2017-05-27 14:30:54,024 : INFO : PROGRESS: at 34.48% examples, 202424 words/s, in_qsize 15, out_qsize 0
2017-05-27 14:30:55,035 : INFO : PROGRESS: at 68.58% examples, 201499 words/s, in_qsize 15, out_qsize 0
2017-05-27 14:30:56,010 : INFO : worker thread finished; awaiting finish of 7 more threads
2017-05-27 14:30:56,017 : INFO : worker thread finished; awaiting finish of 6 more threads
2017-05-27 14:30:56,048 : INFO : PROGRESS: at 96.89% examples, 183861 words/s, in_qsize 5, out_qsize 1
2017-05-27 14:30:56,049 : INFO : worker thread finished; awaiting finish of 5 more threads
2017-05-27 14:30:56,071 : INFO : worker thread finished; awaiting finish of 4 more threads
2017-05-27 14:30:56,084 : INFO : worker thread finished; awaiting finish of 3 more threads
2017-05-27 14:30:56,099 : INFO : worker thread finished; awaiting finish of 2 more threads
2017-05-27 14:30:56,101 : INFO : worker thread finished; awaiting finish of 1 more threads
2017-05-27 14:30:56,104 : INFO : worker thread finished; awaiting finish of 0 more threads
2017-05-27 14:30:56,104 : INFO : training on 2410740 raw words (570328 effective words) took 3.1s, 184129 effective words/s
2017-05-27 14:30:56,105 : INFO : training model with 8 workers on 10545 vocabulary and 300 features, using sg=0 hs=0 sample=0.0001 negative=5 window=8
2017-05-27 14:30:56,107 : INFO : expecting 5700 sentences, matching count from corpus used for vocabulary survey
2017-05-27 14:30:57,134 : INFO : PROGRESS: at 33.13% examples, 197730 words/s, in_qsize 15, out_qsize 0
2017-05-27 14:30:58,140 : INFO : PROGRESS: at 69.96% examples, 206423 words/s, in_qsize 15, out_qsize 0
2017-05-27 14:30:58,876 : INFO : worker thread finished; awaiting finish of 7 more threads
2017-05-27 14:30:58,883 : INFO : worker thread finished; awaiting finish of 6 more threads
2017-05-27 14:30:58,889 : INFO : worker thread finished; awaiting finish of 5 more threads
2017-05-27 14:30:58,937 : INFO : worker thread finished; awaiting finish of 4 more threads
2017-05-27 14:30:58,949 : INFO : worker thread finished; awaiting finish of 3 more threads
2017-05-27 14:30:58,953 : INFO : worker thread finished; awaiting finish of 2 more threads
2017-05-27 14:30:58,960 : INFO : worker thread finished; awaiting finish of 1 more threads
2017-05-27 14:30:58,967 : INFO : worker thread finished; awaiting finish of 0 more threads
2017-05-27 14:30:58,968 : INFO : training on 2410740 raw words (570312 effective words) took 2.9s, 199922 effective words/s
2017-05-27 14:30:58,969 : INFO : training model with 8 workers on 10545 vocabulary and 300 features, using sg=0 hs=0 sample=0.0001 negative=5 window=8
2017-05-27 14:30:58,970 : INFO : expecting 5700 sentences, matching count from corpus used for vocabulary survey
2017-05-27 14:30:59,991 : INFO : PROGRESS: at 32.86% examples, 198045 words/s, in_qsize 16, out_qsize 0
2017-05-27 14:31:00,993 : INFO : PROGRESS: at 68.23% examples, 201443 words/s, in_qsize 16, out_qsize 0
2017-05-27 14:31:01,881 : INFO : worker thread finished; awaiting finish of 7 more threads
2017-05-27 14:31:01,888 : INFO : worker thread finished; awaiting finish of 6 more threads
2017-05-27 14:31:01,907 : INFO : worker thread finished; awaiting finish of 5 more threads
2017-05-27 14:31:01,922 : INFO : worker thread finished; awaiting finish of 4 more threads
2017-05-27 14:31:01,941 : INFO : worker thread finished; awaiting finish of 3 more threads
2017-05-27 14:31:01,948 : INFO : worker thread finished; awaiting finish of 2 more threads
2017-05-27 14:31:01,955 : INFO : worker thread finished; awaiting finish of 1 more threads
2017-05-27 14:31:01,961 : INFO : worker thread finished; awaiting finish of 0 more threads
2017-05-27 14:31:01,962 : INFO : training on 2410740 raw words (570826 effective words) took 3.0s, 191072 effective words/s最后可视化评论分类效果，实现代码如下所示：
代码语言：javascript复制from matplotlib import pyplot as plt
label= data_bi['Class'].ravel()
values = data_bi[['PosSim' , 'NegSim']].values
plt.scatter(values[:,0], values[:,1], c=label, alpha=0.4)
plt.show()效果如下所示：
从上图中可以看到，好评与差评基本上可以通过一条直线区分开（蓝色为差评，红色为好评）
该方法与传统思路完全不同，没有使用词频率，情感词等特征，其优点有：
- 将数据集映射到了极低维度的空间，只有二维
- 一种无监督的学习方法，不需要对原始训练数据进行标注
- 具有普适性，在其他领域也可以用这种方法，只需要先找出该领域极其正和极其负的方法，将其与所有待识别的数据通过doc2vec转化为向量计算距离即可
