文章目录
- 概述
- 官网
- 什么是ngram
- 什么是edge ngram
- ngram和index-time搜索推荐原理
- 例子
概述
继续跟中华石杉老师学习ES，第23篇
课程地址： https://www.roncoo.com/view/55
官网
NGram Tokenizer： https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-ngram-tokenizer.html
NGram Token Filter: https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-ngram-tokenfilter.html
Edge NGram Tokenizer: https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-edgengram-tokenizer.html
Edge NGram Token Filter： https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-edgengram-tokenfilter.html
什么是ngram
什么是ngram
假设有个单词quick，5种长度下的ngram
代码语言：javascript复制ngram length=1，会被拆成 q u i c k
ngram length=2，会被拆成 qu ui ic ck
ngram length=3，会被拆成 qui uic ick
ngram length=4，会被拆成 quic uick
ngram length=5，会被拆成 quick其中任意一个被拆分的部分 就被称为ngram 。
什么是edge ngram
quick，anchor首字母后进行ngram
代码语言：javascript复制q
qu
qui
quic
quick上述拆分方式就被称为edge ngram
使用edge ngram将每个单词都进行进一步的分词切分，用切分后的ngram来实现前缀搜索推荐功能
举个例子 两个doc doc1 hello world doc2 hello we
使用edge ngram拆分
h he hel hell hello -------> 可以匹配 doc1,doc2
w -------> 可以匹配 doc1,doc2 wo wor worl world e ---------> 可以匹配 doc2
使用hello w去搜索
hello --> hello，doc1 w --> w，doc1
doc1中hello和w，而且position也匹配，所以，ok，doc1返回，hello world
ngram和index-time搜索推荐原理
搜索的时候，不用再根据一个前缀，然后扫描整个倒排索引了，而是简单的拿前缀去倒排索引中匹配即可，如果匹配上了，那么就好了，就和match query全文检索一样
例子
代码语言：javascript复制PUT /my_index
{
"settings": {
"analysis": {
"filter": {
"autocomplete_filter": {
"type": "edge_ngram",
"min_gram": 1,
"max_gram": 20
}
},
"analyzer": {
"autocomplete": {
"type": "custom",
"tokenizer": "standard",
"filter": [
"lowercase",
"autocomplete_filter"
]
}
}
}
}
}helloworld 设置
代码语言：javascript复制min ngram = 1
max ngram = 3使用edge_ngram ，则会被拆分为一下 ,
代码语言：javascript复制h
he
hel
知识点： autocomplete
https://www.elastic.co/guide/en/elasticsearch/reference/current/search-analyzer.html
GET /my_index/_analyze
{
"analyzer": "autocomplete",
"text": "helll world"
}
设置mapping , 查询的时候还是使用standard
代码语言：javascript复制PUT /my_index/_mapping/my_type
{
"properties": {
"title": {
"type": "text",
"analyzer": "autocomplete",
"search_analyzer": "standard"
}
}
}造数据
代码语言：javascript复制PUT /my_index/my_type/1
{
"content":"hello Jack"
}
PUT /my_index/my_type/2
{
"content":"hello John"
}
PUT /my_index/my_type/3
{
"content":"hello Jose"
}查询
代码语言：javascript复制GET /my_index/my_type/_search
{
"query": {
"match": {
"content": "hello J"
}
}
}返回：
代码语言：javascript复制{
"took": 7,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 3,
"max_score": 0.2876821,
"hits": [
{
"_index": "my_index",
"_type": "my_type",
"_id": "2",
"_score": 0.2876821,
"_source": {
"content": "hello John"
}
},
{
"_index": "my_index",
"_type": "my_type",
"_id": "1",
"_score": 0.2876821,
"_source": {
"content": "hello Jack"
}
},
{
"_index": "my_index",
"_type": "my_type",
"_id": "3",
"_score": 0.2876821,
"_source": {
"content": "hello Jose"
}
}
]
}
}- 如果用match，只有hello的也会出来，全文检索，只是分数比较低
- 推荐使用match_phrase，要求每个term都有，而且position刚好靠着1位，符合我们的期望的
