361 lines
8.3 KiB
Plaintext
361 lines
8.3 KiB
Plaintext
[[analysis-edgengram-tokenizer]]
|
|
=== Edge n-gram tokenizer
|
|
++++
|
|
<titleabbrev>Edge n-gram</titleabbrev>
|
|
++++
|
|
|
|
The `edge_ngram` tokenizer first breaks text down into words whenever it
|
|
encounters one of a list of specified characters, then it emits
|
|
{wikipedia}/N-gram[N-grams] of each word where the start of
|
|
the N-gram is anchored to the beginning of the word.
|
|
|
|
Edge N-Grams are useful for _search-as-you-type_ queries.
|
|
|
|
TIP: When you need _search-as-you-type_ for text which has a widely known
|
|
order, such as movie or song titles, the
|
|
<<completion-suggester,completion suggester>> is a much more efficient
|
|
choice than edge N-grams. Edge N-grams have the advantage when trying to
|
|
autocomplete words that can appear in any order.
|
|
|
|
[discrete]
|
|
=== Example output
|
|
|
|
With the default settings, the `edge_ngram` tokenizer treats the initial text as a
|
|
single token and produces N-grams with minimum length `1` and maximum length
|
|
`2`:
|
|
|
|
[source,console]
|
|
---------------------------
|
|
POST _analyze
|
|
{
|
|
"tokenizer": "edge_ngram",
|
|
"text": "Quick Fox"
|
|
}
|
|
---------------------------
|
|
|
|
/////////////////////
|
|
|
|
[source,console-result]
|
|
----------------------------
|
|
{
|
|
"tokens": [
|
|
{
|
|
"token": "Q",
|
|
"start_offset": 0,
|
|
"end_offset": 1,
|
|
"type": "word",
|
|
"position": 0
|
|
},
|
|
{
|
|
"token": "Qu",
|
|
"start_offset": 0,
|
|
"end_offset": 2,
|
|
"type": "word",
|
|
"position": 1
|
|
}
|
|
]
|
|
}
|
|
----------------------------
|
|
|
|
/////////////////////
|
|
|
|
|
|
The above sentence would produce the following terms:
|
|
|
|
[source,text]
|
|
---------------------------
|
|
[ Q, Qu ]
|
|
---------------------------
|
|
|
|
NOTE: These default gram lengths are almost entirely useless. You need to
|
|
configure the `edge_ngram` before using it.
|
|
|
|
[discrete]
|
|
=== Configuration
|
|
|
|
The `edge_ngram` tokenizer accepts the following parameters:
|
|
|
|
`min_gram`::
|
|
Minimum length of characters in a gram. Defaults to `1`.
|
|
|
|
`max_gram`::
|
|
+
|
|
--
|
|
Maximum length of characters in a gram. Defaults to `2`.
|
|
|
|
See <<max-gram-limits>>.
|
|
--
|
|
|
|
`token_chars`::
|
|
|
|
Character classes that should be included in a token. Elasticsearch
|
|
will split on characters that don't belong to the classes specified.
|
|
Defaults to `[]` (keep all characters).
|
|
+
|
|
Character classes may be any of the following:
|
|
+
|
|
* `letter` -- for example `a`, `b`, `ï` or `京`
|
|
* `digit` -- for example `3` or `7`
|
|
* `whitespace` -- for example `" "` or `"\n"`
|
|
* `punctuation` -- for example `!` or `"`
|
|
* `symbol` -- for example `$` or `√`
|
|
* `custom` -- custom characters which need to be set using the
|
|
`custom_token_chars` setting.
|
|
|
|
`custom_token_chars`::
|
|
|
|
Custom characters that should be treated as part of a token. For example,
|
|
setting this to `+-_` will make the tokenizer treat the plus, minus and
|
|
underscore sign as part of a token.
|
|
|
|
[discrete]
|
|
[[max-gram-limits]]
|
|
=== Limitations of the `max_gram` parameter
|
|
|
|
The `edge_ngram` tokenizer's `max_gram` value limits the character length of
|
|
tokens. When the `edge_ngram` tokenizer is used with an index analyzer, this
|
|
means search terms longer than the `max_gram` length may not match any indexed
|
|
terms.
|
|
|
|
For example, if the `max_gram` is `3`, searches for `apple` won't match the
|
|
indexed term `app`.
|
|
|
|
To account for this, you can use the
|
|
<<analysis-truncate-tokenfilter,`truncate`>> token filter with a search analyzer
|
|
to shorten search terms to the `max_gram` character length. However, this could
|
|
return irrelevant results.
|
|
|
|
For example, if the `max_gram` is `3` and search terms are truncated to three
|
|
characters, the search term `apple` is shortened to `app`. This means searches
|
|
for `apple` return any indexed terms matching `app`, such as `apply`, `snapped`,
|
|
and `apple`.
|
|
|
|
We recommend testing both approaches to see which best fits your
|
|
use case and desired search experience.
|
|
|
|
[discrete]
|
|
=== Example configuration
|
|
|
|
In this example, we configure the `edge_ngram` tokenizer to treat letters and
|
|
digits as tokens, and to produce grams with minimum length `2` and maximum
|
|
length `10`:
|
|
|
|
[source,console]
|
|
----------------------------
|
|
PUT my-index-00001
|
|
{
|
|
"settings": {
|
|
"analysis": {
|
|
"analyzer": {
|
|
"my_analyzer": {
|
|
"tokenizer": "my_tokenizer"
|
|
}
|
|
},
|
|
"tokenizer": {
|
|
"my_tokenizer": {
|
|
"type": "edge_ngram",
|
|
"min_gram": 2,
|
|
"max_gram": 10,
|
|
"token_chars": [
|
|
"letter",
|
|
"digit"
|
|
]
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
POST my-index-00001/_analyze
|
|
{
|
|
"analyzer": "my_analyzer",
|
|
"text": "2 Quick Foxes."
|
|
}
|
|
----------------------------
|
|
|
|
/////////////////////
|
|
|
|
[source,console-result]
|
|
----------------------------
|
|
{
|
|
"tokens": [
|
|
{
|
|
"token": "Qu",
|
|
"start_offset": 2,
|
|
"end_offset": 4,
|
|
"type": "word",
|
|
"position": 0
|
|
},
|
|
{
|
|
"token": "Qui",
|
|
"start_offset": 2,
|
|
"end_offset": 5,
|
|
"type": "word",
|
|
"position": 1
|
|
},
|
|
{
|
|
"token": "Quic",
|
|
"start_offset": 2,
|
|
"end_offset": 6,
|
|
"type": "word",
|
|
"position": 2
|
|
},
|
|
{
|
|
"token": "Quick",
|
|
"start_offset": 2,
|
|
"end_offset": 7,
|
|
"type": "word",
|
|
"position": 3
|
|
},
|
|
{
|
|
"token": "Fo",
|
|
"start_offset": 8,
|
|
"end_offset": 10,
|
|
"type": "word",
|
|
"position": 4
|
|
},
|
|
{
|
|
"token": "Fox",
|
|
"start_offset": 8,
|
|
"end_offset": 11,
|
|
"type": "word",
|
|
"position": 5
|
|
},
|
|
{
|
|
"token": "Foxe",
|
|
"start_offset": 8,
|
|
"end_offset": 12,
|
|
"type": "word",
|
|
"position": 6
|
|
},
|
|
{
|
|
"token": "Foxes",
|
|
"start_offset": 8,
|
|
"end_offset": 13,
|
|
"type": "word",
|
|
"position": 7
|
|
}
|
|
]
|
|
}
|
|
----------------------------
|
|
|
|
/////////////////////
|
|
|
|
The above example produces the following terms:
|
|
|
|
[source,text]
|
|
---------------------------
|
|
[ Qu, Qui, Quic, Quick, Fo, Fox, Foxe, Foxes ]
|
|
---------------------------
|
|
|
|
Usually we recommend using the same `analyzer` at index time and at search
|
|
time. In the case of the `edge_ngram` tokenizer, the advice is different. It
|
|
only makes sense to use the `edge_ngram` tokenizer at index time, to ensure
|
|
that partial words are available for matching in the index. At search time,
|
|
just search for the terms the user has typed in, for instance: `Quick Fo`.
|
|
|
|
Below is an example of how to set up a field for _search-as-you-type_.
|
|
|
|
Note that the `max_gram` value for the index analyzer is `10`, which limits
|
|
indexed terms to 10 characters. Search terms are not truncated, meaning that
|
|
search terms longer than 10 characters may not match any indexed terms.
|
|
|
|
[source,console]
|
|
-----------------------------------
|
|
PUT my-index-00001
|
|
{
|
|
"settings": {
|
|
"analysis": {
|
|
"analyzer": {
|
|
"autocomplete": {
|
|
"tokenizer": "autocomplete",
|
|
"filter": [
|
|
"lowercase"
|
|
]
|
|
},
|
|
"autocomplete_search": {
|
|
"tokenizer": "lowercase"
|
|
}
|
|
},
|
|
"tokenizer": {
|
|
"autocomplete": {
|
|
"type": "edge_ngram",
|
|
"min_gram": 2,
|
|
"max_gram": 10,
|
|
"token_chars": [
|
|
"letter"
|
|
]
|
|
}
|
|
}
|
|
}
|
|
},
|
|
"mappings": {
|
|
"properties": {
|
|
"title": {
|
|
"type": "text",
|
|
"analyzer": "autocomplete",
|
|
"search_analyzer": "autocomplete_search"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
PUT my-index-00001/_doc/1
|
|
{
|
|
"title": "Quick Foxes" <1>
|
|
}
|
|
|
|
POST my-index-00001/_refresh
|
|
|
|
GET my-index-00001/_search
|
|
{
|
|
"query": {
|
|
"match": {
|
|
"title": {
|
|
"query": "Quick Fo", <2>
|
|
"operator": "and"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
-----------------------------------
|
|
|
|
<1> The `autocomplete` analyzer indexes the terms `[qu, qui, quic, quick, fo, fox, foxe, foxes]`.
|
|
<2> The `autocomplete_search` analyzer searches for the terms `[quick, fo]`, both of which appear in the index.
|
|
|
|
/////////////////////
|
|
|
|
[source,console-result]
|
|
----------------------------
|
|
{
|
|
"took": $body.took,
|
|
"timed_out": false,
|
|
"_shards": {
|
|
"total": 1,
|
|
"successful": 1,
|
|
"skipped" : 0,
|
|
"failed": 0
|
|
},
|
|
"hits": {
|
|
"total" : {
|
|
"value": 1,
|
|
"relation": "eq"
|
|
},
|
|
"max_score": 0.5753642,
|
|
"hits": [
|
|
{
|
|
"_index": "my-index-00001",
|
|
"_type": "_doc",
|
|
"_id": "1",
|
|
"_score": 0.5753642,
|
|
"_source": {
|
|
"title": "Quick Foxes"
|
|
}
|
|
}
|
|
]
|
|
}
|
|
}
|
|
----------------------------
|
|
// TESTRESPONSE[s/"took".*/"took": "$body.took",/]
|
|
/////////////////////
|