OpenSearch/docs/reference/analysis/tokenizers/pattern-tokenizer.asciidoc

265 lines
4.9 KiB
Plaintext
Raw Normal View History

[[analysis-pattern-tokenizer]]
=== Pattern Tokenizer
The `pattern` tokenizer uses a regular expression to either split text into
terms whenever it matches a word separator, or to capture matching text as
terms.
The default pattern is `\W+`, which splits text whenever it encounters
non-word characters.
[float]
=== Example output
[source,js]
---------------------------
POST _analyze
{
"tokenizer": "pattern",
"text": "The foo_bar_size's default is 5."
}
---------------------------
// CONSOLE
/////////////////////
[source,js]
----------------------------
{
"tokens": [
{
"token": "The",
"start_offset": 0,
"end_offset": 3,
"type": "word",
"position": 0
},
{
"token": "foo_bar_size",
"start_offset": 4,
"end_offset": 16,
"type": "word",
"position": 1
},
{
"token": "s",
"start_offset": 17,
"end_offset": 18,
"type": "word",
"position": 2
},
{
"token": "default",
"start_offset": 19,
"end_offset": 26,
"type": "word",
"position": 3
},
{
"token": "is",
"start_offset": 27,
"end_offset": 29,
"type": "word",
"position": 4
},
{
"token": "5",
"start_offset": 30,
"end_offset": 31,
"type": "word",
"position": 5
}
]
}
----------------------------
// TESTRESPONSE
/////////////////////
The above sentence would produce the following terms:
[source,text]
---------------------------
[ The, foo_bar_size, s, default, is, 5 ]
---------------------------
[float]
=== Configuration
The `pattern` tokenizer accepts the following parameters:
[horizontal]
`pattern`::
A http://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html[Java regular expression], defaults to `\W+`.
`flags`::
Java regular expression http://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html#field.summary[flags].
lags should be pipe-separated, eg `"CASE_INSENSITIVE|COMMENTS"`.
`group`::
Which capture group to extract as tokens. Defaults to `-1` (split).
[float]
=== Example configuration
In this example, we configure the `pattern` tokenizer to break text into
tokens when it encounters commas:
[source,js]
----------------------------
PUT my_index
{
"settings": {
"analysis": {
"analyzer": {
"my_analyzer": {
"tokenizer": "my_tokenizer"
}
},
"tokenizer": {
"my_tokenizer": {
"type": "pattern",
"pattern": ","
}
}
}
}
}
POST my_index/_analyze
{
"analyzer": "my_analyzer",
"text": "comma,separated,values"
}
----------------------------
// CONSOLE
/////////////////////
[source,js]
----------------------------
{
"tokens": [
{
"token": "comma",
"start_offset": 0,
"end_offset": 5,
"type": "word",
"position": 0
},
{
"token": "separated",
"start_offset": 6,
"end_offset": 15,
"type": "word",
"position": 1
},
{
"token": "values",
"start_offset": 16,
"end_offset": 22,
"type": "word",
"position": 2
}
]
}
----------------------------
// TESTRESPONSE
/////////////////////
The above example produces the following terms:
[source,text]
---------------------------
[ comma, separated, values ]
---------------------------
In the next example, we configure the `pattern` tokenizer to capture values
enclosed in double quotes (ignoring embedded escaped quotes `\"`). The regex
itself looks like this:
"((?:\\"|[^"]|\\")*)"
And reads as follows:
* A literal `"`
* Start capturing:
** A literal `\"` OR any character except `"`
** Repeat until no more characters match
* A literal closing `"`
When the pattern is specified in JSON, the `"` and `\` characters need to be
escaped, so the pattern ends up looking like:
\"((?:\\\\\"|[^\"]|\\\\\")+)\"
[source,js]
----------------------------
PUT my_index
{
"settings": {
"analysis": {
"analyzer": {
"my_analyzer": {
"tokenizer": "my_tokenizer"
}
},
"tokenizer": {
"my_tokenizer": {
"type": "pattern",
"pattern": "\"((?:\\\\\"|[^\"]|\\\\\")+)\"",
"group": 1
}
}
}
}
}
POST my_index/_analyze
{
"analyzer": "my_analyzer",
"text": "\"value\", \"value with embedded \\\" quote\""
}
----------------------------
// CONSOLE
/////////////////////
[source,js]
----------------------------
{
"tokens": [
{
"token": "value",
"start_offset": 1,
"end_offset": 6,
"type": "word",
"position": 0
},
{
"token": "value with embedded \\\" quote",
"start_offset": 10,
"end_offset": 38,
"type": "word",
"position": 1
}
]
}
----------------------------
// TESTRESPONSE
/////////////////////
The above example produces the following two terms:
[source,text]
---------------------------
[ value, value with embedded \" quote ]
---------------------------