honeymoose
/
OpenSearch
mirror of https://github.com/honeymoose/OpenSearch.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
OpenSearch/docs/reference/analysis/analyzers/pattern-analyzer.asciidoc

416 lines
8.9 KiB
Plaintext
Raw Blame History

[[analysis-pattern-analyzer]]
=== Pattern Analyzer
The `pattern` analyzer uses a regular expression to split the text into terms.
The regular expression should match the *token separators* not the tokens
themselves. The regular expression defaults to `\W+` (or all non-word characters).
[WARNING]
.Beware of Pathological Regular Expressions
========================================
The pattern analyzer uses
http://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html[Java Regular Expressions].
A badly written regular expression could run very slowly or even throw a
StackOverflowError and cause the node it is running on to exit suddenly.
Read more about http://www.regular-expressions.info/catastrophic.html[pathological regular expressions and how to avoid them].
========================================
[float]
=== Example output
[source,js]
---------------------------
POST _analyze
{
"analyzer": "pattern",
"text": "The 2 QUICK Brown-Foxes jumped over the lazy dog's bone."
}
---------------------------
// CONSOLE
/////////////////////
[source,js]
----------------------------
{
"tokens": [
{
"token": "the",
"start_offset": 0,
"end_offset": 3,
"type": "word",
"position": 0
},
{
"token": "2",
"start_offset": 4,
"end_offset": 5,
"type": "word",
"position": 1
},
{
"token": "quick",
"start_offset": 6,
"end_offset": 11,
"type": "word",
"position": 2
},
{
"token": "brown",
"start_offset": 12,
"end_offset": 17,
"type": "word",
"position": 3
},
{
"token": "foxes",
"start_offset": 18,
"end_offset": 23,
"type": "word",
"position": 4
},
{
"token": "jumped",
"start_offset": 24,
"end_offset": 30,
"type": "word",
"position": 5
},
{
"token": "over",
"start_offset": 31,
"end_offset": 35,
"type": "word",
"position": 6
},
{
"token": "the",
"start_offset": 36,
"end_offset": 39,
"type": "word",
"position": 7
},
{
"token": "lazy",
"start_offset": 40,
"end_offset": 44,
"type": "word",
"position": 8
},
{
"token": "dog",
"start_offset": 45,
"end_offset": 48,
"type": "word",
"position": 9
},
{
"token": "s",
"start_offset": 49,
"end_offset": 50,
"type": "word",
"position": 10
},
{
"token": "bone",
"start_offset": 51,
"end_offset": 55,
"type": "word",
"position": 11
}
]
}
----------------------------
// TESTRESPONSE
/////////////////////
The above sentence would produce the following terms:
[source,text]
---------------------------
[ the, 2, quick, brown, foxes, jumped, over, the, lazy, dog, s, bone ]
---------------------------
[float]
=== Configuration
The `pattern` analyzer accepts the following parameters:
[horizontal]
`pattern`::
A http://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html[Java regular expression], defaults to `\W+`.
`flags`::
Java regular expression http://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html#field.summary[flags].
Flags should be pipe-separated, eg `"CASE_INSENSITIVE|COMMENTS"`.
`lowercase`::
Should terms be lowercased or not. Defaults to `true`.
`stopwords`::
A pre-defined stop words list like `_english_` or an array containing a
list of stop words. Defaults to `\_none_`.
`stopwords_path`::
The path to a file containing stop words.
See the <<analysis-stop-tokenfilter,Stop Token Filter>> for more information
about stop word configuration.
[float]
=== Example configuration
In this example, we configure the `pattern` analyzer to split email addresses
on non-word characters or on underscores (`\W|_`), and to lower-case the result:
[source,js]
----------------------------
PUT my_index
{
"settings": {
"analysis": {
"analyzer": {
"my_email_analyzer": {
"type": "pattern",
"pattern": "\\W|_", <1>
"lowercase": true
}
}
}
}
}
POST my_index/_analyze
{
"analyzer": "my_email_analyzer",
"text": "John_Smith@foo-bar.com"
}
----------------------------
// CONSOLE
<1> The backslashes in the pattern need to be escaped when specifying the
pattern as a JSON string.
/////////////////////
[source,js]
----------------------------
{
"tokens": [
{
"token": "john",
"start_offset": 0,
"end_offset": 4,
"type": "word",
"position": 0
},
{
"token": "smith",
"start_offset": 5,
"end_offset": 10,
"type": "word",
"position": 1
},
{
"token": "foo",
"start_offset": 11,
"end_offset": 14,
"type": "word",
"position": 2
},
{
"token": "bar",
"start_offset": 15,
"end_offset": 18,
"type": "word",
"position": 3
},
{
"token": "com",
"start_offset": 19,
"end_offset": 22,
"type": "word",
"position": 4
}
]
}
----------------------------
// TESTRESPONSE
/////////////////////
The above example produces the following terms:
[source,text]
---------------------------
[ john, smith, foo, bar, com ]
---------------------------
[float]
==== CamelCase tokenizer
The following more complicated example splits CamelCase text into tokens:
[source,js]
--------------------------------------------------
PUT my_index
{
"settings": {
"analysis": {
"analyzer": {
"camel": {
"type": "pattern",
"pattern": "([^\\p{L}\\d]+)|(?<=\\D)(?=\\d)|(?<=\\d)(?=\\D)|(?<=[\\p{L}&&[^\\p{Lu}]])(?=\\p{Lu})|(?<=\\p{Lu})(?=\\p{Lu}[\\p{L}&&[^\\p{Lu}]])"
}
}
}
}
}
GET my_index/_analyze
{
"analyzer": "camel",
"text": "MooseX::FTPClass2_beta"
}
--------------------------------------------------
// CONSOLE
/////////////////////
[source,js]
----------------------------
{
"tokens": [
{
"token": "moose",
"start_offset": 0,
"end_offset": 5,
"type": "word",
"position": 0
},
{
"token": "x",
"start_offset": 5,
"end_offset": 6,
"type": "word",
"position": 1
},
{
"token": "ftp",
"start_offset": 8,
"end_offset": 11,
"type": "word",
"position": 2
},
{
"token": "class",
"start_offset": 11,
"end_offset": 16,
"type": "word",
"position": 3
},
{
"token": "2",
"start_offset": 16,
"end_offset": 17,
"type": "word",
"position": 4
},
{
"token": "beta",
"start_offset": 18,
"end_offset": 22,
"type": "word",
"position": 5
}
]
}
----------------------------
// TESTRESPONSE
/////////////////////
The above example produces the following terms:
[source,text]
---------------------------
[ moose, x, ftp, class, 2, beta ]
---------------------------
The regex above is easier to understand as:
[source,regex]
--------------------------------------------------
([^\p{L}\d]+) # swallow non letters and numbers,
| (?<=\D)(?=\d) # or non-number followed by number,
| (?<=\d)(?=\D) # or number followed by non-number,
| (?<=[ \p{L} && [^\p{Lu}]]) # or lower case
(?=\p{Lu}) # followed by upper case,
| (?<=\p{Lu}) # or upper case
(?=\p{Lu} # followed by upper case
[\p{L}&&[^\p{Lu}]] # then lower case
)
--------------------------------------------------
[float]
=== Definition
The `pattern` anlayzer consists of:
Tokenizer::
* <<analysis-pattern-tokenizer,Pattern Tokenizer>>
Token Filters::
* <<analysis-lowercase-tokenfilter,Lower Case Token Filter>>
* <<analysis-stop-tokenfilter,Stop Token Filter>> (disabled by default)
If you need to customize the `pattern` analyzer beyond the configuration
parameters then you need to recreate it as a `custom` analyzer and modify
it, usually by adding token filters. This would recreate the built-in
`pattern` analyzer and you can use it as a starting point for further
customization:
[source,js]
----------------------------------------------------
PUT /pattern_example
{
"settings": {
"analysis": {
"tokenizer": {
"split_on_non_word": {
"type": "pattern",
"pattern": "\\W+" <1>
}
},
"analyzer": {
"rebuilt_pattern": {
"tokenizer": "split_on_non_word",
"filter": [
"lowercase" <2>
]
}
}
}
}
}
----------------------------------------------------
// CONSOLE
// TEST[s/\n$/\nstartyaml\n - compare_analyzers: {index: pattern_example, first: pattern, second: rebuilt_pattern}\nendyaml\n/]
<1> The default pattern is `\W+` which splits on non-word characters
and this is where you'd change it.
<2> You'd add other token filters after `lowercase`.
Reference in New Issue View Git Blame Copy Permalink