mirror of
https://github.com/honeymoose/OpenSearch.git
synced 2025-02-05 20:48:22 +00:00
25aac4f77f
The "include_type_name" parameter was temporarily introduced in #37285 to facilitate moving the default parameter setting to "false" in many places in the documentation code snippets. Most of the places can simply be reverted without causing errors. In this change I looked for asciidoc files that contained the "include_type_name=true" addition when creating new indices but didn't look likey they made use of the "_doc" type for mappings. This is mostly the case e.g. in the analysis docs where index creating often only contains settings. I manually corrected the use of types in some places where the docs still used an explicit type name and not the dummy "_doc" type.
106 lines
2.5 KiB
Plaintext
106 lines
2.5 KiB
Plaintext
[[analysis-simplepattern-tokenizer]]
|
|
=== Simple Pattern Tokenizer
|
|
|
|
experimental[This functionality is marked as experimental in Lucene]
|
|
|
|
The `simple_pattern` tokenizer uses a regular expression to capture matching
|
|
text as terms. The set of regular expression features it supports is more
|
|
limited than the <<analysis-pattern-tokenizer,`pattern`>> tokenizer, but the
|
|
tokenization is generally faster.
|
|
|
|
This tokenizer does not support splitting the input on a pattern match, unlike
|
|
the <<analysis-pattern-tokenizer,`pattern`>> tokenizer. To split on pattern
|
|
matches using the same restricted regular expression subset, see the
|
|
<<analysis-simplepatternsplit-tokenizer,`simple_pattern_split`>> tokenizer.
|
|
|
|
This tokenizer uses {lucene-core-javadoc}/org/apache/lucene/util/automaton/RegExp.html[Lucene regular expressions].
|
|
For an explanation of the supported features and syntax, see <<regexp-syntax,Regular Expression Syntax>>.
|
|
|
|
The default pattern is the empty string, which produces no terms. This
|
|
tokenizer should always be configured with a non-default pattern.
|
|
|
|
[float]
|
|
=== Configuration
|
|
|
|
The `simple_pattern` tokenizer accepts the following parameters:
|
|
|
|
[horizontal]
|
|
`pattern`::
|
|
{lucene-core-javadoc}/org/apache/lucene/util/automaton/RegExp.html[Lucene regular expression], defaults to the empty string.
|
|
|
|
[float]
|
|
=== Example configuration
|
|
|
|
This example configures the `simple_pattern` tokenizer to produce terms that are
|
|
three-digit numbers
|
|
|
|
[source,js]
|
|
----------------------------
|
|
PUT my_index
|
|
{
|
|
"settings": {
|
|
"analysis": {
|
|
"analyzer": {
|
|
"my_analyzer": {
|
|
"tokenizer": "my_tokenizer"
|
|
}
|
|
},
|
|
"tokenizer": {
|
|
"my_tokenizer": {
|
|
"type": "simple_pattern",
|
|
"pattern": "[0123456789]{3}"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
POST my_index/_analyze
|
|
{
|
|
"analyzer": "my_analyzer",
|
|
"text": "fd-786-335-514-x"
|
|
}
|
|
----------------------------
|
|
// CONSOLE
|
|
|
|
/////////////////////
|
|
|
|
[source,js]
|
|
----------------------------
|
|
{
|
|
"tokens" : [
|
|
{
|
|
"token" : "786",
|
|
"start_offset" : 3,
|
|
"end_offset" : 6,
|
|
"type" : "word",
|
|
"position" : 0
|
|
},
|
|
{
|
|
"token" : "335",
|
|
"start_offset" : 7,
|
|
"end_offset" : 10,
|
|
"type" : "word",
|
|
"position" : 1
|
|
},
|
|
{
|
|
"token" : "514",
|
|
"start_offset" : 11,
|
|
"end_offset" : 14,
|
|
"type" : "word",
|
|
"position" : 2
|
|
}
|
|
]
|
|
}
|
|
----------------------------
|
|
// TESTRESPONSE
|
|
|
|
/////////////////////
|
|
|
|
The above example produces these terms:
|
|
|
|
[source,text]
|
|
---------------------------
|
|
[ 786, 335, 514 ]
|
|
---------------------------
|