mirror of
https://github.com/honeymoose/OpenSearch.git
synced 2025-02-05 20:48:22 +00:00
25aac4f77f
The "include_type_name" parameter was temporarily introduced in #37285 to facilitate moving the default parameter setting to "false" in many places in the documentation code snippets. Most of the places can simply be reverted without causing errors. In this change I looked for asciidoc files that contained the "include_type_name=true" addition when creating new indices but didn't look likey they made use of the "_doc" type for mappings. This is mostly the case e.g. in the analysis docs where index creating often only contains settings. I manually corrected the use of types in some places where the docs still used an explicit type name and not the dummy "_doc" type.
173 lines
3.4 KiB
Plaintext
173 lines
3.4 KiB
Plaintext
[[analysis-pathhierarchy-tokenizer]]
|
|
=== Path Hierarchy Tokenizer
|
|
|
|
The `path_hierarchy` tokenizer takes a hierarchical value like a filesystem
|
|
path, splits on the path separator, and emits a term for each component in the
|
|
tree.
|
|
|
|
[float]
|
|
=== Example output
|
|
|
|
[source,js]
|
|
---------------------------
|
|
POST _analyze
|
|
{
|
|
"tokenizer": "path_hierarchy",
|
|
"text": "/one/two/three"
|
|
}
|
|
---------------------------
|
|
// CONSOLE
|
|
|
|
/////////////////////
|
|
|
|
[source,js]
|
|
----------------------------
|
|
{
|
|
"tokens": [
|
|
{
|
|
"token": "/one",
|
|
"start_offset": 0,
|
|
"end_offset": 4,
|
|
"type": "word",
|
|
"position": 0
|
|
},
|
|
{
|
|
"token": "/one/two",
|
|
"start_offset": 0,
|
|
"end_offset": 8,
|
|
"type": "word",
|
|
"position": 0
|
|
},
|
|
{
|
|
"token": "/one/two/three",
|
|
"start_offset": 0,
|
|
"end_offset": 14,
|
|
"type": "word",
|
|
"position": 0
|
|
}
|
|
]
|
|
}
|
|
----------------------------
|
|
// TESTRESPONSE
|
|
|
|
/////////////////////
|
|
|
|
|
|
|
|
The above text would produce the following terms:
|
|
|
|
[source,text]
|
|
---------------------------
|
|
[ /one, /one/two, /one/two/three ]
|
|
---------------------------
|
|
|
|
[float]
|
|
=== Configuration
|
|
|
|
The `path_hierarchy` tokenizer accepts the following parameters:
|
|
|
|
[horizontal]
|
|
`delimiter`::
|
|
The character to use as the path separator. Defaults to `/`.
|
|
|
|
`replacement`::
|
|
An optional replacement character to use for the delimiter.
|
|
Defaults to the `delimiter`.
|
|
|
|
`buffer_size`::
|
|
The number of characters read into the term buffer in a single pass.
|
|
Defaults to `1024`. The term buffer will grow by this size until all the
|
|
text has been consumed. It is advisable not to change this setting.
|
|
|
|
`reverse`::
|
|
If set to `true`, emits the tokens in reverse order. Defaults to `false`.
|
|
|
|
`skip`::
|
|
The number of initial tokens to skip. Defaults to `0`.
|
|
|
|
[float]
|
|
=== Example configuration
|
|
|
|
In this example, we configure the `path_hierarchy` tokenizer to split on `-`
|
|
characters, and to replace them with `/`. The first two tokens are skipped:
|
|
|
|
[source,js]
|
|
----------------------------
|
|
PUT my_index
|
|
{
|
|
"settings": {
|
|
"analysis": {
|
|
"analyzer": {
|
|
"my_analyzer": {
|
|
"tokenizer": "my_tokenizer"
|
|
}
|
|
},
|
|
"tokenizer": {
|
|
"my_tokenizer": {
|
|
"type": "path_hierarchy",
|
|
"delimiter": "-",
|
|
"replacement": "/",
|
|
"skip": 2
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
POST my_index/_analyze
|
|
{
|
|
"analyzer": "my_analyzer",
|
|
"text": "one-two-three-four-five"
|
|
}
|
|
----------------------------
|
|
// CONSOLE
|
|
|
|
/////////////////////
|
|
|
|
[source,js]
|
|
----------------------------
|
|
{
|
|
"tokens": [
|
|
{
|
|
"token": "/three",
|
|
"start_offset": 7,
|
|
"end_offset": 13,
|
|
"type": "word",
|
|
"position": 0
|
|
},
|
|
{
|
|
"token": "/three/four",
|
|
"start_offset": 7,
|
|
"end_offset": 18,
|
|
"type": "word",
|
|
"position": 0
|
|
},
|
|
{
|
|
"token": "/three/four/five",
|
|
"start_offset": 7,
|
|
"end_offset": 23,
|
|
"type": "word",
|
|
"position": 0
|
|
}
|
|
]
|
|
}
|
|
----------------------------
|
|
// TESTRESPONSE
|
|
|
|
/////////////////////
|
|
|
|
|
|
The above example produces the following terms:
|
|
|
|
[source,text]
|
|
---------------------------
|
|
[ /three, /three/four, /three/four/five ]
|
|
---------------------------
|
|
|
|
If we were to set `reverse` to `true`, it would produce the following:
|
|
|
|
[source,text]
|
|
---------------------------
|
|
[ one/two/three/, two/three/, three/ ]
|
|
---------------------------
|