mirror of
https://github.com/honeymoose/OpenSearch.git
synced 2025-02-06 04:58:50 +00:00
25aac4f77f
The "include_type_name" parameter was temporarily introduced in #37285 to facilitate moving the default parameter setting to "false" in many places in the documentation code snippets. Most of the places can simply be reverted without causing errors. In this change I looked for asciidoc files that contained the "include_type_name=true" addition when creating new indices but didn't look likey they made use of the "_doc" type for mappings. This is mostly the case e.g. in the analysis docs where index creating often only contains settings. I manually corrected the use of types in some places where the docs still used an explicit type name and not the dummy "_doc" type.
265 lines
5.3 KiB
Plaintext
265 lines
5.3 KiB
Plaintext
[[analysis-classic-tokenizer]]
|
|
=== Classic Tokenizer
|
|
|
|
The `classic` tokenizer is a grammar based tokenizer that is good for English
|
|
language documents. This tokenizer has heuristics for special treatment of
|
|
acronyms, company names, email addresses, and internet host names. However,
|
|
these rules don't always work, and the tokenizer doesn't work well for most
|
|
languages other than English:
|
|
|
|
* It splits words at most punctuation characters, removing punctuation. However, a
|
|
dot that's not followed by whitespace is considered part of a token.
|
|
|
|
* It splits words at hyphens, unless there's a number in the token, in which case
|
|
the whole token is interpreted as a product number and is not split.
|
|
|
|
* It recognizes email addresses and internet hostnames as one token.
|
|
|
|
[float]
|
|
=== Example output
|
|
|
|
[source,js]
|
|
---------------------------
|
|
POST _analyze
|
|
{
|
|
"tokenizer": "classic",
|
|
"text": "The 2 QUICK Brown-Foxes jumped over the lazy dog's bone."
|
|
}
|
|
---------------------------
|
|
// CONSOLE
|
|
|
|
/////////////////////
|
|
|
|
[source,js]
|
|
----------------------------
|
|
{
|
|
"tokens": [
|
|
{
|
|
"token": "The",
|
|
"start_offset": 0,
|
|
"end_offset": 3,
|
|
"type": "<ALPHANUM>",
|
|
"position": 0
|
|
},
|
|
{
|
|
"token": "2",
|
|
"start_offset": 4,
|
|
"end_offset": 5,
|
|
"type": "<ALPHANUM>",
|
|
"position": 1
|
|
},
|
|
{
|
|
"token": "QUICK",
|
|
"start_offset": 6,
|
|
"end_offset": 11,
|
|
"type": "<ALPHANUM>",
|
|
"position": 2
|
|
},
|
|
{
|
|
"token": "Brown",
|
|
"start_offset": 12,
|
|
"end_offset": 17,
|
|
"type": "<ALPHANUM>",
|
|
"position": 3
|
|
},
|
|
{
|
|
"token": "Foxes",
|
|
"start_offset": 18,
|
|
"end_offset": 23,
|
|
"type": "<ALPHANUM>",
|
|
"position": 4
|
|
},
|
|
{
|
|
"token": "jumped",
|
|
"start_offset": 24,
|
|
"end_offset": 30,
|
|
"type": "<ALPHANUM>",
|
|
"position": 5
|
|
},
|
|
{
|
|
"token": "over",
|
|
"start_offset": 31,
|
|
"end_offset": 35,
|
|
"type": "<ALPHANUM>",
|
|
"position": 6
|
|
},
|
|
{
|
|
"token": "the",
|
|
"start_offset": 36,
|
|
"end_offset": 39,
|
|
"type": "<ALPHANUM>",
|
|
"position": 7
|
|
},
|
|
{
|
|
"token": "lazy",
|
|
"start_offset": 40,
|
|
"end_offset": 44,
|
|
"type": "<ALPHANUM>",
|
|
"position": 8
|
|
},
|
|
{
|
|
"token": "dog's",
|
|
"start_offset": 45,
|
|
"end_offset": 50,
|
|
"type": "<APOSTROPHE>",
|
|
"position": 9
|
|
},
|
|
{
|
|
"token": "bone",
|
|
"start_offset": 51,
|
|
"end_offset": 55,
|
|
"type": "<ALPHANUM>",
|
|
"position": 10
|
|
}
|
|
]
|
|
}
|
|
----------------------------
|
|
// TESTRESPONSE
|
|
|
|
/////////////////////
|
|
|
|
|
|
The above sentence would produce the following terms:
|
|
|
|
[source,text]
|
|
---------------------------
|
|
[ The, 2, QUICK, Brown, Foxes, jumped, over, the, lazy, dog's, bone ]
|
|
---------------------------
|
|
|
|
[float]
|
|
=== Configuration
|
|
|
|
The `classic` tokenizer accepts the following parameters:
|
|
|
|
[horizontal]
|
|
`max_token_length`::
|
|
|
|
The maximum token length. If a token is seen that exceeds this length then
|
|
it is split at `max_token_length` intervals. Defaults to `255`.
|
|
|
|
[float]
|
|
=== Example configuration
|
|
|
|
In this example, we configure the `classic` tokenizer to have a
|
|
`max_token_length` of 5 (for demonstration purposes):
|
|
|
|
[source,js]
|
|
----------------------------
|
|
PUT my_index
|
|
{
|
|
"settings": {
|
|
"analysis": {
|
|
"analyzer": {
|
|
"my_analyzer": {
|
|
"tokenizer": "my_tokenizer"
|
|
}
|
|
},
|
|
"tokenizer": {
|
|
"my_tokenizer": {
|
|
"type": "classic",
|
|
"max_token_length": 5
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
POST my_index/_analyze
|
|
{
|
|
"analyzer": "my_analyzer",
|
|
"text": "The 2 QUICK Brown-Foxes jumped over the lazy dog's bone."
|
|
}
|
|
----------------------------
|
|
// CONSOLE
|
|
|
|
/////////////////////
|
|
|
|
[source,js]
|
|
----------------------------
|
|
{
|
|
"tokens": [
|
|
{
|
|
"token": "The",
|
|
"start_offset": 0,
|
|
"end_offset": 3,
|
|
"type": "<ALPHANUM>",
|
|
"position": 0
|
|
},
|
|
{
|
|
"token": "2",
|
|
"start_offset": 4,
|
|
"end_offset": 5,
|
|
"type": "<ALPHANUM>",
|
|
"position": 1
|
|
},
|
|
{
|
|
"token": "QUICK",
|
|
"start_offset": 6,
|
|
"end_offset": 11,
|
|
"type": "<ALPHANUM>",
|
|
"position": 2
|
|
},
|
|
{
|
|
"token": "Brown",
|
|
"start_offset": 12,
|
|
"end_offset": 17,
|
|
"type": "<ALPHANUM>",
|
|
"position": 3
|
|
},
|
|
{
|
|
"token": "Foxes",
|
|
"start_offset": 18,
|
|
"end_offset": 23,
|
|
"type": "<ALPHANUM>",
|
|
"position": 4
|
|
},
|
|
{
|
|
"token": "over",
|
|
"start_offset": 31,
|
|
"end_offset": 35,
|
|
"type": "<ALPHANUM>",
|
|
"position": 6
|
|
},
|
|
{
|
|
"token": "the",
|
|
"start_offset": 36,
|
|
"end_offset": 39,
|
|
"type": "<ALPHANUM>",
|
|
"position": 7
|
|
},
|
|
{
|
|
"token": "lazy",
|
|
"start_offset": 40,
|
|
"end_offset": 44,
|
|
"type": "<ALPHANUM>",
|
|
"position": 8
|
|
},
|
|
{
|
|
"token": "dog's",
|
|
"start_offset": 45,
|
|
"end_offset": 50,
|
|
"type": "<APOSTROPHE>",
|
|
"position": 9
|
|
},
|
|
{
|
|
"token": "bone",
|
|
"start_offset": 51,
|
|
"end_offset": 55,
|
|
"type": "<ALPHANUM>",
|
|
"position": 10
|
|
}
|
|
]
|
|
}
|
|
----------------------------
|
|
// TESTRESPONSE
|
|
|
|
/////////////////////
|
|
|
|
|
|
The above example produces the following terms:
|
|
|
|
[source,text]
|
|
---------------------------
|
|
[ The, 2, QUICK, Brown, Foxes, jumpe, d, over, the, lazy, dog's, bone ]
|
|
---------------------------
|