[Feature] Adding a char_group tokenizer (#24186)
=== Char Group Tokenizer
The `char_group` tokenizer breaks text into terms whenever it encounters
a
character which is in a defined set. It is mostly useful for cases where
a simple
custom tokenization is desired, and the overhead of use of the
<<analysis-pattern-tokenizer, `pattern` tokenizer>>
is not acceptable.
=== Configuration
The `char_group` tokenizer accepts one parameter:
`tokenize_on_chars`::
A string containing a list of characters to tokenize the string on.
Whenever a character
from this list is encountered, a new token is started. Also supports
escaped values like `\\n` and `\\f`,
and in addition `\\s` to represent whitespace, `\\d` to represent
digits and `\\w` to represent letters.
Defaults to an empty list.
=== Example output
```The 2 QUICK Brown-Foxes jumped over the lazy dog's bone for $2```
When the configuration `\\s-:<>` is used for `tokenize_on_chars`, the
above sentence would produce the following terms:
```[ The, 2, QUICK, Brown, Foxes, jumped, over, the, lazy, dog's, bone,
for, $2 ]```
2018-05-22 10:26:31 -04:00
|
|
|
[[analysis-chargroup-tokenizer]]
|
|
|
|
|
=== Char Group Tokenizer
|
|
|
|
|
|
|
|
|
|
The `char_group` tokenizer breaks text into terms whenever it encounters a
|
|
|
|
|
character which is in a defined set. It is mostly useful for cases where a simple
|
|
|
|
|
custom tokenization is desired, and the overhead of use of the <<analysis-pattern-tokenizer, `pattern` tokenizer>>
|
|
|
|
|
is not acceptable.
|
|
|
|
|
|
|
|
|
|
[float]
|
|
|
|
|
=== Configuration
|
|
|
|
|
|
|
|
|
|
The `char_group` tokenizer accepts one parameter:
|
|
|
|
|
|
|
|
|
|
[horizontal]
|
|
|
|
|
`tokenize_on_chars`::
|
|
|
|
|
A list containing a list of characters to tokenize the string on. Whenever a character
|
|
|
|
|
from this list is encountered, a new token is started. This accepts either single
|
2019-01-07 08:44:12 -05:00
|
|
|
characters like e.g. `-`, or character groups: `whitespace`, `letter`, `digit`,
|
[Feature] Adding a char_group tokenizer (#24186)
=== Char Group Tokenizer
The `char_group` tokenizer breaks text into terms whenever it encounters
a
character which is in a defined set. It is mostly useful for cases where
a simple
custom tokenization is desired, and the overhead of use of the
<<analysis-pattern-tokenizer, `pattern` tokenizer>>
is not acceptable.
=== Configuration
The `char_group` tokenizer accepts one parameter:
`tokenize_on_chars`::
A string containing a list of characters to tokenize the string on.
Whenever a character
from this list is encountered, a new token is started. Also supports
escaped values like `\\n` and `\\f`,
and in addition `\\s` to represent whitespace, `\\d` to represent
digits and `\\w` to represent letters.
Defaults to an empty list.
=== Example output
```The 2 QUICK Brown-Foxes jumped over the lazy dog's bone for $2```
When the configuration `\\s-:<>` is used for `tokenize_on_chars`, the
above sentence would produce the following terms:
```[ The, 2, QUICK, Brown, Foxes, jumped, over, the, lazy, dog's, bone,
for, $2 ]```
2018-05-22 10:26:31 -04:00
|
|
|
`punctuation`, `symbol`.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
[float]
|
|
|
|
|
=== Example output
|
|
|
|
|
|
2019-09-09 13:38:14 -04:00
|
|
|
[source,console]
|
[Feature] Adding a char_group tokenizer (#24186)
=== Char Group Tokenizer
The `char_group` tokenizer breaks text into terms whenever it encounters
a
character which is in a defined set. It is mostly useful for cases where
a simple
custom tokenization is desired, and the overhead of use of the
<<analysis-pattern-tokenizer, `pattern` tokenizer>>
is not acceptable.
=== Configuration
The `char_group` tokenizer accepts one parameter:
`tokenize_on_chars`::
A string containing a list of characters to tokenize the string on.
Whenever a character
from this list is encountered, a new token is started. Also supports
escaped values like `\\n` and `\\f`,
and in addition `\\s` to represent whitespace, `\\d` to represent
digits and `\\w` to represent letters.
Defaults to an empty list.
=== Example output
```The 2 QUICK Brown-Foxes jumped over the lazy dog's bone for $2```
When the configuration `\\s-:<>` is used for `tokenize_on_chars`, the
above sentence would produce the following terms:
```[ The, 2, QUICK, Brown, Foxes, jumped, over, the, lazy, dog's, bone,
for, $2 ]```
2018-05-22 10:26:31 -04:00
|
|
|
---------------------------
|
|
|
|
|
POST _analyze
|
|
|
|
|
{
|
|
|
|
|
"tokenizer": {
|
|
|
|
|
"type": "char_group",
|
|
|
|
|
"tokenize_on_chars": [
|
|
|
|
|
"whitespace",
|
|
|
|
|
"-",
|
|
|
|
|
"\n"
|
|
|
|
|
]
|
|
|
|
|
},
|
|
|
|
|
"text": "The QUICK brown-fox"
|
|
|
|
|
}
|
|
|
|
|
---------------------------
|
|
|
|
|
|
|
|
|
|
returns
|
|
|
|
|
|
2019-09-06 09:22:08 -04:00
|
|
|
[source,console-result]
|
[Feature] Adding a char_group tokenizer (#24186)
=== Char Group Tokenizer
The `char_group` tokenizer breaks text into terms whenever it encounters
a
character which is in a defined set. It is mostly useful for cases where
a simple
custom tokenization is desired, and the overhead of use of the
<<analysis-pattern-tokenizer, `pattern` tokenizer>>
is not acceptable.
=== Configuration
The `char_group` tokenizer accepts one parameter:
`tokenize_on_chars`::
A string containing a list of characters to tokenize the string on.
Whenever a character
from this list is encountered, a new token is started. Also supports
escaped values like `\\n` and `\\f`,
and in addition `\\s` to represent whitespace, `\\d` to represent
digits and `\\w` to represent letters.
Defaults to an empty list.
=== Example output
```The 2 QUICK Brown-Foxes jumped over the lazy dog's bone for $2```
When the configuration `\\s-:<>` is used for `tokenize_on_chars`, the
above sentence would produce the following terms:
```[ The, 2, QUICK, Brown, Foxes, jumped, over, the, lazy, dog's, bone,
for, $2 ]```
2018-05-22 10:26:31 -04:00
|
|
|
---------------------------
|
|
|
|
|
{
|
|
|
|
|
"tokens": [
|
|
|
|
|
{
|
|
|
|
|
"token": "The",
|
|
|
|
|
"start_offset": 0,
|
|
|
|
|
"end_offset": 3,
|
|
|
|
|
"type": "word",
|
|
|
|
|
"position": 0
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"token": "QUICK",
|
|
|
|
|
"start_offset": 4,
|
|
|
|
|
"end_offset": 9,
|
|
|
|
|
"type": "word",
|
|
|
|
|
"position": 1
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"token": "brown",
|
|
|
|
|
"start_offset": 10,
|
|
|
|
|
"end_offset": 15,
|
|
|
|
|
"type": "word",
|
|
|
|
|
"position": 2
|
|
|
|
|
},
|
|
|
|
|
{
|
|
|
|
|
"token": "fox",
|
|
|
|
|
"start_offset": 16,
|
|
|
|
|
"end_offset": 19,
|
|
|
|
|
"type": "word",
|
|
|
|
|
"position": 3
|
|
|
|
|
}
|
|
|
|
|
]
|
|
|
|
|
}
|
|
|
|
|
---------------------------
|
