22 lines
838 B
Plaintext
22 lines
838 B
Plaintext
|
[[analysis-classic-tokenizer]]
|
||
|
|
=== Classic Tokenizer
|
||
|
|
|
||
|
|
coming[1.3.0]
|
||
|
|
|
||
|
|
A tokenizer of type `classic` providing grammar based tokenizer that is
|
||
|
|
a good tokenizer for English language documents. This tokenizer has
|
||
|
|
heuristics for special treatment of acronyms, company names, email addresses,
|
||
|
|
and internet host names. However, these rules don't always work, and
|
||
|
|
the tokenizer doesn't work well for most languages other than English.
|
||
|
|
|
||
|
|
The following are settings that can be set for a `classic` tokenizer
|
||
|
|
type:
|
||
|
|
|
||
|
|
[cols="<,<",options="header",]
|
||
|
|
|=======================================================================
|
||
|
|
|Setting |Description
|
||
|
|
|`max_token_length` |The maximum token length. If a token is seen that
|
||
|
|
exceeds this length then it is discarded. Defaults to `255`.
|
||
|
|
|=======================================================================
|
||
|
|
|