16 lines
577 B
Plaintext
16 lines
577 B
Plaintext
[[analysis-lowercase-tokenizer]]
|
|
=== Lowercase Tokenizer
|
|
|
|
A tokenizer of type `lowercase` that performs the function of
|
|
<<analysis-letter-tokenizer,Letter
|
|
Tokenizer>> and
|
|
<<analysis-lowercase-tokenfilter,Lower
|
|
Case Token Filter>> together. It divides text at non-letters and converts
|
|
them to lower case. While it is functionally equivalent to the
|
|
combination of
|
|
<<analysis-letter-tokenizer,Letter
|
|
Tokenizer>> and
|
|
<<analysis-lowercase-tokenfilter,Lower
|
|
Case Token Filter>>, there is a performance advantage to doing the two
|
|
tasks at once, hence this (redundant) implementation.
|