mirror of
https://github.com/honeymoose/OpenSearch.git
synced 2025-02-05 20:48:22 +00:00
9c85569883
We weren't doing it before because we weren't starting the plugins. Now we are. The hardest part of this was handling the files the tests expect to be on the filesystem. extraConfigFiles was broken.
513 lines
14 KiB
Plaintext
513 lines
14 KiB
Plaintext
[[analysis-icu]]
|
|
=== ICU Analysis Plugin
|
|
|
|
The ICU Analysis plugin integrates the Lucene ICU module into elasticsearch,
|
|
adding extended Unicode support using the http://site.icu-project.org/[ICU]
|
|
libraries, including better analysis of Asian languages, Unicode
|
|
normalization, Unicode-aware case folding, collation support, and
|
|
transliteration.
|
|
|
|
[[analysis-icu-install]]
|
|
[float]
|
|
==== Installation
|
|
|
|
This plugin can be installed using the plugin manager:
|
|
|
|
[source,sh]
|
|
----------------------------------------------------------------
|
|
sudo bin/elasticsearch-plugin install analysis-icu
|
|
----------------------------------------------------------------
|
|
|
|
The plugin must be installed on every node in the cluster, and each node must
|
|
be restarted after installation.
|
|
|
|
[[analysis-icu-remove]]
|
|
[float]
|
|
==== Removal
|
|
|
|
The plugin can be removed with the following command:
|
|
|
|
[source,sh]
|
|
----------------------------------------------------------------
|
|
sudo bin/elasticsearch-plugin remove analysis-icu
|
|
----------------------------------------------------------------
|
|
|
|
The node must be stopped before removing the plugin.
|
|
|
|
[[analysis-icu-normalization-charfilter]]
|
|
==== ICU Normalization Character Filter
|
|
|
|
Normalizes characters as explained
|
|
http://userguide.icu-project.org/transforms/normalization[here].
|
|
It registers itself as the `icu_normalizer` character filter, which is
|
|
available to all indices without any further configuration. The type of
|
|
normalization can be specified with the `name` parameter, which accepts `nfc`,
|
|
`nfkc`, and `nfkc_cf` (default). Set the `mode` parameter to `decompose` to
|
|
convert `nfc` to `nfd` or `nfkc` to `nfkd` respectively:
|
|
|
|
Here are two examples, the default usage and a customised character filter:
|
|
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
PUT icu_sample
|
|
{
|
|
"settings": {
|
|
"index": {
|
|
"analysis": {
|
|
"analyzer": {
|
|
"nfkc_cf_normalized": { <1>
|
|
"tokenizer": "icu_tokenizer",
|
|
"char_filter": [
|
|
"icu_normalizer"
|
|
]
|
|
},
|
|
"nfd_normalized": { <2>
|
|
"tokenizer": "icu_tokenizer",
|
|
"char_filter": [
|
|
"nfd_normalizer"
|
|
]
|
|
}
|
|
},
|
|
"char_filter": {
|
|
"nfd_normalizer": {
|
|
"type": "icu_normalizer",
|
|
"name": "nfc",
|
|
"mode": "decompose"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
|
|
<1> Uses the default `nfkc_cf` normalization.
|
|
<2> Uses the customized `nfd_normalizer` token filter, which is set to use `nfc` normalization with decomposition.
|
|
|
|
[[analysis-icu-tokenizer]]
|
|
==== ICU Tokenizer
|
|
|
|
Tokenizes text into words on word boundaries, as defined in
|
|
http://www.unicode.org/reports/tr29/[UAX #29: Unicode Text Segmentation].
|
|
It behaves much like the {ref}/analysis-standard-tokenizer.html[`standard` tokenizer],
|
|
but adds better support for some Asian languages by using a dictionary-based
|
|
approach to identify words in Thai, Lao, Chinese, Japanese, and Korean, and
|
|
using custom rules to break Myanmar and Khmer text into syllables.
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
PUT icu_sample
|
|
{
|
|
"settings": {
|
|
"index": {
|
|
"analysis": {
|
|
"analyzer": {
|
|
"my_icu_analyzer": {
|
|
"tokenizer": "icu_tokenizer"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
|
|
===== Rules customization
|
|
|
|
experimental[]
|
|
|
|
You can customize the `icu-tokenizer` behavior by specifying per-script rule files, see the
|
|
http://userguide.icu-project.org/boundaryanalysis#TOC-RBBI-Rules[RBBI rules syntax reference]
|
|
for a more detailed explanation.
|
|
|
|
To add icu tokenizer rules, set the `rule_files` settings, which should contain a comma-separated list of
|
|
`code:rulefile` pairs in the following format:
|
|
http://unicode.org/iso15924/iso15924-codes.html[four-letter ISO 15924 script code],
|
|
followed by a colon, then a rule file name. Rule files are placed `ES_HOME/config` directory.
|
|
|
|
As a demonstration of how the rule files can be used, save the following user file to `$ES_HOME/config/KeywordTokenizer.rbbi`:
|
|
|
|
[source,text]
|
|
-----------------------
|
|
.+ {200};
|
|
-----------------------
|
|
|
|
Then create an analyzer to use this rule file as follows:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
PUT icu_sample
|
|
{
|
|
"settings": {
|
|
"index":{
|
|
"analysis":{
|
|
"tokenizer" : {
|
|
"icu_user_file" : {
|
|
"type" : "icu_tokenizer",
|
|
"rule_files" : "Latn:KeywordTokenizer.rbbi"
|
|
}
|
|
},
|
|
"analyzer" : {
|
|
"my_analyzer" : {
|
|
"type" : "custom",
|
|
"tokenizer" : "icu_user_file"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
GET _cluster/health?wait_for_status=yellow
|
|
|
|
POST icu_sample/_analyze?analyzer=my_analyzer&text=Elasticsearch. Wow!
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
|
|
The above `analyze` request returns the following:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
{
|
|
"tokens": [
|
|
{
|
|
"token": "Elasticsearch. Wow!",
|
|
"start_offset": 0,
|
|
"end_offset": 19,
|
|
"type": "<ALPHANUM>",
|
|
"position": 0
|
|
}
|
|
]
|
|
}
|
|
--------------------------------------------------
|
|
// TESTRESPONSE
|
|
|
|
[[analysis-icu-normalization]]
|
|
==== ICU Normalization Token Filter
|
|
|
|
Normalizes characters as explained
|
|
http://userguide.icu-project.org/transforms/normalization[here]. It registers
|
|
itself as the `icu_normalizer` token filter, which is available to all indices
|
|
without any further configuration. The type of normalization can be specified
|
|
with the `name` parameter, which accepts `nfc`, `nfkc`, and `nfkc_cf`
|
|
(default).
|
|
|
|
You should probably prefer the <<analysis-icu-normalization-charfilter,Normalization character filter>>.
|
|
|
|
Here are two examples, the default usage and a customised token filter:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
PUT icu_sample
|
|
{
|
|
"settings": {
|
|
"index": {
|
|
"analysis": {
|
|
"analyzer": {
|
|
"nfkc_cf_normalized": { <1>
|
|
"tokenizer": "icu_tokenizer",
|
|
"filter": [
|
|
"icu_normalizer"
|
|
]
|
|
},
|
|
"nfc_normalized": { <2>
|
|
"tokenizer": "icu_tokenizer",
|
|
"filter": [
|
|
"nfc_normalizer"
|
|
]
|
|
}
|
|
},
|
|
"filter": {
|
|
"nfc_normalizer": {
|
|
"type": "icu_normalizer",
|
|
"name": "nfc"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
|
|
<1> Uses the default `nfkc_cf` normalization.
|
|
<2> Uses the customized `nfc_normalizer` token filter, which is set to use `nfc` normalization.
|
|
|
|
|
|
[[analysis-icu-folding]]
|
|
==== ICU Folding Token Filter
|
|
|
|
Case folding of Unicode characters based on `UTR#30`, like the
|
|
{ref}/analysis-asciifolding-tokenfilter.html[ASCII-folding token filter]
|
|
on steroids. It registers itself as the `icu_folding` token filter and is
|
|
available to all indices:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
PUT icu_sample
|
|
{
|
|
"settings": {
|
|
"index": {
|
|
"analysis": {
|
|
"analyzer": {
|
|
"folded": {
|
|
"tokenizer": "icu_tokenizer",
|
|
"filter": [
|
|
"icu_folding"
|
|
]
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
|
|
The ICU folding token filter already does Unicode normalization, so there is
|
|
no need to use Normalize character or token filter as well.
|
|
|
|
Which letters are folded can be controlled by specifying the
|
|
`unicodeSetFilter` parameter, which accepts a
|
|
http://icu-project.org/apiref/icu4j/com/ibm/icu/text/UnicodeSet.html[UnicodeSet].
|
|
|
|
The following example exempts Swedish characters from folding. It is important
|
|
to note that both upper and lowercase forms should be specified, and that
|
|
these filtered character are not lowercased which is why we add the
|
|
`lowercase` filter as well:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
PUT icu_sample
|
|
{
|
|
"settings": {
|
|
"index": {
|
|
"analysis": {
|
|
"analyzer": {
|
|
"swedish_analyzer": {
|
|
"tokenizer": "icu_tokenizer",
|
|
"filter": [
|
|
"swedish_folding",
|
|
"lowercase"
|
|
]
|
|
}
|
|
},
|
|
"filter": {
|
|
"swedish_folding": {
|
|
"type": "icu_folding",
|
|
"unicodeSetFilter": "[^åäöÅÄÖ]"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
|
|
[[analysis-icu-collation]]
|
|
==== ICU Collation Token Filter
|
|
|
|
Collations are used for sorting documents in a language-specific word order.
|
|
The `icu_collation` token filter is available to all indices and defaults to
|
|
using the
|
|
https://www.elastic.co/guide/en/elasticsearch/guide/current/sorting-collations.html#uca[DUCET collation],
|
|
which is a best-effort attempt at language-neutral sorting.
|
|
|
|
Below is an example of how to set up a field for sorting German names in
|
|
``phonebook'' order:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
PUT /my_index
|
|
{
|
|
"settings": {
|
|
"analysis": {
|
|
"filter": {
|
|
"german_phonebook": {
|
|
"type": "icu_collation",
|
|
"language": "de",
|
|
"country": "DE",
|
|
"variant": "@collation=phonebook"
|
|
}
|
|
},
|
|
"analyzer": {
|
|
"german_phonebook": {
|
|
"tokenizer": "keyword",
|
|
"filter": [ "german_phonebook" ]
|
|
}
|
|
}
|
|
}
|
|
},
|
|
"mappings": {
|
|
"user": {
|
|
"properties": {
|
|
"name": { <1>
|
|
"type": "text",
|
|
"fields": {
|
|
"sort": { <2>
|
|
"type": "text",
|
|
"fielddata": true,
|
|
"analyzer": "german_phonebook"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
GET _cluster/health?wait_for_status=yellow
|
|
|
|
GET _search <3>
|
|
{
|
|
"query": {
|
|
"match": {
|
|
"name": "Fritz"
|
|
}
|
|
},
|
|
"sort": "name.sort"
|
|
}
|
|
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
|
|
<1> The `name` field uses the `standard` analyzer, and so support full text queries.
|
|
<2> The `name.sort` field uses the `keyword` analyzer to preserve the name as
|
|
a single token, and applies the `german_phonebook` token filter to index
|
|
the value in German phonebook sort order.
|
|
<3> An example query which searches the `name` field and sorts on the `name.sort` field.
|
|
|
|
===== Collation options
|
|
|
|
`strength`::
|
|
|
|
The strength property determines the minimum level of difference considered
|
|
significant during comparison. Possible values are : `primary`, `secondary`,
|
|
`tertiary`, `quaternary` or `identical`. See the
|
|
http://icu-project.org/apiref/icu4j/com/ibm/icu/text/Collator.html[ICU Collation documentation]
|
|
for a more detailed explanation for each value. Defaults to `tertiary`
|
|
unless otherwise specified in the collation.
|
|
|
|
`decomposition`::
|
|
|
|
Possible values: `no` (default, but collation-dependent) or `canonical`.
|
|
Setting this decomposition property to `canonical` allows the Collator to
|
|
handle unnormalized text properly, producing the same results as if the text
|
|
were normalized. If `no` is set, it is the user's responsibility to insure
|
|
that all text is already in the appropriate form before a comparison or before
|
|
getting a CollationKey. Adjusting decomposition mode allows the user to select
|
|
between faster and more complete collation behavior. Since a great many of the
|
|
world's languages do not require text normalization, most locales set `no` as
|
|
the default decomposition mode.
|
|
|
|
The following options are expert only:
|
|
|
|
`alternate`::
|
|
|
|
Possible values: `shifted` or `non-ignorable`. Sets the alternate handling for
|
|
strength `quaternary` to be either shifted or non-ignorable. Which boils down
|
|
to ignoring punctuation and whitespace.
|
|
|
|
`caseLevel`::
|
|
|
|
Possible values: `true` or `false` (default). Whether case level sorting is
|
|
required. When strength is set to `primary` this will ignore accent
|
|
differences.
|
|
|
|
|
|
`caseFirst`::
|
|
|
|
Possible values: `lower` or `upper`. Useful to control which case is sorted
|
|
first when case is not ignored for strength `tertiary`. The default depends on
|
|
the collation.
|
|
|
|
`numeric`::
|
|
|
|
Possible values: `true` or `false` (default) . Whether digits are sorted
|
|
according to their numeric representation. For example the value `egg-9` is
|
|
sorted before the value `egg-21`.
|
|
|
|
|
|
`variableTop`::
|
|
|
|
Single character or contraction. Controls what is variable for `alternate`.
|
|
|
|
`hiraganaQuaternaryMode`::
|
|
|
|
Possible values: `true` or `false`. Distinguishing between Katakana and
|
|
Hiragana characters in `quaternary` strength.
|
|
|
|
|
|
[[analysis-icu-transform]]
|
|
==== ICU Transform Token Filter
|
|
|
|
Transforms are used to process Unicode text in many different ways, such as
|
|
case mapping, normalization, transliteration and bidirectional text handling.
|
|
|
|
You can define which transformation you want to apply with the `id` parameter
|
|
(defaults to `Null`), and specify text direction with the `dir` parameter
|
|
which accepts `forward` (default) for LTR and `reverse` for RTL. Custom
|
|
rulesets are not yet supported.
|
|
|
|
For example:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
PUT icu_sample
|
|
{
|
|
"settings": {
|
|
"index": {
|
|
"analysis": {
|
|
"analyzer": {
|
|
"latin": {
|
|
"tokenizer": "keyword",
|
|
"filter": [
|
|
"myLatinTransform"
|
|
]
|
|
}
|
|
},
|
|
"filter": {
|
|
"myLatinTransform": {
|
|
"type": "icu_transform",
|
|
"id": "Any-Latin; NFD; [:Nonspacing Mark:] Remove; NFC" <1>
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
GET _cluster/health?wait_for_status=yellow
|
|
|
|
GET icu_sample/_analyze?analyzer=latin
|
|
{
|
|
"text": "你好" <2>
|
|
}
|
|
|
|
GET icu_sample/_analyze?analyzer=latin
|
|
{
|
|
"text": "здравствуйте" <3>
|
|
}
|
|
|
|
GET icu_sample/_analyze?analyzer=latin
|
|
{
|
|
"text": "こんにちは" <4>
|
|
}
|
|
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
|
|
<1> This transforms transliterates characters to Latin, and separates accents
|
|
from their base characters, removes the accents, and then puts the
|
|
remaining text into an unaccented form.
|
|
|
|
<2> Returns `ni hao`.
|
|
<3> Returns `zdravstvujte`.
|
|
<4> Returns `kon'nichiha`.
|
|
|
|
For more documentation, Please see the http://userguide.icu-project.org/transforms/general[user guide of ICU Transform].
|