[[analysis-nori]] === Korean (nori) Analysis Plugin The Korean (nori) Analysis plugin integrates Lucene nori analysis module into elasticsearch. It uses the https://bitbucket.org/eunjeon/mecab-ko-dic[mecab-ko-dic dictionary] to perform morphological analysis of Korean texts. :plugin_name: analysis-nori include::install_remove.asciidoc[] [[analysis-nori-analyzer]] ==== `nori` analyzer The `nori` analyzer consists of the following tokenizer and token filters: * <> * <> token filter * <> token filter * {ref}/analysis-lowercase-tokenfilter.html[`lowercase`] token filter It supports the `decompound_mode` and `user_dictionary` settings from <> and the `stoptags` setting from <>. [[analysis-nori-tokenizer]] ==== `nori_tokenizer` The `nori_tokenizer` accepts the following settings: `decompound_mode`:: + -- The decompound mode determines how the tokenizer handles compound tokens. It can be set to: `none`:: No decomposition for compounds. Example output: 가거도항 가곡역 `discard`:: Decomposes compounds and discards the original form (*default*). Example output: 가곡역 => 가곡, 역 `mixed`:: Decomposes compounds and keeps the original form. Example output: 가곡역 => 가곡역, 가곡, 역 -- `discard_punctuation`:: Whether punctuation should be discarded from the output. Defaults to `true`. `user_dictionary`:: + -- The Nori tokenizer uses the https://bitbucket.org/eunjeon/mecab-ko-dic[mecab-ko-dic dictionary] by default. A `user_dictionary` with custom nouns (`NNG`) may be appended to the default dictionary. The dictionary should have the following format: [source,txt] ----------------------- [ ... ] ----------------------- The first token is mandatory and represents the custom noun that should be added in the dictionary. For compound nouns the custom segmentation can be provided after the first token (`[ ... ]`). The segmentation of the custom compound nouns is controlled by the `decompound_mode` setting. As a demonstration of how the user dictionary can be used, save the following dictionary to `$ES_HOME/config/userdict_ko.txt`: [source,txt] ----------------------- c++ <1> C샤프 세종 세종시 세종 시 <2> ----------------------- <1> A simple noun <2> A compound noun (`세종시`) followed by its decomposition: `세종` and `시`. Then create an analyzer as follows: [source,console] -------------------------------------------------- PUT nori_sample { "settings": { "index": { "analysis": { "tokenizer": { "nori_user_dict": { "type": "nori_tokenizer", "decompound_mode": "mixed", "discard_punctuation": "false", "user_dictionary": "userdict_ko.txt" } }, "analyzer": { "my_analyzer": { "type": "custom", "tokenizer": "nori_user_dict" } } } } } } GET nori_sample/_analyze { "analyzer": "my_analyzer", "text": "세종시" <1> } -------------------------------------------------- <1> Sejong city The above `analyze` request returns the following: [source,console-result] -------------------------------------------------- { "tokens" : [ { "token" : "세종시", "start_offset" : 0, "end_offset" : 3, "type" : "word", "position" : 0, "positionLength" : 2 <1> }, { "token" : "세종", "start_offset" : 0, "end_offset" : 2, "type" : "word", "position" : 0 }, { "token" : "시", "start_offset" : 2, "end_offset" : 3, "type" : "word", "position" : 1 }] } -------------------------------------------------- <1> This is a compound token that spans two positions (`mixed` mode). -- `user_dictionary_rules`:: + -- You can also inline the rules directly in the tokenizer definition using the `user_dictionary_rules` option: [source,console] -------------------------------------------------- PUT nori_sample { "settings": { "index": { "analysis": { "tokenizer": { "nori_user_dict": { "type": "nori_tokenizer", "decompound_mode": "mixed", "user_dictionary_rules": ["c++", "C샤프", "세종", "세종시 세종 시"] } }, "analyzer": { "my_analyzer": { "type": "custom", "tokenizer": "nori_user_dict" } } } } } } -------------------------------------------------- -- The `nori_tokenizer` sets a number of additional attributes per token that are used by token filters to modify the stream. You can view all these additional attributes with the following request: [source,console] -------------------------------------------------- GET _analyze { "tokenizer": "nori_tokenizer", "text": "뿌리가 깊은 나무는", <1> "attributes" : ["posType", "leftPOS", "rightPOS", "morphemes", "reading"], "explain": true } -------------------------------------------------- <1> A tree with deep roots Which responds with: [source,console-result] -------------------------------------------------- { "detail": { "custom_analyzer": true, "charfilters": [], "tokenizer": { "name": "nori_tokenizer", "tokens": [ { "token": "뿌리", "start_offset": 0, "end_offset": 2, "type": "word", "position": 0, "leftPOS": "NNG(General Noun)", "morphemes": null, "posType": "MORPHEME", "reading": null, "rightPOS": "NNG(General Noun)" }, { "token": "가", "start_offset": 2, "end_offset": 3, "type": "word", "position": 1, "leftPOS": "J(Ending Particle)", "morphemes": null, "posType": "MORPHEME", "reading": null, "rightPOS": "J(Ending Particle)" }, { "token": "깊", "start_offset": 4, "end_offset": 5, "type": "word", "position": 2, "leftPOS": "VA(Adjective)", "morphemes": null, "posType": "MORPHEME", "reading": null, "rightPOS": "VA(Adjective)" }, { "token": "은", "start_offset": 5, "end_offset": 6, "type": "word", "position": 3, "leftPOS": "E(Verbal endings)", "morphemes": null, "posType": "MORPHEME", "reading": null, "rightPOS": "E(Verbal endings)" }, { "token": "나무", "start_offset": 7, "end_offset": 9, "type": "word", "position": 4, "leftPOS": "NNG(General Noun)", "morphemes": null, "posType": "MORPHEME", "reading": null, "rightPOS": "NNG(General Noun)" }, { "token": "는", "start_offset": 9, "end_offset": 10, "type": "word", "position": 5, "leftPOS": "J(Ending Particle)", "morphemes": null, "posType": "MORPHEME", "reading": null, "rightPOS": "J(Ending Particle)" } ] }, "tokenfilters": [] } } -------------------------------------------------- [[analysis-nori-speech]] ==== `nori_part_of_speech` token filter The `nori_part_of_speech` token filter removes tokens that match a set of part-of-speech tags. The list of supported tags and their meanings can be found here: {lucene-core-javadoc}/../analyzers-nori/org/apache/lucene/analysis/ko/POS.Tag.html[Part of speech tags] It accepts the following setting: `stoptags`:: An array of part-of-speech tags that should be removed. and defaults to: [source,js] -------------------------------------------------- "stoptags": [ "E", "IC", "J", "MAG", "MAJ", "MM", "SP", "SSC", "SSO", "SC", "SE", "XPN", "XSA", "XSN", "XSV", "UNA", "NA", "VSV" ] -------------------------------------------------- // NOTCONSOLE For example: [source,console] -------------------------------------------------- PUT nori_sample { "settings": { "index": { "analysis": { "analyzer": { "my_analyzer": { "tokenizer": "nori_tokenizer", "filter": [ "my_posfilter" ] } }, "filter": { "my_posfilter": { "type": "nori_part_of_speech", "stoptags": [ "NR" <1> ] } } } } } } GET nori_sample/_analyze { "analyzer": "my_analyzer", "text": "여섯 용이" <2> } -------------------------------------------------- <1> Korean numerals should be removed (`NR`) <2> Six dragons Which responds with: [source,console-result] -------------------------------------------------- { "tokens" : [ { "token" : "용", "start_offset" : 3, "end_offset" : 4, "type" : "word", "position" : 1 }, { "token" : "이", "start_offset" : 4, "end_offset" : 5, "type" : "word", "position" : 2 } ] } -------------------------------------------------- [[analysis-nori-readingform]] ==== `nori_readingform` token filter The `nori_readingform` token filter rewrites tokens written in Hanja to their Hangul form. [source,console] -------------------------------------------------- PUT nori_sample { "settings": { "index":{ "analysis":{ "analyzer" : { "my_analyzer" : { "tokenizer" : "nori_tokenizer", "filter" : ["nori_readingform"] } } } } } } GET nori_sample/_analyze { "analyzer": "my_analyzer", "text": "鄕歌" <1> } -------------------------------------------------- <1> A token written in Hanja: Hyangga Which responds with: [source,console-result] -------------------------------------------------- { "tokens" : [ { "token" : "향가", <1> "start_offset" : 0, "end_offset" : 2, "type" : "word", "position" : 0 }] } -------------------------------------------------- <1> The Hanja form is replaced by the Hangul translation. [[analysis-nori-number]] ==== `nori_number` token filter The `nori_number` token filter normalizes Korean numbers to regular Arabic decimal numbers in half-width characters. Korean numbers are often written using a combination of Hangul and Arabic numbers with various kinds punctuation. For example, ３．２천 means 3200. This filter does this kind of normalization and allows a search for 3200 to match ３．２천 in text, but can also be used to make range facets based on the normalized numbers and so on. [NOTE] ==== Notice that this analyzer uses a token composition scheme and relies on punctuation tokens being found in the token stream. Please make sure your `nori_tokenizer` has `discard_punctuation` set to false. In case punctuation characters, such as U+FF0E(．), is removed from the token stream, this filter would find input tokens ３ and ２천 and give outputs 3 and 2000 instead of 3200, which is likely not the intended result. If you want to remove punctuation characters from your index that are not part of normalized numbers, add a `stop` token filter with the punctuation you wish to remove after `nori_number` in your analyzer chain. ==== Below are some examples of normalizations this filter supports. The input is untokenized text and the result is the single term attribute emitted for the input. - 영영칠 -> 7 - 일영영영 -> 1000 - 삼천2백2십삼 -> 3223 - 조육백만오천일 -> 1000006005001 - ３.２천 -> 3200 - １.２만３４５.６７ -> 12345.67 - 4,647.100 -> 4647.1 - 15,7 -> 157 (be aware of this weakness) For example: [source,console] -------------------------------------------------- PUT nori_sample { "settings": { "index": { "analysis": { "analyzer": { "my_analyzer": { "tokenizer": "tokenizer_discard_puncuation_false", "filter": [ "part_of_speech_stop_sp", "nori_number" ] } }, "tokenizer": { "tokenizer_discard_puncuation_false": { "type": "nori_tokenizer", "discard_punctuation": "false" } }, "filter": { "part_of_speech_stop_sp": { "type": "nori_part_of_speech", "stoptags": ["SP"] } } } } } } GET nori_sample/_analyze { "analyzer": "my_analyzer", "text": "십만이천오백과 ３.２천" } -------------------------------------------------- Which results in: [source,console-result] -------------------------------------------------- { "tokens" : [{ "token" : "102500", "start_offset" : 0, "end_offset" : 6, "type" : "word", "position" : 0 }, { "token" : "과", "start_offset" : 6, "end_offset" : 7, "type" : "word", "position" : 1 }, { "token" : "3200", "start_offset" : 8, "end_offset" : 12, "type" : "word", "position" : 2 }] } --------------------------------------------------