new Text analyzers page + Lang analyzer section (#1376)
* test new DCO bypass * for dco auto sign test Signed-off-by: alicejw <alicejw@amazon.com> * test dco check Signed-off-by: alicejw <alicejw@amazon.com> * for new analyzers page Signed-off-by: alicejw <alicejw@amazon.com> * test dco check after pull from main Signed-off-by: alicejw <alicejw@amazon.com> * for new text analyzers page Signed-off-by: alicejw <alicejw@amazon.com> * remove lang analyzers section from fulltext page, add link to new page text analyzers Signed-off-by: alicejw <alicejw@amazon.com> * rename page to text-analyzers Signed-off-by: alicejw <alicejw@amazon.com> * rmv test text for DCO check Signed-off-by: alicejw <alicejw@amazon.com> * for querydsl analyzers Signed-off-by: alicejw <alicejw@amazon.com> * for note about other 7 analyzer sections to be published soon Signed-off-by: alicejw <alicejw@amazon.com> * for definitions of 7 specialized analyzers and note that full reference is in-progress to be published soon Signed-off-by: alicejw <alicejw@amazon.com> * add note to learn more and point to concepts page Signed-off-by: alicejw <alicejw@amazon.com> * for peer edit comments Signed-off-by: alicejw <alicejw@amazon.com> * add new line Signed-off-by: alicejw <alicejw@amazon.com> * remove specialized modifier for the text analyzers Signed-off-by: alicejw <alicejw@amazon.com> * doc review comments Signed-off-by: alicejw <alicejw@amazon.com> * change title Signed-off-by: alicejw <alicejw@amazon.com> * better page title Signed-off-by: alicejw <alicejw@amazon.com> Signed-off-by: alicejw <alicejw@amazon.com>
This commit is contained in:
Alice Williams
GitHub
parent
2492a54d88
commit
84d8be017f
|
|
@ -395,40 +395,7 @@ GET _search
|
|||
|
||||
## Convert text with analyzers
|
||||
|
||||
OpenSearch provides the analyzer option to convert your structured text into the format that works best for your searches. You can use the following options with the analyzer field: standard, simple, whitespace, stop, keyword, pattern, fingerprint, and language. Different analyzers have different character filters, tokenizers, and token filters. The stop analyzer, for example, removes stop words (e.g., “an,” “but,” “this”) from the query string.
|
||||
|
||||
OpenSearch supports the following language values with the `analyzer` option:
|
||||
arabic, armenian, basque, bengali, brazilian, bulgarian, catalan, czech, danish, dutch, english, estonian, finnish, french, galicia, german, greek, hindi, hungarian, indonesian, irish, italian, latvian, lithuanian, norwegian, persian, portuguese, romanian, russian, sorani, spanish, swedish, turkish, and thai.
|
||||
|
||||
To use the analyzer when you map an index, specify the value within your query. For example, to map your index with the French language analyzer, specify the `french` value for the analyzer field:
|
||||
|
||||
```json
|
||||
"analyzer": "french"
|
||||
```
|
||||
|
||||
#### Sample Request
|
||||
|
||||
The following query maps an index with the language analyzer set to `french`:
|
||||
|
||||
```json
|
||||
PUT my-index-000001
|
||||
|
||||
{
|
||||
"mappings": {
|
||||
"properties": {
|
||||
"text": {
|
||||
"type": "text",
|
||||
"fields": {
|
||||
"french": {
|
||||
"type": "text",
|
||||
"analyzer": "french"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
To convert unstructured text into the format that you need, you can use the text analyzers. To learn more about how to convert unstructured text into structured text that is optimized for search, see [Text analyzers]({{site.url}}{{site.baseurl}}/opensearch/query-dsl/text-analyzers).
|
||||
|
||||
<!-- TO do: each of the options needs its own section with an example. Convert table to individual sections, and then give a streamlined list with valid values. -->
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,104 @@
|
|||
---
|
||||
layout: default
|
||||
title: Text analyzers
|
||||
parent: Query DSL
|
||||
nav_order: 41
|
||||
---
|
||||
|
||||
|
||||
# Optimizing text for searches with text analyzers
|
||||
|
||||
OpenSearch applies text analysis during indexing or searching for `text` fields. There is a standard analyzer that OpenSearch uses by default for text analysis. To optimize unstructured text for search, you can convert it into structured text with our text analyzers.
|
||||
|
||||
## Text analyzers
|
||||
|
||||
OpenSearch provides several text analyzers to convert your structured text into the format that works best for your searches.
|
||||
|
||||
OpenSearch supports the following text analyzers:
|
||||
|
||||
1. **Standard analyzer** – parses strings into terms at word boundaries per the Unicode text segmentation algorithm. It removes most, but not all punctuation. It converts strings to lowercase. You can remove stop words if you turn on that option, but it does not remove stop words by default.
|
||||
1. **Simple analyzer** – converts strings to lowercase, and removes non-letter characters when it splits a string into tokens on any non-letter character.
|
||||
1. **Whitespace analyzer** – parses strings into terms between each whitespace.
|
||||
1. **Stop analyzer** – Converts strings to lowercase and removes non-letter characters by splitting strings into tokens at each non-letter character. It also removes stop words (e.g. "but," or "this") from strings.
|
||||
1. **Keyword analyzer** – receives a string as input and outputs the entire string as one term.
|
||||
1. **Pattern analyzer** – splits strings into terms using regular expressions and supports converting strings to lowercase. It also supports removing stop words.
|
||||
1. **Language analyzer** – provides analyzers specific to multiple languages.
|
||||
1. **Fingerprint analyzer** – creates a fingerprint to use as a duplicate detector.
|
||||
|
||||
The full specialized text analyzers reference is in-progress and will be published soon.
|
||||
{: .note }
|
||||
|
||||
## How to use text analyzers
|
||||
|
||||
If you want to use a text analyzer, specify the name of the analyzer for the `analyzer` field: standard, simple, whitespace, stop, keyword, pattern, fingerprint, and language.
|
||||
|
||||
Each analyzer consists of one tokenizer and zero or more token filters. Different analyzers have different character filters, tokenizers and token filters. To pre-process the string before the tokenizer is applied, you can use one or more character filters.
|
||||
|
||||
#### Example: Specify the standard analyzer in a simple query
|
||||
|
||||
```json
|
||||
GET _search
|
||||
{
|
||||
"query": {
|
||||
"match": {
|
||||
"title": "A brief history of Time",
|
||||
"analyzer": "standard"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- This is a list of the 7 individual new pages we need to write
|
||||
If you want to select one of the text analyzers, see [Text analyzers reference]({{site.url}}{{site.baseurl}}/opensearch/query-dsl/specialized-analyzers).
|
||||
|
||||
## Specialized text analyzers
|
||||
|
||||
1. Standard analyzer
|
||||
1. Simple
|
||||
1. Whitespace
|
||||
1. Stop
|
||||
1. Keyword
|
||||
1. Pattern
|
||||
1. Language
|
||||
1. Fingerprint
|
||||
-->
|
||||
|
||||
## Language analyzer
|
||||
|
||||
OpenSearch supports the following language values with the `analyzer` option:
|
||||
arabic, armenian, basque, bengali, brazilian, bulgarian, catalan, czech, danish, dutch, english, estonian, finnish, french, galicia, german, greek, hindi, hungarian, indonesian, irish, italian, latvian, lithuanian, norwegian, persian, portuguese, romanian, russian, sorani, spanish, swedish, turkish, and thai.
|
||||
|
||||
To use the analyzer when you map an index, specify the value within your query. For example, to map your index with the French language analyzer, specify the `french` value for the analyzer field:
|
||||
|
||||
```json
|
||||
"analyzer": "french"
|
||||
```
|
||||
|
||||
#### Sample Request
|
||||
|
||||
The following query maps an index with the language analyzer set to `french`:
|
||||
|
||||
```json
|
||||
PUT my-index-000001
|
||||
|
||||
{
|
||||
"mappings": {
|
||||
"properties": {
|
||||
"text": {
|
||||
"type": "text",
|
||||
"fields": {
|
||||
"french": {
|
||||
"type": "text",
|
||||
"analyzer": "french"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
<!-- TO do: each of the options needs its own section with an example. Convert table to individual sections, and then give a streamlined list with valid values. -->
|
||||
Loading…
Reference in New Issue