Steve-O-Murphy analyzer apis documentation (#1594)
* Add analyzer endpoints Signed-off-by: Steve Murphy <stevemurphy@BCSR-11413.local> * Post Add analyzer endpoints documentation Signed-off-by: ariamarble <armarble@amazon.com> * made suggested changes Signed-off-by: ariamarble <armarble@amazon.com> * made editorial changes Signed-off-by: ariamarble <armarble@amazon.com> Signed-off-by: Steve Murphy <stevemurphy@BCSR-11413.local> Signed-off-by: ariamarble <armarble@amazon.com> Co-authored-by: Steve Murphy <stevemurphy@BCSR-11413.local>
This commit is contained in:
Aria Marble
GitHub
parent
128f57abba
commit
e7f5ebb336
|
|
@ -0,0 +1,13 @@
|
||||||
|
---
|
||||||
|
layout: default
|
||||||
|
title: Analyze API
|
||||||
|
parent: REST API reference
|
||||||
|
has_children: true
|
||||||
|
nav_order: 7
|
||||||
|
redirect_from:
|
||||||
|
- /opensearch/rest-api/analyze-apis/
|
||||||
|
---
|
||||||
|
|
||||||
|
# Analyze API
|
||||||
|
|
||||||
|
The analyze API allows you to perform text analysis, which is the process of converting unstructured text into individual tokens (usually words) that are optimized for search.
|
||||||
|
|
@ -0,0 +1,670 @@
|
||||||
|
---
|
||||||
|
layout: default
|
||||||
|
title: Perform text analysis
|
||||||
|
parent: Analyze API
|
||||||
|
grand_parent: REST API reference
|
||||||
|
nav_order: 2
|
||||||
|
---
|
||||||
|
|
||||||
|
# Perform text analysis
|
||||||
|
|
||||||
|
The perform text analysis API analyzes a text string and returns the resulting tokens.
|
||||||
|
|
||||||
|
If you use the security plugin, you must have the `manage index` privilege. If you simply want to analyze text, you must have the `manager cluster` privilege.
|
||||||
|
{: .note}
|
||||||
|
|
||||||
|
## Path and HTTP methods
|
||||||
|
|
||||||
|
```
|
||||||
|
GET /_analyze
|
||||||
|
GET /{index}/_analyze
|
||||||
|
POST /_analyze
|
||||||
|
POST /{index}/_analyze
|
||||||
|
```
|
||||||
|
|
||||||
|
Although you can issue an analyzer request via both `GET` and `POST` requests, the two have important distinctions. A `GET` request causes data to be cached in the index so that the next time the data is requested, it is retrieved faster. A `POST` request sends a string that does not already exist to the analyzer to be compared to data that is already in the index. `POST` requests are not cached.
|
||||||
|
{: .note}
|
||||||
|
|
||||||
|
## Path parameter
|
||||||
|
|
||||||
|
You can include the following optional path parameter in your request.
|
||||||
|
|
||||||
|
Parameter | Data Type | Description
|
||||||
|
:--- | :--- | :---
|
||||||
|
index | String | Index that is used to derive the analyzer.
|
||||||
|
|
||||||
|
## Query parameters
|
||||||
|
|
||||||
|
You can include the following optional query parameters in your request.
|
||||||
|
|
||||||
|
Field | Data Type | Description
|
||||||
|
:--- | :--- | :---
|
||||||
|
analyzer | String | The name of the analyzer to apply to the `text` field. The analyzer can be built in or configured in the index.<br /><br />If `analyzer` is not specified, the analyze API uses the analyzer defined in the mapping of the `field` field.<br /><br />If the `field` field is not specified, the analyze API uses the default analyzer for the index.<br /><br > If no index is specified or the index does not have a default analyzer, the analyze API uses the standard analyzer.
|
||||||
|
attributes | Array of Strings | Array of token attributes for filtering the output of the `explain` field.
|
||||||
|
char_filter | Array of Strings | Array of character filters for preprocessing characters before the `tokenizer` field.
|
||||||
|
explain | Boolean | If true, causes the response to include token attributes and additional details. Defaults to `false`.
|
||||||
|
field | String | Field for deriving the analyzer. <br /><br > If you specify `field`, you must also specify the `index` path parameter. <br /><br > If you specify the `analyzer` field, it overrides the value of `field`. <br /><br > If you do not specify `field`, the analyze API uses the default analyzer for the index. <br /><br > If you do not specify the `index` field, or the index does not have a default analyzer, the analyze API uses the standard analyzer.
|
||||||
|
filter | Array of Strings | Array of token filters to apply after the `tokenizer` field.
|
||||||
|
normalizer | String | Normalizer for converting text into a single token.
|
||||||
|
tokenizer | String | Tokenizer for converting the `text` field into tokens.
|
||||||
|
|
||||||
|
The following query parameter is required.
|
||||||
|
|
||||||
|
Field | Data Type | Description
|
||||||
|
:--- | :--- | :---
|
||||||
|
text | String or Array of Strings | Text to analyze. If you provide an array of strings, the text is analyzed as a multi-value field.
|
||||||
|
|
||||||
|
#### Sample requests
|
||||||
|
|
||||||
|
[Analyze array of text strings](#analyze-array-of-text-strings)
|
||||||
|
|
||||||
|
[Apply a built-in analyzer](#apply-a-built-in-analyzer)
|
||||||
|
|
||||||
|
[Apply a custom analyzer](#apply-a-custom-analyzer)
|
||||||
|
|
||||||
|
[Apply a custom transient analyzer](#apply-a-custom-transient-analyzer)
|
||||||
|
|
||||||
|
[Specify an index](#specify-an-index)
|
||||||
|
|
||||||
|
[Derive the analyzer from an index field](#derive-the-analyzer-from-an-index-field)
|
||||||
|
|
||||||
|
[Specify a normalizer](#specify-a-normalizer)
|
||||||
|
|
||||||
|
[Get token details](#get-token-details)
|
||||||
|
|
||||||
|
[Set a token limit](#set-a-token-limit)
|
||||||
|
|
||||||
|
#### Analyze array of text strings
|
||||||
|
|
||||||
|
When you pass an array of strings to the `text` field, it is analyzed as a multi-value field.
|
||||||
|
|
||||||
|
````json
|
||||||
|
GET /_analyze
|
||||||
|
{
|
||||||
|
"analyzer" : "standard",
|
||||||
|
"text" : ["first array element", "second array element"]
|
||||||
|
}
|
||||||
|
````
|
||||||
|
|
||||||
|
The previous request returns the following fields:
|
||||||
|
|
||||||
|
````json
|
||||||
|
{
|
||||||
|
"tokens" : [
|
||||||
|
{
|
||||||
|
"token" : "first",
|
||||||
|
"start_offset" : 0,
|
||||||
|
"end_offset" : 5,
|
||||||
|
"type" : "<ALPHANUM>",
|
||||||
|
"position" : 0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"token" : "array",
|
||||||
|
"start_offset" : 6,
|
||||||
|
"end_offset" : 11,
|
||||||
|
"type" : "<ALPHANUM>",
|
||||||
|
"position" : 1
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"token" : "element",
|
||||||
|
"start_offset" : 12,
|
||||||
|
"end_offset" : 19,
|
||||||
|
"type" : "<ALPHANUM>",
|
||||||
|
"position" : 2
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"token" : "second",
|
||||||
|
"start_offset" : 20,
|
||||||
|
"end_offset" : 26,
|
||||||
|
"type" : "<ALPHANUM>",
|
||||||
|
"position" : 3
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"token" : "array",
|
||||||
|
"start_offset" : 27,
|
||||||
|
"end_offset" : 32,
|
||||||
|
"type" : "<ALPHANUM>",
|
||||||
|
"position" : 4
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"token" : "element",
|
||||||
|
"start_offset" : 33,
|
||||||
|
"end_offset" : 40,
|
||||||
|
"type" : "<ALPHANUM>",
|
||||||
|
"position" : 5
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
````
|
||||||
|
|
||||||
|
#### Apply a built-in analyzer
|
||||||
|
|
||||||
|
If you omit the `index` path parameter, you can apply any of the built-in analyzers to the text string.
|
||||||
|
|
||||||
|
The following request analyzes text using the `standard` built-in analyzer:
|
||||||
|
|
||||||
|
````json
|
||||||
|
GET /_analyze
|
||||||
|
{
|
||||||
|
"analyzer" : "standard",
|
||||||
|
"text" : "OpenSearch text analysis"
|
||||||
|
}
|
||||||
|
````
|
||||||
|
|
||||||
|
The previous request returns the following fields:
|
||||||
|
|
||||||
|
````json
|
||||||
|
{
|
||||||
|
"tokens" : [
|
||||||
|
{
|
||||||
|
"token" : "opensearch",
|
||||||
|
"start_offset" : 0,
|
||||||
|
"end_offset" : 10,
|
||||||
|
"type" : "<ALPHANUM>",
|
||||||
|
"position" : 0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"token" : "text",
|
||||||
|
"start_offset" : 11,
|
||||||
|
"end_offset" : 15,
|
||||||
|
"type" : "<ALPHANUM>",
|
||||||
|
"position" : 1
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"token" : "analysis",
|
||||||
|
"start_offset" : 16,
|
||||||
|
"end_offset" : 24,
|
||||||
|
"type" : "<ALPHANUM>",
|
||||||
|
"position" : 2
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
````
|
||||||
|
|
||||||
|
#### Apply a custom analyzer
|
||||||
|
|
||||||
|
You can create your own analyzer and specify it in an analyze request.
|
||||||
|
|
||||||
|
In this scenario, a custom analyzer `lowercase_ascii_folding` has been created and associated with the `books2` index. The analyzer converts text to lowercase and converts non-ASCII characters to ASCII.
|
||||||
|
|
||||||
|
The following request applies the custom analyzer to the provided text:
|
||||||
|
|
||||||
|
````json
|
||||||
|
GET /books2/_analyze
|
||||||
|
{
|
||||||
|
"analyzer": "lowercase_ascii_folding",
|
||||||
|
"text" : "Le garçon m'a SUIVI."
|
||||||
|
}
|
||||||
|
````
|
||||||
|
|
||||||
|
The previous request returns the following fields:
|
||||||
|
|
||||||
|
````json
|
||||||
|
{
|
||||||
|
"tokens" : [
|
||||||
|
{
|
||||||
|
"token" : "le",
|
||||||
|
"start_offset" : 0,
|
||||||
|
"end_offset" : 2,
|
||||||
|
"type" : "<ALPHANUM>",
|
||||||
|
"position" : 0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"token" : "garcon",
|
||||||
|
"start_offset" : 3,
|
||||||
|
"end_offset" : 9,
|
||||||
|
"type" : "<ALPHANUM>",
|
||||||
|
"position" : 1
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"token" : "m'a",
|
||||||
|
"start_offset" : 10,
|
||||||
|
"end_offset" : 13,
|
||||||
|
"type" : "<ALPHANUM>",
|
||||||
|
"position" : 2
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"token" : "suivi",
|
||||||
|
"start_offset" : 14,
|
||||||
|
"end_offset" : 19,
|
||||||
|
"type" : "<ALPHANUM>",
|
||||||
|
"position" : 3
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
````
|
||||||
|
|
||||||
|
#### Apply a custom transient analyzer
|
||||||
|
|
||||||
|
You can build a custom transient analyzer from tokenizers, token filters, or character filters. Use the `filter` parameter to specify token filters.
|
||||||
|
|
||||||
|
The following request uses the `uppercase` character filter to convert the text to uppercase:
|
||||||
|
|
||||||
|
````json
|
||||||
|
GET /_analyze
|
||||||
|
{
|
||||||
|
"tokenizer" : "keyword",
|
||||||
|
"filter" : ["uppercase"],
|
||||||
|
"text" : "OpenSearch filter"
|
||||||
|
}
|
||||||
|
````
|
||||||
|
|
||||||
|
The previous request returns the following fields:
|
||||||
|
|
||||||
|
````json
|
||||||
|
{
|
||||||
|
"tokens" : [
|
||||||
|
{
|
||||||
|
"token" : "OPENSEARCH FILTER",
|
||||||
|
"start_offset" : 0,
|
||||||
|
"end_offset" : 17,
|
||||||
|
"type" : "word",
|
||||||
|
"position" : 0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
````
|
||||||
|
<hr />
|
||||||
|
|
||||||
|
The following request uses the `html_strip` filter to remove HTML characters from the text:
|
||||||
|
|
||||||
|
````json
|
||||||
|
GET /_analyze
|
||||||
|
{
|
||||||
|
"tokenizer" : "keyword",
|
||||||
|
"filter" : ["lowercase"],
|
||||||
|
"char_filter" : ["html_strip"],
|
||||||
|
"text" : "<b>Leave</b> right now!"
|
||||||
|
}
|
||||||
|
````
|
||||||
|
|
||||||
|
The previous request returns the following fields:
|
||||||
|
|
||||||
|
```` json
|
||||||
|
{
|
||||||
|
"tokens" : [
|
||||||
|
{
|
||||||
|
"token" : "leave right now!",
|
||||||
|
"start_offset" : 3,
|
||||||
|
"end_offset" : 23,
|
||||||
|
"type" : "word",
|
||||||
|
"position" : 0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
````
|
||||||
|
|
||||||
|
<hr />
|
||||||
|
|
||||||
|
You can combine filters using an array.
|
||||||
|
|
||||||
|
The following request combines a `lowercase` translation with a `stop` filter that removes the words in the `stopwords` array:
|
||||||
|
|
||||||
|
````json
|
||||||
|
GET /_analyze
|
||||||
|
{
|
||||||
|
"tokenizer" : "whitespace",
|
||||||
|
"filter" : ["lowercase", {"type": "stop", "stopwords": [ "to", "in"]}],
|
||||||
|
"text" : "how to train your dog in five steps"
|
||||||
|
}
|
||||||
|
````
|
||||||
|
|
||||||
|
The previous request returns the following fields:
|
||||||
|
|
||||||
|
````json
|
||||||
|
{
|
||||||
|
"tokens" : [
|
||||||
|
{
|
||||||
|
"token" : "how",
|
||||||
|
"start_offset" : 0,
|
||||||
|
"end_offset" : 3,
|
||||||
|
"type" : "word",
|
||||||
|
"position" : 0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"token" : "train",
|
||||||
|
"start_offset" : 7,
|
||||||
|
"end_offset" : 12,
|
||||||
|
"type" : "word",
|
||||||
|
"position" : 2
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"token" : "your",
|
||||||
|
"start_offset" : 13,
|
||||||
|
"end_offset" : 17,
|
||||||
|
"type" : "word",
|
||||||
|
"position" : 3
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"token" : "dog",
|
||||||
|
"start_offset" : 18,
|
||||||
|
"end_offset" : 21,
|
||||||
|
"type" : "word",
|
||||||
|
"position" : 4
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"token" : "five",
|
||||||
|
"start_offset" : 25,
|
||||||
|
"end_offset" : 29,
|
||||||
|
"type" : "word",
|
||||||
|
"position" : 6
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"token" : "steps",
|
||||||
|
"start_offset" : 30,
|
||||||
|
"end_offset" : 35,
|
||||||
|
"type" : "word",
|
||||||
|
"position" : 7
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
````
|
||||||
|
|
||||||
|
#### Specify an index
|
||||||
|
|
||||||
|
You can analyze text using an index's default analyzer, or you can specify a different analyzer.
|
||||||
|
|
||||||
|
The following request analyzes the provided text using the default analyzer associated with the `books` index:
|
||||||
|
|
||||||
|
````json
|
||||||
|
GET /books/_analyze
|
||||||
|
{
|
||||||
|
"text" : "OpenSearch analyze test"
|
||||||
|
}
|
||||||
|
````
|
||||||
|
|
||||||
|
The previous request returns the following fields:
|
||||||
|
|
||||||
|
````json
|
||||||
|
|
||||||
|
"tokens" : [
|
||||||
|
{
|
||||||
|
"token" : "opensearch",
|
||||||
|
"start_offset" : 0,
|
||||||
|
"end_offset" : 10,
|
||||||
|
"type" : "<ALPHANUM>",
|
||||||
|
"position" : 0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"token" : "analyze",
|
||||||
|
"start_offset" : 11,
|
||||||
|
"end_offset" : 18,
|
||||||
|
"type" : "<ALPHANUM>",
|
||||||
|
"position" : 1
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"token" : "test",
|
||||||
|
"start_offset" : 19,
|
||||||
|
"end_offset" : 23,
|
||||||
|
"type" : "<ALPHANUM>",
|
||||||
|
"position" : 2
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
````
|
||||||
|
|
||||||
|
<hr />
|
||||||
|
|
||||||
|
The following request analyzes the provided text using the `keyword` analyzer, which returns the entire text value as a single token:
|
||||||
|
|
||||||
|
````json
|
||||||
|
GET /books/_analyze
|
||||||
|
{
|
||||||
|
"analyzer" : "keyword",
|
||||||
|
"text" : "OpenSearch analyze test"
|
||||||
|
}
|
||||||
|
````
|
||||||
|
|
||||||
|
The previous request returns the following fields:
|
||||||
|
|
||||||
|
````json
|
||||||
|
{
|
||||||
|
"tokens" : [
|
||||||
|
{
|
||||||
|
"token" : "OpenSearch analyze test",
|
||||||
|
"start_offset" : 0,
|
||||||
|
"end_offset" : 23,
|
||||||
|
"type" : "word",
|
||||||
|
"position" : 0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
````
|
||||||
|
|
||||||
|
#### Derive the analyzer from an index field
|
||||||
|
|
||||||
|
You can pass text and a field in the index. The API looks up the field's analyzer and uses it to analyze the text.
|
||||||
|
|
||||||
|
If the mapping does not exist, the API uses the standard analyzer, which converts all text to lowercase and tokenizes based on white space.
|
||||||
|
|
||||||
|
The following request causes the analysis to be based on the mapping for `name`:
|
||||||
|
|
||||||
|
````json
|
||||||
|
GET /books2/_analyze
|
||||||
|
{
|
||||||
|
"field" : "name",
|
||||||
|
"text" : "OpenSearch analyze test"
|
||||||
|
}
|
||||||
|
````
|
||||||
|
|
||||||
|
The previous request returns the following fields:
|
||||||
|
|
||||||
|
````json
|
||||||
|
{
|
||||||
|
"tokens" : [
|
||||||
|
{
|
||||||
|
"token" : "opensearch",
|
||||||
|
"start_offset" : 0,
|
||||||
|
"end_offset" : 10,
|
||||||
|
"type" : "<ALPHANUM>",
|
||||||
|
"position" : 0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"token" : "analyze",
|
||||||
|
"start_offset" : 11,
|
||||||
|
"end_offset" : 18,
|
||||||
|
"type" : "<ALPHANUM>",
|
||||||
|
"position" : 1
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"token" : "test",
|
||||||
|
"start_offset" : 19,
|
||||||
|
"end_offset" : 23,
|
||||||
|
"type" : "<ALPHANUM>",
|
||||||
|
"position" : 2
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
````
|
||||||
|
|
||||||
|
#### Specify a normalizer
|
||||||
|
|
||||||
|
Instead of using a keyword field, you can use the normalizer associated with the index. A normalizer causes the analysis change to produce a single token.
|
||||||
|
|
||||||
|
In this example, the `books2` index includes a normalizer called `to_lower_fold_ascii` that converts text to lowercase and translates non-ASCII text to ASCII.
|
||||||
|
|
||||||
|
The following request applies `to_lower_fold_ascii` to the text:
|
||||||
|
|
||||||
|
````json
|
||||||
|
GET /books2/_analyze
|
||||||
|
{
|
||||||
|
"normalizer" : "to_lower_fold_ascii",
|
||||||
|
"text" : "C'est le garçon qui m'a suivi."
|
||||||
|
}
|
||||||
|
````
|
||||||
|
|
||||||
|
The previous request returns the following fields:
|
||||||
|
|
||||||
|
````json
|
||||||
|
{
|
||||||
|
"tokens" : [
|
||||||
|
{
|
||||||
|
"token" : "c'est le garcon qui m'a suivi.",
|
||||||
|
"start_offset" : 0,
|
||||||
|
"end_offset" : 30,
|
||||||
|
"type" : "word",
|
||||||
|
"position" : 0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
````
|
||||||
|
|
||||||
|
<hr />
|
||||||
|
|
||||||
|
You can create a custom transient normalizer with token and character filters.
|
||||||
|
|
||||||
|
The following request uses the `uppercase` character filter to convert the given text to all uppercase:
|
||||||
|
|
||||||
|
````json
|
||||||
|
GET /_analyze
|
||||||
|
{
|
||||||
|
"filter" : ["uppercase"],
|
||||||
|
"text" : "That is the boy who followed me."
|
||||||
|
}
|
||||||
|
````
|
||||||
|
|
||||||
|
The previous request returns the following fields:
|
||||||
|
|
||||||
|
````json
|
||||||
|
{
|
||||||
|
"tokens" : [
|
||||||
|
{
|
||||||
|
"token" : "THAT IS THE BOY WHO FOLLOWED ME.",
|
||||||
|
"start_offset" : 0,
|
||||||
|
"end_offset" : 32,
|
||||||
|
"type" : "word",
|
||||||
|
"position" : 0
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
````
|
||||||
|
|
||||||
|
#### Get token details
|
||||||
|
|
||||||
|
You can obtain additional details for all tokens by setting the `explain` attribute to `true`.
|
||||||
|
|
||||||
|
The following request provides detailed token information for the `reverse` filter used with the `standard` tokenizer:
|
||||||
|
|
||||||
|
````json
|
||||||
|
GET /_analyze
|
||||||
|
{
|
||||||
|
"tokenizer" : "standard",
|
||||||
|
"filter" : ["reverse"],
|
||||||
|
"text" : "OpenSearch analyze test",
|
||||||
|
"explain" : true,
|
||||||
|
"attributes" : ["keyword"]
|
||||||
|
}
|
||||||
|
````
|
||||||
|
|
||||||
|
The previous request returns the following fields:
|
||||||
|
|
||||||
|
````json
|
||||||
|
{
|
||||||
|
"detail" : {
|
||||||
|
"custom_analyzer" : true,
|
||||||
|
"charfilters" : [ ],
|
||||||
|
"tokenizer" : {
|
||||||
|
"name" : "standard",
|
||||||
|
"tokens" : [
|
||||||
|
{
|
||||||
|
"token" : "OpenSearch",
|
||||||
|
"start_offset" : 0,
|
||||||
|
"end_offset" : 10,
|
||||||
|
"type" : "<ALPHANUM>",
|
||||||
|
"position" : 0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"token" : "analyze",
|
||||||
|
"start_offset" : 11,
|
||||||
|
"end_offset" : 18,
|
||||||
|
"type" : "<ALPHANUM>",
|
||||||
|
"position" : 1
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"token" : "test",
|
||||||
|
"start_offset" : 19,
|
||||||
|
"end_offset" : 23,
|
||||||
|
"type" : "<ALPHANUM>",
|
||||||
|
"position" : 2
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"tokenfilters" : [
|
||||||
|
{
|
||||||
|
"name" : "reverse",
|
||||||
|
"tokens" : [
|
||||||
|
{
|
||||||
|
"token" : "hcraeSnepO",
|
||||||
|
"start_offset" : 0,
|
||||||
|
"end_offset" : 10,
|
||||||
|
"type" : "<ALPHANUM>",
|
||||||
|
"position" : 0
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"token" : "ezylana",
|
||||||
|
"start_offset" : 11,
|
||||||
|
"end_offset" : 18,
|
||||||
|
"type" : "<ALPHANUM>",
|
||||||
|
"position" : 1
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"token" : "tset",
|
||||||
|
"start_offset" : 19,
|
||||||
|
"end_offset" : 23,
|
||||||
|
"type" : "<ALPHANUM>",
|
||||||
|
"position" : 2
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
````
|
||||||
|
|
||||||
|
#### Set a token limit
|
||||||
|
|
||||||
|
You can set a limit to the number of tokens generated. Setting a lower value reduces a node's memory usage. The default value is 10000.
|
||||||
|
|
||||||
|
The following request limits the tokens to four:
|
||||||
|
|
||||||
|
````json
|
||||||
|
PUT /books2
|
||||||
|
{
|
||||||
|
"settings" : {
|
||||||
|
"index.analyze.max_token_count" : 4
|
||||||
|
}
|
||||||
|
}
|
||||||
|
````
|
||||||
|
The preceding request is an index API rather than an analyze API. See [DYNAMIC INDEX SETTINGS]({{site.url}}{{site.baseurl}}/opensearch/rest-api/index-apis/create-index/#dynamic-index-settings) for additional details.
|
||||||
|
{: .note}
|
||||||
|
|
||||||
|
### Response fields
|
||||||
|
|
||||||
|
The text analysis endpoints return the following response fields.
|
||||||
|
|
||||||
|
Field | Data Type | Description
|
||||||
|
:--- | :--- | :---
|
||||||
|
tokens | Array | Array of tokens derived from the `text`. See [token object](#token-object).
|
||||||
|
detail | Object | Details about the analysis and each token. Included only when you request token details. See [detail object](#detail-object).
|
||||||
|
|
||||||
|
#### Token object
|
||||||
|
|
||||||
|
Field | Data Type | Description
|
||||||
|
:--- | :--- | :---
|
||||||
|
token | String | The token's text.
|
||||||
|
start_offset | Integer | The token's starting position within the original text string. Offsets are zero-based.
|
||||||
|
end_offset | Integer | The token's ending position within the original text string.
|
||||||
|
type | String | Classification of the token: `<ALPHANUM>`, `<NUM>`, and so on. The tokenizer usually sets the type, but some filters define their own types. For example, the synonym filter defines the `<SYNONYM>` type.
|
||||||
|
position | Integer | The token's position within the `tokens` array.
|
||||||
|
|
||||||
|
#### Detail object
|
||||||
|
|
||||||
|
Field | Data Type | Description
|
||||||
|
:--- | :--- | :---
|
||||||
|
custom_analyzer | Boolean | Whether the analyzer applied to the text is custom or built in.
|
||||||
|
charfilters | Array | List of character filters applied to the text.
|
||||||
|
tokenizer | Object | Name of the tokenizer applied to the text and a list of tokens<sup>*</sup> with content before the token filters were applied.
|
||||||
|
tokenfilters | Array | List of token filters applied to the text. Each token filter includes the filter's name and a list of tokens<sup>*</sup> with content after the filters were applied. Token filters are listed in the order they are specified in the request.
|
||||||
|
|
||||||
|
See [token object](#token-object) for token field descriptions.
|
||||||
|
{: .note}
|
||||||
|
|
@ -0,0 +1,37 @@
|
||||||
|
---
|
||||||
|
layout: default
|
||||||
|
title: Analysis API Terminology
|
||||||
|
parent: Analyze API
|
||||||
|
grand_parent: REST API reference
|
||||||
|
nav_order: 1
|
||||||
|
---
|
||||||
|
|
||||||
|
# Terminology
|
||||||
|
|
||||||
|
The following sections provide descriptions of important text analysis terms.
|
||||||
|
|
||||||
|
## Analyzers
|
||||||
|
|
||||||
|
Analyzers tell OpenSearch how to index and search text. An analyzer is composed of three components: a tokenizer, zero or more token filters, and zero or more character filters.
|
||||||
|
|
||||||
|
OpenSearch provides *built-in* analyzers. For example, the `standard` built-in analyzer converts text to lowercase and breaks text into tokens based on word boundaries such as carriage returns and white space. The `standard` analyzer is also called the *default* analyzer and is used when no analyzer is specified in the text analysis request.
|
||||||
|
|
||||||
|
If needed, you can combine tokenizers, token filters, and character filters to create a *custom* analyzer.
|
||||||
|
|
||||||
|
#### Tokenizers
|
||||||
|
|
||||||
|
Tokenizers break unstuctured text into tokens and maintain metadata about tokens, such as their start and ending positions in the text.
|
||||||
|
|
||||||
|
#### Character filters
|
||||||
|
|
||||||
|
Character filters examine text and perform translations, such as changing, removing, and adding characters.
|
||||||
|
|
||||||
|
#### Token filters
|
||||||
|
|
||||||
|
Token filters modify tokens, performing operations such as converting a token's characters to uppercase and adding or removing tokens.
|
||||||
|
|
||||||
|
## Normalizers
|
||||||
|
|
||||||
|
Similar to analyzers, normalizers tokenize text but return a single token only. Normalizers do not employ tokenizers; they make limited use of character and token filters, such as those that operate on one character at a time.
|
||||||
|
|
||||||
|
By default, OpenSearch does not apply normalizers. To apply normalizers, you must add them to your data before creating an index.
|
||||||
Loading…
Reference in New Issue