Add index codec documentation (#4589)
* Add index codec documentation Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Fix links Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Apply suggestions from code review Co-authored-by: Melissa Vagi <vagimeli@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> * Rewording and doc review comments Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Apply suggestions from code review Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> --------- Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Co-authored-by: Melissa Vagi <vagimeli@amazon.com> Co-authored-by: Nathan Bower <nbower@amazon.com>
This commit is contained in:
kolchfa-aws
GitHub
parent
60a85c7e60
commit
afbfa85da1
|
|
@ -649,7 +649,7 @@ PUT /books2
|
|||
````
|
||||
{% include copy-curl.html %}
|
||||
|
||||
The preceding request is an index API rather than an analyze API. See [DYNAMIC INDEX SETTINGS]({{site.url}}{{site.baseurl}}/api-reference/index-apis/create-index/#dynamic-index-settings) for additional details.
|
||||
The preceding request is an index API rather than an analyze API. See [DYNAMIC INDEX SETTINGS]({{site.url}}{{site.baseurl}}/im-plugin/index-settings/#dynamic-index-settings) for additional details.
|
||||
{: .note}
|
||||
|
||||
### Response fields
|
||||
|
|
|
|||
|
|
@ -50,7 +50,7 @@ All alias parameters are optional.
|
|||
|
||||
Parameter | Data Type | Description
|
||||
:--- | :--- | :---
|
||||
master_timeout | Time | The amount of time to wait for a response from the master node. Default is `30s`.
|
||||
cluster_manager_timeout | Time | The amount of time to wait for a response from the cluster manager node. Default is `30s`.
|
||||
timeout | Time | The amount of time to wait for a response from the cluster. Default is `30s`.
|
||||
|
||||
## Request body
|
||||
|
|
|
|||
|
|
@ -55,14 +55,14 @@ Parameter | Type | Description
|
|||
<source-index> | String | The source index to clone.
|
||||
<target-index> | String | The index to create and add cloned data to.
|
||||
wait_for_active_shards | String | The number of active shards that must be available before OpenSearch processes the request. Default is 1 (only the primary shard). Set to all or a positive integer. Values greater than 1 require replicas. For example, if you specify a value of 3, the index must have two replicas distributed across two additional nodes for the operation to succeed.
|
||||
master_timeout | Time | How long to wait for a connection to the master node. Default is `30s`.
|
||||
cluster_manager_timeout | Time | How long to wait for a connection to the cluster manager node. Default is `30s`.
|
||||
timeout | Time | How long to wait for the request to return. Default is `30s`.
|
||||
wait_for_completion | Boolean | When set to `false`, the request returns immediately instead of after the operation is finished. To monitor the operation status, use the [Tasks API]({{site.url}}{{site.baseurl}}/api-reference/tasks/) with the task ID returned by the request. Default is `true`.
|
||||
task_execution_timeout | Time | The explicit task execution timeout. Only useful when wait_for_completion is set to `false`. Default is `1h`.
|
||||
|
||||
## Request body
|
||||
|
||||
The clone index API operation creates a new target index, so you can specify any [index settings]({{site.url}}{{site.baseurl}}/api-reference/index-apis/create-index/#index-settings) and [aliases]({{site.url}}{{site.baseurl}}/opensearch/index-alias) to apply to the target index.
|
||||
The clone index API operation creates a new target index, so you can specify any [index settings]({{site.url}}{{site.baseurl}}/im-plugin/index-settings/) and [aliases]({{site.url}}{{site.baseurl}}/opensearch/index-alias/) to apply to the target index.
|
||||
|
||||
## Response
|
||||
|
||||
|
|
|
|||
|
|
@ -37,7 +37,7 @@ allow_no_indices | Boolean | Whether to ignore wildcards that don't match any in
|
|||
expand_wildcards | String | Expands wildcard expressions to different indexes. Combine multiple values with commas. Available values are all (match all indexes), open (match open indexes), closed (match closed indexes), hidden (match hidden indexes), and none (do not accept wildcard expressions). Default is open.
|
||||
ignore_unavailable | Boolean | If true, OpenSearch does not search for missing or closed indexes. Default is false.
|
||||
wait_for_active_shards | String | Specifies the number of active shards that must be available before OpenSearch processes the request. Default is 1 (only the primary shard). Set to all or a positive integer. Values greater than 1 require replicas. For example, if you specify a value of 3, the index must have two replicas distributed across two additional nodes for the request to succeed.
|
||||
master_timeout | Time | How long to wait for a connection to the master node. Default is `30s`.
|
||||
cluster_manager_timeout | Time | How long to wait for a connection to the cluster manager node. Default is `30s`.
|
||||
timeout | Time | How long to wait for a response from the cluster. Default is `30s`.
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -14,9 +14,45 @@ Introduced 1.0
|
|||
|
||||
While you can create an index by using a document as a base, you can also create an empty index for later use.
|
||||
|
||||
## Example
|
||||
When creating an index, you can specify its mappings, settings, and aliases.
|
||||
|
||||
The following example demonstrates how to create an index with a non-default number of primary and replica shards, specifies that `age` is of type `integer`, and assigns a `sample-alias1` alias to the index.
|
||||
## Path and HTTP methods
|
||||
|
||||
```
|
||||
PUT <index-name>
|
||||
```
|
||||
|
||||
## Index naming restrictions
|
||||
|
||||
OpenSearch indexes have the following naming restrictions:
|
||||
|
||||
- All letters must be lowercase.
|
||||
- Index names can't begin with underscores (`_`) or hyphens (`-`).
|
||||
- Index names can't contain spaces, commas, or the following characters:
|
||||
|
||||
`:`, `"`, `*`, `+`, `/`, `\`, `|`, `?`, `#`, `>`, or `<`
|
||||
|
||||
## Path parameters
|
||||
|
||||
| Parameter | Description |
|
||||
:--- | :---
|
||||
| index | String | The index name. Must conform to the [index naming restrictions](#index-naming-restrictions). Required. |
|
||||
|
||||
## Query parameters
|
||||
|
||||
You can include the following query parameters in your request. All parameters are optional.
|
||||
|
||||
Parameter | Type | Description
|
||||
:--- | :--- | :---
|
||||
wait_for_active_shards | String | Specifies the number of active shards that must be available before OpenSearch processes the request. Default is 1 (only the primary shard). Set to `all` or a positive integer. Values greater than 1 require replicas. For example, if you specify a value of 3, the index must have two replicas distributed across two additional nodes for the request to succeed.
|
||||
cluster_manager_timeout | Time | How long to wait for a connection to the cluster manager node. Default is `30s`.
|
||||
timeout | Time | How long to wait for the request to return. Default is `30s`.
|
||||
|
||||
## Request body
|
||||
|
||||
As part of your request, you can optionally specify [index settings]({{site.url}}{{site.baseurl}}/im-plugin/index-settings/), [mappings]({{site.url}}{{site.baseurl}}/field-types/index/), and [aliases]({{site.url}}{{site.baseurl}}/opensearch/index-alias/) for your newly created index.
|
||||
|
||||
#### Example request
|
||||
|
||||
```json
|
||||
PUT /sample-index1
|
||||
|
|
@ -39,109 +75,3 @@ PUT /sample-index1
|
|||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Path and HTTP methods
|
||||
|
||||
```
|
||||
PUT <index-name>
|
||||
```
|
||||
|
||||
## Index naming restrictions
|
||||
|
||||
OpenSearch indexes have the following naming restrictions:
|
||||
|
||||
- All letters must be lowercase.
|
||||
- Index names can't begin with underscores (`_`) or hyphens (`-`).
|
||||
- Index names can't contain spaces, commas, or the following characters:
|
||||
|
||||
`:`, `"`, `*`, `+`, `/`, `\`, `|`, `?`, `#`, `>`, or `<`
|
||||
|
||||
## URL parameters
|
||||
|
||||
You can include the following URL parameters in your request. All parameters are optional.
|
||||
|
||||
Parameter | Type | Description
|
||||
:--- | :--- | :---
|
||||
wait_for_active_shards | String | Specifies the number of active shards that must be available before OpenSearch processes the request. Default is 1 (only the primary shard). Set to `all` or a positive integer. Values greater than 1 require replicas. For example, if you specify a value of 3, the index must have two replicas distributed across two additional nodes for the request to succeed.
|
||||
master_timeout | Time | How long to wait for a connection to the master node. Default is `30s`.
|
||||
timeout | Time | How long to wait for the request to return. Default is `30s`.
|
||||
|
||||
## Request body
|
||||
|
||||
As part of your request, you can supply parameters in your request's body that specify index settings, mappings, and [aliases]({{site.url}}{{site.baseurl}}/opensearch/index-alias/) for your newly created index. The following sections provide more information about index settings and mappings.
|
||||
|
||||
### Index settings
|
||||
Index settings are separated into two types: static index settings and dynamic index settings. Static index settings are settings that you specify at index creation and can't change later. You can change dynamic settings at any time, including at index creation.
|
||||
|
||||
#### Static index settings
|
||||
|
||||
Setting | Description
|
||||
:--- | :---
|
||||
index.number_of_shards | The number of primary shards in the index. Default is 1.
|
||||
index.number_of_routing_shards | The number of routing shards used to split an index.
|
||||
index.shard.check_on_startup | Whether the index's shards should be checked for corruption. Available options are `false` (do not check for corruption), `checksum` (check for physical corruption), and `true` (check for both physical and logical corruption). Default is `false`.
|
||||
index.routing_partition_size | The number of shards a custom routing value can go to. Routing helps an imbalanced cluster by relocating values to a subset of shards rather than just a single shard. To enable, set this value to greater than 1 but less than `index.number_of_shards`. Default is 1.
|
||||
index.soft_deletes.retention_lease.period | The maximum amount of time to retain a shard's history of operations. Default is `12h`.
|
||||
index.load_fixed_bitset_filters_eagerly | Whether OpenSearch should pre-load cached filters. Available options are `true` and `false`. Default is `true`.
|
||||
index.hidden | Whether the index should be hidden. Hidden indexes are not returned as part of queries that have wildcards. Available options are `true` and `false`. Default is `false`.
|
||||
|
||||
#### Dynamic index Settings
|
||||
|
||||
Setting | Description
|
||||
:--- | :---
|
||||
index.number_of_replicas | The number of replica shards each primary shard should have. For example, if you have 4 primary shards and set `index.number_of_replicas` to 3, the index has 12 replica shards. Default is 1.
|
||||
index.auto_expand_replicas | Whether the cluster should automatically add replica shards based on the number of data nodes. Specify a lower bound and upper limit (for example, 0-9), or `all` for the upper limit. For example, if you have 5 data nodes and set `index.auto_expand_replicas` to 0-3, then the cluster does not automatically add another replica shard. However, if you set this value to `0-all` and add 2 more nodes for a total of 7, the cluster will expand to now have 6 replica shards. Default is disabled.
|
||||
index.codec | Determines how the index’s stored fields are compressed and stored on the disk. This setting impacts the size of the index shards and the performance of the index operations. Available values are `default', 'best_compression`, `zstd`, and `zstd_no_dict`. Two new codecs are introduced in OpenSearch 2.9: `zstd` and `zstd_no_dict`. They provide an option to configure the compression level as an index setting, `index.codec.compression_level`, that is not available for other codecs. For information about each setting, see [Index codec settings](#Index-codec-settings). This setting is optional and cannot be updated dynamically.
|
||||
index.codec.compression_level | The compression level setting provides a trade-off between compression ratio and speed. A higher compression level results in a higher compression ratio (less storage size) with a trade-off on speed, that is, slower compression and decompression speeds (slower indexing and search latencies). Currently, `zstd` and `zstd_no_dict` support compression levels from 1 to 6. Similar to `index.codec`, `index.codec.compression_level` is an optional index setting. The default compression level is 3 if not provided. This setting cannot be updated dynamically.
|
||||
index.search.idle.after | Amount of time a shard should wait for a search or get request until it goes idle. Default is `30s`.
|
||||
index.refresh_interval | How often the index should refresh, which publishes its most recent changes and makes them available for searching. Can be set to `-1` to disable refreshing. Default is `1s`.
|
||||
index.max_result_window | The maximum value of `from` + `size` for searches to the index. `from` is the starting index to search from, and `size` is the amount of results to return. Default: 10000.
|
||||
index.max_inner_result_window | Maximum value of `from` + `size` to return nested search hits and most relevant document aggregated during the query. `from` is the starting index to search from, and `size` is the amount of top hits to return. Default is 100.
|
||||
index.max_rescore_window | The maximum value of `window_size` for rescore requests to the index. Rescore requests reorder the index's documents and return a new score, which can be more precise. Default is the same as index.max_inner_result_window or 10000 by default.
|
||||
index.max_docvalue_fields_search | Maximum amount of `docvalue_fields` allowed in a query. Default is 100.
|
||||
index.max_script_fields | Maximum amount of `script_fields` allowed in a query. Default is 32.
|
||||
index.max_ngram_diff | Maximum difference between `min_gram` and `max_gram` values for `NGramTokenizer` and `NGramTokenFilter` fields. Default is 1.
|
||||
index.max_shingle_diff | Maximum difference between `max_shingle_size` and `min_shingle_size` to feed into the `shingle` token filter. Default is 3.
|
||||
index.max_refresh_listeners | Maximum amount of refresh listeners each shard is allowed to have.
|
||||
index.analyze.max_token_count | Maximum amount of tokens that can return from the `_analyze` API operation. Default is 10000.
|
||||
index.highlight.max_analyzed_offset | The amount of characters a highlight request can analyze. Default is 1000000.
|
||||
index.max_terms_count | The maximum amount of terms a terms query can accept. Default is 65536.
|
||||
index.max_regex_length | The maximum character length of regex that can be in a regexp query. Default is 1000.
|
||||
index.query.default_field | A field or list of fields that OpenSearch uses in queries in case a field isn't specified in the parameters.
|
||||
index.routing.allocation.enable | Specifies options for the index’s shard allocation. Available options are all (allow allocation for all shards), primaries (allow allocation only for primary shards), new_primaries (allow allocation only for new primary shards), and none (do not allow allocation). Default is all.
|
||||
index.routing.rebalance.enable | Enables shard rebalancing for the index. Available options are `all` (allow rebalancing for all shards), `primaries` (allow rebalancing only for primary shards), `replicas` (allow rebalancing only for replicas), and `none` (do not allow rebalancing). Default is `all`.
|
||||
index.gc_deletes | Amount of time to retain a deleted document's version number. Default is `60s`.
|
||||
index.default_pipeline | The default ingest node pipeline for the index. If the default pipeline is set and the pipeline does not exist, then index requests fail. The pipeline name `_none` specifies that the index does not have an ingest pipeline.
|
||||
index.final_pipeline | The final ingest node pipeline for the index. If the final pipeline is set and the pipeline does not exist, then index requests fail. The pipeline name `_none` specifies that the index does not have an ingest pipeline.
|
||||
|
||||
#### Index codec settings
|
||||
The `index.codec` setting of an OpenSearch index determines how the index’s stored fields are compressed and stored on the disk. The setting impacts the size of the index shards and the performance of the index operations. OpenSearch provides support for four different codecs that can be used for compressing the stored fields. Each codec offers different trade-offs between compression ratio (storage size) and indexing performance (speed). The available codecs are:
|
||||
* `default` -- This codec employs the `LZ4` algorithm with a preset dictionary, which prioritizes performance over compression ratio. It offers faster indexing and search operations when compared with `best_compression` but may result in larger index/shard sizes. If no codec is provided in the index settings, then `LZ4` is used as the default algorithm for compression.
|
||||
* `best_compression` -- This codec utilizes `zlib` as an underlying algorithm for compression. It achieves high compression ratios resulting in smaller index sizes. However, this may incur additional CPU usage during index operations and may subsequently result in high indexing and search latencies.
|
||||
* `zstd` -- This codec uses the [`Zstandard` compression algorithm](https://github.com/facebook/zstd), which provides a good balance between compression ratio and speed. It provides significant compression comparable to the `best_compression` codec with reasonable CPU usage and improved indexing/search performance comparable to the `default` codec.
|
||||
* `zstd_no_dict` -- This codec is similar to `zstd` but excludes the dictionary compression feature. It provides faster indexing and search operations compared to `zstd` at the expense of a slightly larger index size.
|
||||
|
||||
Index settings can be updated using a PUT request. Here's an example using the curl commands to close an index, update the settings, and open an index:
|
||||
|
||||
```json
|
||||
POST /your_index/_close
|
||||
```
|
||||
{% include copy-curl.html %}
|
||||
|
||||
```json
|
||||
PUT /your_index/_settings
|
||||
{
|
||||
"index": {
|
||||
"codec": "zstd_no_dict",
|
||||
"codec.compression_level": 3
|
||||
}
|
||||
}
|
||||
```
|
||||
{% include copy-curl.html %}
|
||||
|
||||
```json
|
||||
POST /your_index/_open
|
||||
```
|
||||
{% include copy-curl.html %}
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -45,7 +45,7 @@ Query parameter | Data type | Description
|
|||
:--- | :--- | :---
|
||||
accept_data_loss | Boolean | Must be set to `true` for an `import` or `delete` because OpenSearch is unaware of where the dangling index data came from.
|
||||
timeout | Time units | The amount of time to wait for a response. If no response is received in the defined time period, an error is returned. Default is `30` seconds.
|
||||
master_timeout | Time units | The amount of time to wait for the connection to the cluster manager. If no response is received in the defined time period, an error is returned. Default is `30` seconds.
|
||||
cluster_manager_timeout | Time units | The amount of time to wait for a connection to the cluster manager. If no response is received in the defined time period, an error is returned. Default is `30` seconds.
|
||||
|
||||
## Examples
|
||||
|
||||
|
|
|
|||
|
|
@ -35,7 +35,7 @@ Parameter | Type | Description
|
|||
allow_no_indices | Boolean | Whether to ignore wildcards that don't match any indexes. Default is true.
|
||||
expand_wildcards | String | Expands wildcard expressions to different indexes. Combine multiple values with commas. Available values are all (match all indexes), open (match open indexes), closed (match closed indexes), hidden (match hidden indexes), and none (do not accept wildcard expressions), which must be used with open, closed, or both. Default is open.
|
||||
ignore_unavailable | Boolean | If true, OpenSearch does not include missing or closed indexes in the response.
|
||||
master_timeout | Time | How long to wait for a connection to the master node. Default is `30s`.
|
||||
cluster_manager_timeout | Time | How long to wait for a connection to the cluster manager node. Default is `30s`.
|
||||
timeout | Time | How long to wait for the response to return. Default is `30s`.
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -37,7 +37,7 @@ expand_wildcards | String | Expands wildcard expressions to different indexes. C
|
|||
flat_settings | Boolean | Whether to return settings in the flat form, which can improve readability, especially for heavily nested settings. For example, the flat form of "index": { "creation_date": "123456789" } is "index.creation_date": "123456789".
|
||||
include_defaults | Boolean | Whether to include default settings as part of the response. This parameter is useful for identifying the names and current values of settings you want to update.
|
||||
ignore_unavailable | Boolean | If true, OpenSearch does not search for missing or closed indexes. Default is false.
|
||||
local | Boolean | Whether to return information from only the local node instead of from the master node. Default is false.
|
||||
local | Boolean | Whether to return information from only the local node instead of from the cluster manager node. Default is false.
|
||||
|
||||
|
||||
## Response
|
||||
|
|
|
|||
|
|
@ -37,8 +37,8 @@ expand_wildcards | String | Expands wildcard expressions to different indexes. C
|
|||
flat_settings | Boolean | Whether to return settings in the flat form, which can improve readability, especially for heavily nested settings. For example, the flat form of "index": { "creation_date": "123456789" } is "index.creation_date": "123456789".
|
||||
include_defaults | Boolean | Whether to include default settings as part of the response. This parameter is useful for identifying the names and current values of settings you want to update.
|
||||
ignore_unavailable | Boolean | If true, OpenSearch does not include missing or closed indexes in the response.
|
||||
local | Boolean | Whether to return information from only the local node instead of from the master node. Default is false.
|
||||
master_timeout | Time | How long to wait for a connection to the master node. Default is `30s`.
|
||||
local | Boolean | Whether to return information from only the local node instead of from the cluster manager node. Default is false.
|
||||
cluster_manager_timeout | Time | How long to wait for a connection to the cluster manager node. Default is `30s`.
|
||||
|
||||
|
||||
## Response
|
||||
|
|
|
|||
|
|
@ -41,8 +41,8 @@ expand_wildcards | String | Expands wildcard expressions to different indexes. C
|
|||
flat_settings | Boolean | Whether to return settings in the flat form, which can improve readability, especially for heavily nested settings. For example, the flat form of “index”: { “creation_date”: “123456789” } is “index.creation_date”: “123456789”.
|
||||
include_defaults | String | Whether to include default settings, including settings used within OpenSearch plugins, in the response. Default is false.
|
||||
ignore_unavailable | Boolean | If true, OpenSearch does not include missing or closed indexes in the response.
|
||||
local | Boolean | Whether to return information from the local node only instead of the master node. Default is false.
|
||||
master_timeout | Time | How long to wait for a connection to the master node. Default is `30s`.
|
||||
local | Boolean | Whether to return information from the local node only instead of the cluster manager node. Default is false.
|
||||
cluster_manager_timeout | Time | How long to wait for a connection to the cluster manager node. Default is `30s`.
|
||||
|
||||
## Response
|
||||
|
||||
|
|
|
|||
|
|
@ -37,7 +37,7 @@ allow_no_indices | Boolean | Whether to ignore wildcards that don't match any in
|
|||
expand_wildcards | String | Expands wildcard expressions to different indexes. Combine multiple values with commas. Available values are all (match all indexes), open (match open indexes), closed (match closed indexes), hidden (match hidden indexes), and none (do not accept wildcard expressions). Default is open.
|
||||
ignore_unavailable | Boolean | If true, OpenSearch does not search for missing or closed indexes. Default is false.
|
||||
wait_for_active_shards | String | Specifies the number of active shards that must be available before OpenSearch processes the request. Default is 1 (only the primary shard). Set to all or a positive integer. Values greater than 1 require replicas. For example, if you specify a value of 3, the index must have two replicas distributed across two additional nodes for the request to succeed.
|
||||
master_timeout | Time | How long to wait for a connection to the master node. Default is `30s`.
|
||||
cluster_manager_timeout | Time | How long to wait for a connection to the cluster manager node. Default is `30s`.
|
||||
timeout | Time | How long to wait for a response from the cluster. Default is `30s`.
|
||||
wait_for_completion | Boolean | When set to `false`, the request returns immediately instead of after the operation is finished. To monitor the operation status, use the [Tasks API]({{site.url}}{{site.baseurl}}/api-reference/tasks/) with the task ID returned by the request. Default is `true`.
|
||||
task_execution_timeout | Time | The explicit task execution timeout. Only useful when wait_for_completion is set to `false`. Default is `1h`.
|
||||
|
|
|
|||
|
|
@ -51,7 +51,7 @@ Parameter | Type | description
|
|||
<index-name> | String | The index to shrink.
|
||||
<target-index> | String | The target index to shrink the source index into.
|
||||
wait_for_active_shards | String | Specifies the number of active shards that must be available before OpenSearch processes the request. Default is 1 (only the primary shard). Set to all or a positive integer. Values greater than 1 require replicas. For example, if you specify a value of 3, the index must have two replicas distributed across two additional nodes for the request to succeed.
|
||||
master_timeout | Time | How long to wait for a connection to the master node. Default is `30s`.
|
||||
cluster_manager_timeout | Time | How long to wait for a connection to the cluster manager node. Default is `30s`.
|
||||
timeout | Time | How long to wait for the request to return a response. Default is `30s`.
|
||||
wait_for_completion | Boolean | When set to `false`, the request returns immediately instead of after the operation is finished. To monitor the operation status, use the [Tasks API]({{site.url}}{{site.baseurl}}/api-reference/tasks/) with the task ID returned by the request. Default is `true`.
|
||||
task_execution_timeout | Time | The explicit task execution timeout. Only useful when wait_for_completion is set to `false`. Default is `1h`.
|
||||
|
|
@ -63,7 +63,7 @@ You can use the request body to configure some index settings for the target ind
|
|||
Field | Type | Description
|
||||
:--- | :--- | :---
|
||||
alias | Object | Sets an alias for the target index. Can have the fields `filter`, `index_routing`, `is_hidden`, `is_write_index`, `routing`, or `search_routing`. See [Index Aliases]({{site.url}}{{site.baseurl}}/api-reference/alias/#request-body).
|
||||
settings | Object | Index settings you can apply to your target index. See [Index Settings]({{site.url}}{{site.baseurl}}/api-reference/index-apis/create-index/#index-settings).
|
||||
settings | Object | Index settings you can apply to your target index. See [Index Settings]({{site.url}}{{site.baseurl}}/im-plugin/index-settings/).
|
||||
[max_shard_size](#the-max_shard_size-parameter) | Bytes | Specifies the maximum size of a primary shard in the target index. Because `max_shard_size` conflicts with the `index.number_of_shards` setting, you cannot set both of them at the same time.
|
||||
|
||||
### The `max_shard_size` parameter
|
||||
|
|
@ -78,4 +78,8 @@ The primary shard count of the target index is the smallest factor of the source
|
|||
The maximum number of primary shards for the target index is equal to the number of primary shards in the source index because the shrink operation is used to reduce the primary shard count. As an example, consider a source index with 5 primary shards that occupy a total of 600 GB of storage. If `max_shard_size` is 100 GB, the minimum number of primary shards is 600/100, which is 6. However, because the number of primary shards in the source index is smaller than 6, the number of primary shards in the target index is set to 5.
|
||||
|
||||
The minimum number of primary shards for the target index is 1.
|
||||
{: .note}
|
||||
{: .note}
|
||||
|
||||
## Index codec considerations
|
||||
|
||||
For index codec considerations, see [Index codecs]({{site.url}}{{site.baseurl}}/im-plugin/index-codecs/#splits-and-shrinks).
|
||||
|
|
@ -55,14 +55,14 @@ Parameter | Type | Description
|
|||
<source-index> | String | The source index to split.
|
||||
<target-index> | String | The index to create.
|
||||
wait_for_active_shards | String | The number of active shards that must be available before OpenSearch processes the request. Default is 1 (only the primary shard). Set to all or a positive integer. Values greater than 1 require replicas. For example, if you specify a value of 3, the index must have two replicas distributed across two additional nodes for the operation to succeed.
|
||||
master_timeout | Time | How long to wait for a connection to the master node. Default is `30s`.
|
||||
cluster_manager_timeout | Time | How long to wait for a connection to the cluster manager node. Default is `30s`.
|
||||
timeout | Time | How long to wait for the request to return. Default is `30s`.
|
||||
wait_for_completion | Boolean | When set to `false`, the request returns immediately instead of after the operation is finished. To monitor the operation status, use the [Tasks API]({{site.url}}{{site.baseurl}}/api-reference/tasks/) with the task ID returned by the request. Default is `true`.
|
||||
task_execution_timeout | Time | The explicit task execution timeout. Only useful when wait_for_completion is set to `false`. Default is `1h`.
|
||||
|
||||
## Request body
|
||||
|
||||
The split index API operation creates a new target index, so you can specify any [index settings]({{site.url}}{{site.baseurl}}/api-reference/index-apis/create-index/#index-settings) and [aliases]({{site.url}}{{site.baseurl}}/opensearch/index-alias) to apply to the target index.
|
||||
The split index API operation creates a new target index, so you can specify any [index settings]({{site.url}}{{site.baseurl}}/im-plugin/index-settings/) and [aliases]({{site.url}}{{site.baseurl}}/opensearch/index-alias/) to apply to the target index.
|
||||
|
||||
## Response
|
||||
|
||||
|
|
@ -73,3 +73,7 @@ The split index API operation creates a new target index, so you can specify any
|
|||
"index": "split-index1"
|
||||
}
|
||||
```
|
||||
|
||||
## Index codec considerations
|
||||
|
||||
For index codec considerations, see [Index codecs]({{site.url}}{{site.baseurl}}/im-plugin/index-codecs/#splits-and-shrinks).
|
||||
|
|
@ -11,7 +11,7 @@ redirect_from:
|
|||
Introduced 1.0
|
||||
{: .label .label-purple }
|
||||
|
||||
You can use the update settings API operation to update index-level settings. You can change dynamic index settings at any time, but static settings cannot be changed after index creation. For more information about static and dynamic index settings, see [Create index]({{site.url}}{{site.baseurl}}/api-reference/index-apis/create-index).
|
||||
You can use the update settings API operation to update index-level settings. You can change dynamic index settings at any time, but static settings cannot be changed after index creation. For more information about static and dynamic index settings, see [Create index]({{site.url}}{{site.baseurl}}/api-reference/index-apis/create-index/).
|
||||
|
||||
Aside from the static and dynamic index settings, you can also update individual plugins' settings. To get the full list of updatable settings, run `GET <target-index>/_settings?include_defaults=true`.
|
||||
|
||||
|
|
@ -34,7 +34,7 @@ PUT /sample-index1/_settings
|
|||
PUT /<target-index>/_settings
|
||||
```
|
||||
|
||||
## URL parameters
|
||||
## Query parameters
|
||||
|
||||
All update settings parameters are optional.
|
||||
|
||||
|
|
@ -45,7 +45,7 @@ expand_wildcards | String | Expands wildcard expressions to different indexes. C
|
|||
flat_settings | Boolean | Whether to return settings in the flat form, which can improve readability, especially for heavily nested settings. For example, the flat form of “index”: { “creation_date”: “123456789” } is “index.creation_date”: “123456789”.
|
||||
ignore_unavailable | Boolean | If true, OpenSearch does not include missing or closed indexes in the response.
|
||||
preserve_existing | Boolean | Whether to preserve existing index settings. Default is false.
|
||||
master_timeout | Time | How long to wait for a connection to the master node. Default is `30s`.
|
||||
cluster_manager_timeout | Time | How long to wait for a connection to the cluster manager node. Default is `30s`.
|
||||
timeout | Time | How long to wait for a connection to return. Default is `30s`.
|
||||
|
||||
## Request body
|
||||
|
|
|
|||
|
|
@ -81,10 +81,10 @@ When `datastore.type` is set to `opensearch`, the following reporting settings c
|
|||
| `datastore.ssl.verification_mode` | String | When set to the default `full`, the metrics store’s SSL certificate is checked. To disable certificate verification, set this value to `none`. |
|
||||
| `datastore.ssl.certificate_authorities` | String | Determines the local file system path to the certificate authority’s signing certificate.
|
||||
| `datastore.user` | Username | Sets the username for the metrics store |
|
||||
| `datastore.password`: | String | Sets the password for the metrics store. Alternatively, this password can be configured using the `OSB_DATASTORE_PASSWORD` environment variable, which avoids storing credentials in a plain text file. The environment variable takes precedence over the config file if both define a password. |
|
||||
| `datastore.password` | String | Sets the password for the metrics store. Alternatively, this password can be configured using the `OSB_DATASTORE_PASSWORD` environment variable, which avoids storing credentials in a plain text file. The environment variable takes precedence over the config file if both define a password. |
|
||||
| `datastore.probe.cluster_version` | String | Enables automatic detection of the metrics store’s version. Default is `true`. |
|
||||
| `datastore.number_of_shards` | Integer | The number of primary shards that the `opensearch-*` indexes should have. Any updates to this setting after initial index creation will only be applied to new `opensearch-*` indexes. Default is the [OpenSearch static index value]({{site.url}}{{site.baseurl}}/api-reference/index-apis/create-index/#static-index-settings). |
|
||||
| `datastore.number_of_replicas` | Integer | The number of replicas each primary shard in the datastore contains. Any updates to this setting after initial index creation will only be applied to new `opensearch-* `indexes. Default is the [OpenSearch static index value]({{site.url}}{{site.baseurl}}/api-reference/index-apis/create-index/#static-index-settings). |
|
||||
| `datastore.number_of_shards` | Integer | The number of primary shards that the `opensearch-*` indexes should have. Any updates to this setting after initial index creation will only be applied to new `opensearch-*` indexes. Default is the [OpenSearch static index value]({{site.url}}{{site.baseurl}}/im-plugin/index-settings/#static-index-settings). |
|
||||
| `datastore.number_of_replicas` | Integer | The number of replicas each primary shard in the datastore contains. Any updates to this setting after initial index creation will only be applied to new `opensearch-* `indexes. Default is the [OpenSearch static index value]({{site.url}}{{site.baseurl}}/im-plugin/index-settings/#static-index-settings). |
|
||||
|
||||
### Examples
|
||||
|
||||
|
|
|
|||
|
|
@ -151,7 +151,7 @@ The **Delete** button [deletes]({{site.url}}{{site.baseurl}}/api-reference/snaps
|
|||
|
||||
<img src="{{site.url}}{{site.baseurl}}/images/restore-snapshot/restore-snapshot-custom.png" alt="Custom settings" width="450">
|
||||
|
||||
For more information about index settings, see [Index settings]({{site.url}}{{site.baseurl}}/api-reference/index-apis/create-index/#index-settings).
|
||||
For more information about index settings, see [Index settings]({{site.url}}{{site.baseurl}}/im-plugin/index-settings/).
|
||||
|
||||
For a list of settings that you cannot change or ignore, see [Restore snapshots]({{site.url}}{{site.baseurl}}/opensearch/snapshots/snapshot-restore#restore-snapshots).
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,82 @@
|
|||
---
|
||||
layout: default
|
||||
title: Index codecs
|
||||
nav_order: 3
|
||||
parent: Index settings
|
||||
---
|
||||
|
||||
# Index codecs
|
||||
|
||||
Index codecs determine how the index’s stored fields are compressed and stored on disk. The index codec is controlled by the static `index.codec` setting that specifies the compression algorithm. The setting impacts the index shard size and index operation performance.
|
||||
|
||||
## Supported codecs
|
||||
|
||||
OpenSearch provides support for four codecs that can be used for compressing the stored fields. Each codec offers different tradeoffs between compression ratio (storage size) and indexing performance (speed):
|
||||
|
||||
* `default` -- This codec employs the [LZ4 algorithm](https://en.wikipedia.org/wiki/LZ4_(compression_algorithm)) with a preset dictionary, which prioritizes performance over compression ratio. It offers faster indexing and search operations when compared with `best_compression` but may result in larger index/shard sizes. If no codec is provided in the index settings, then LZ4 is used as the default algorithm for compression.
|
||||
* `best_compression` -- This codec uses [zlib](https://en.wikipedia.org/wiki/Zlib) as an underlying algorithm for compression. It achieves high compression ratios that result in smaller index sizes. However, this may incur additional CPU usage during index operations and may subsequently result in high indexing and search latencies.
|
||||
* `zstd` (OpenSearch 2.9 and later) -- This codec uses the [Zstandard compression algorithm](https://github.com/facebook/zstd), which provides a good balance between compression ratio and speed. It provides significant compression comparable to the `best_compression` codec with reasonable CPU usage and improved indexing and search performance compared to the `default` codec.
|
||||
* `zstd_no_dict` (OpenSearch 2.9 and later) -- This codec is similar to `zstd` but excludes the dictionary compression feature. It provides faster indexing and search operations compared to `zstd` at the expense of a slightly larger index size.
|
||||
|
||||
For the `zstd` and `zstd_no_dict` codecs, you can optionally specify a compression level in the `index.codec.compression_level` setting. This setting takes integers in the [1, 6] range. A higher compression level results in a higher compression ratio (smaller storage size) with a tradeoff in speed (slower compression and decompression speeds lead to greater indexing and search latencies).
|
||||
|
||||
When an index segment is created, it uses the current index codec for compression. If you update the index codec, any segment created after the update will use the new compression algorithm. For specific operation considerations, see [Index codec considerations for index operations](#index-codec-considerations-for-index-operations).
|
||||
{: .note}
|
||||
|
||||
## Choosing a codec
|
||||
|
||||
The choice of index codec impacts the amount of disk space required to store the index data. Codecs like `best_compression`, `zstd`, and `zstd_no_dict` can achieve higher compression ratios, resulting in smaller index sizes. Conversely, the `default` codec doesn’t prioritize compression ratio, resulting in larger index sizes but with faster search operations than `best_compression`. The `zstd` and `zstd_no_dict` codecs ensure better search performance than the other two codecs.
|
||||
|
||||
## Index codec considerations for index operations
|
||||
|
||||
The following index codec considerations apply to various index operations.
|
||||
|
||||
### Writes
|
||||
|
||||
Every index consists of shards, each of which is further divided into Lucene segments. During index writes, the new segments are created based on the codec specified in the index settings. If you update the codec for an index, the new segments will use the new codec algorithm.
|
||||
|
||||
### Merges
|
||||
|
||||
During segment merges, OpenSearch combines smaller index segments into larger segments in order to provide optimal resource utilization and improve performance. The index codec setting influences the speed and efficiency of the merge operations. The number of merges that happen on an index is a factor of the segment size, and a smaller segment size directly translates into smaller merge sizes. If you update the `index.codec` setting, the new merge operations will use the new codec when creating merged segments. The merged segments will have the compression characteristics of the new codec.
|
||||
|
||||
### Splits and shrinks
|
||||
|
||||
The [Split API]({{site.url}}{{site.baseurl}}/api-reference/index-apis/split/) splits an original index into a new index where each original primary shard is divided into two or more primary shards. The [Shrink API]({{site.url}}{{site.baseurl}}/api-reference/index-apis/shrink-index/) shrinks an existing index to a new index with a smaller number of primary shards. As part of split or shrink operations, any newly created segments will use the latest codec settings.
|
||||
|
||||
### Snapshots
|
||||
|
||||
When creating a [snapshot]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/snapshots/index/), the index codec setting influences the size of the snapshot and the time required for its creation. If the codec of an index is updated, newly created snapshots will use the latest codec setting. The resulting snapshot size will reflect the compression characteristics of the latest codec setting. Existing segments included in the snapshot will retain their original compression characteristics.
|
||||
|
||||
When you restore the indexes from a snapshot of a cluster to another cluster, it is important to verify that the target cluster supports the codecs of the segments in the source snapshot. For example, if the source snapshot contains segments of the `zstd` or `zstd_no_dict` codecs (introduced in OpenSearch 2.9), you won't be able to restore the snapshot to a cluster that runs on an older OpenSearch version because it doesn't support these codecs.
|
||||
|
||||
### Reindexing
|
||||
|
||||
When you are performing a [reindex]({{site.url}}{{site.baseurl}}/im-plugin/reindex-data/) operation from a source index, the new segments created in the target index will have the properties of the codec settings of the target index.
|
||||
|
||||
### Index rollups and transforms
|
||||
|
||||
When an index [rollup]({{site.url}}{{site.baseurl}}/im-plugin/index-rollups/) or [transform]({{site.url}}{{site.baseurl}}/im-plugin/index-transforms/) job is completed, the segments created in the target index will have the properties of the index codec specified during target index creation, irrespective of the source index codec. If the target index is created dynamically through a rollup job, the default codec is used for segments of the target index.
|
||||
|
||||
Changing the index codec setting does not affect the size of existing segments. Only new segments created after the update will reflect the new codec setting. To ensure consistent segment sizes and compression ratios, it may be necessary to perform a reindexing or other indexing operation, such as a merge.
|
||||
{: .important}
|
||||
|
||||
## Performance tuning and benchmarking
|
||||
|
||||
Depending on your specific use case, you might need to experiment with different index codec settings to fine-tune the performance of your OpenSearch cluster. Conducting benchmark tests with different codecs and measuring the impact on indexing speed, search performance, and resource utilization can help you identify the optimal index codec setting for your workload. With the `zstd` and `zstd_no_dict` codecs, you can also fine-tune the compression level in order to identify the optimal configuration for your cluster.
|
||||
|
||||
### Benchmarking
|
||||
|
||||
The following table provides a performance comparison of the `best_compression`, `zstd`, and `zstd_no_dict` codecs against the `default` codec. The tests were performed with the [`nyc_taxi`](https://github.com/topics/nyc-taxi-dataset) dataset. The results are listed in terms of percent change, and bold results indicate performance improvement.
|
||||
|
||||
| | `best_compression` | `zstd` | `zstd_no_dict` |
|
||||
|:--- |:--- |:--- |:--- |
|
||||
|**Write** | | |
|
||||
|Median Latency |0% |0% |−1% |
|
||||
|p90 Latency |3% |2% |**−5%** |
|
||||
|Throughput |−2% |**7%** |**14%** |
|
||||
|**Read** | | |
|
||||
|Median Latency |0% |1% |0% |
|
||||
|p90 Latency |1% |1% |**−2%** |
|
||||
|**Disk** | | |
|
||||
| Compression ratio |**−34%** |**−35%** |**−30%** |
|
||||
|
||||
|
|
@ -343,7 +343,7 @@ POST example_rollup/_search
|
|||
}
|
||||
```
|
||||
|
||||
#### Sample Response
|
||||
#### Example response
|
||||
|
||||
```json
|
||||
{
|
||||
|
|
@ -829,4 +829,8 @@ The response contains two buckets, "Error" and "Success", and the document count
|
|||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
```
|
||||
|
||||
## Index codec considerations
|
||||
|
||||
For index codec considerations, see [Index codecs]({{site.url}}{{site.baseurl}}/im-plugin/index-codecs/#index-rollups-and-transforms).
|
||||
|
|
@ -0,0 +1,122 @@
|
|||
---
|
||||
layout: default
|
||||
title: Index settings
|
||||
nav_order: 3
|
||||
has_children: true
|
||||
---
|
||||
|
||||
# Index settings
|
||||
|
||||
You can specify index settings at index creation. There are two types of index settings:
|
||||
|
||||
- [Static index settings](#static-index-settings) are settings that you cannot update while the index is open. To update a static setting, you must close the index, update the setting, and then reopen the index.
|
||||
- [Dynamic index settings](#dynamic-index-settings) are settings that you can update at any time.
|
||||
|
||||
## Specifying a setting when creating an index
|
||||
|
||||
When creating an index, you can specify its static or dynamic settings as follows:
|
||||
|
||||
```json
|
||||
PUT /testindex
|
||||
{
|
||||
"settings": {
|
||||
"index.number_of_shards": 1,
|
||||
"index.number_of_replicas": 2
|
||||
}
|
||||
}
|
||||
```
|
||||
{% include copy-curl.html %}
|
||||
|
||||
## Static index settings
|
||||
|
||||
The following table lists all available static index settings.
|
||||
|
||||
Setting | Description
|
||||
:--- | :---
|
||||
index.number_of_shards | The number of primary shards in the index. Default is 1.
|
||||
index.number_of_routing_shards | The number of routing shards used to split an index.
|
||||
index.shard.check_on_startup | Whether the index's shards should be checked for corruption. Available options are `false` (do not check for corruption), `checksum` (check for physical corruption), and `true` (check for both physical and logical corruption). Default is `false`.
|
||||
index.codec | Determines how the index’s stored fields are compressed and stored on disk. This setting impacts the size of the index shards and the performance of the index operations. Valid values are: <br> - `default`<br> - `best_compression`<br> - `zstd` (OpenSearch 2.9 and later)<br> - `zstd_no_dict`(OpenSearch 2.9 and later). <br>For `zstd` and `zstd_no_dict`, you can specify the compression level in the `index.codec.compression_level` setting. For more information, see [Index codec settings]({{site.url}}{{site.baseurl}}/im-plugin/index-codecs/). Optional. Default is `default`.
|
||||
index.codec.compression_level | The compression level setting provides a tradeoff between compression ratio and speed. A higher compression level results in a higher compression ratio (smaller storage size) with a tradeoff in speed (slower compression and decompression speeds lead to greater indexing and search latencies). Can only be specified if `index.codec` is set to `zstd` and `zstd_no_dict` compression levels in OpenSearch 2.9 and later. Valid values are integers in the [1, 6] range. For more information, see [Index codec settings]({{site.url}}{{site.baseurl}}/im-plugin/index-codecs/). Optional. Default is 3.
|
||||
index.routing_partition_size | The number of shards a custom routing value can go to. Routing helps an imbalanced cluster by relocating values to a subset of shards rather than a single shard. To enable routing, set this value to greater than 1 but less than `index.number_of_shards`. Default is 1.
|
||||
index.soft_deletes.retention_lease.period | The maximum amount of time to retain a shard's history of operations. Default is `12h`.
|
||||
index.load_fixed_bitset_filters_eagerly | Whether OpenSearch should preload cached filters. Available options are `true` and `false`. Default is `true`.
|
||||
index.hidden | Whether the index should be hidden. Hidden indexes are not returned as part of queries that have wildcards. Available options are `true` and `false`. Default is `false`.
|
||||
|
||||
## Updating a static index setting
|
||||
|
||||
You can update a static index setting only on a closed index. The following example demonstrates updating the index codec setting.
|
||||
|
||||
First, close an index:
|
||||
|
||||
```json
|
||||
POST /testindex/_close
|
||||
```
|
||||
{% include copy-curl.html %}
|
||||
|
||||
Then update the settings by sending a request to the `_settings` endpoint:
|
||||
|
||||
```json
|
||||
PUT /testindex/_settings
|
||||
{
|
||||
"index": {
|
||||
"codec": "zstd_no_dict",
|
||||
"codec.compression_level": 3
|
||||
}
|
||||
}
|
||||
```
|
||||
{% include copy-curl.html %}
|
||||
|
||||
Last, reopen the index to enable read and write operations:
|
||||
|
||||
```json
|
||||
POST /testindex/_open
|
||||
```
|
||||
{% include copy-curl.html %}
|
||||
|
||||
For more information about updating settings, including supported query parameters, see [Update settings]({{site.url}}{{site.baseurl}}/api-reference/index-apis/update-settings/).
|
||||
|
||||
## Dynamic index settings
|
||||
|
||||
The following table lists all available dynamic index settings.
|
||||
|
||||
Setting | Description
|
||||
:--- | :---
|
||||
index.number_of_replicas | The number of replica shards each primary shard should have. For example, if you have 4 primary shards and set `index.number_of_replicas` to 3, the index has 12 replica shards. Default is 1.
|
||||
index.auto_expand_replicas | Whether the cluster should automatically add replica shards based on the number of data nodes. Specify a lower bound and upper limit (for example, 0--9) or `all` for the upper limit. For example, if you have 5 data nodes and set `index.auto_expand_replicas` to 0--3, then the cluster does not automatically add another replica shard. However, if you set this value to `0-all` and add 2 more nodes for a total of 7, the cluster will expand to now have 6 replica shards. Default is disabled.
|
||||
index.search.idle.after | The amount of time a shard should wait for a search or get request until it goes idle. Default is `30s`.
|
||||
index.refresh_interval | How often the index should refresh, which publishes its most recent changes and makes them available for searching. Can be set to `-1` to disable refreshing. Default is `1s`.
|
||||
index.max_result_window | The maximum value of `from` + `size` for searches of the index. `from` is the starting index to search from, and `size` is the number of results to return. Default is 10000.
|
||||
index.max_inner_result_window | The maximum value of `from` + `size` that specifies the number of returned nested search hits and most relevant document aggregated during the query. `from` is the starting index to search from, and `size` is the number of top hits to return. Default is 100.
|
||||
index.max_rescore_window | The maximum value of `window_size` for rescore requests to the index. Rescore requests reorder the index's documents and return a new score, which can be more precise. Default is the same as `index.max_inner_result_window` or 10000 by default.
|
||||
index.max_docvalue_fields_search | The maximum number of `docvalue_fields` allowed in a query. Default is 100.
|
||||
index.max_script_fields | The maximum number of `script_fields` allowed in a query. Default is 32.
|
||||
index.max_ngram_diff | The maximum difference between `min_gram` and `max_gram` values for the NGramTokenizer and NGramTokenFilter. Default is 1.
|
||||
index.max_shingle_diff | The maximum difference between `max_shingle_size` and `min_shingle_size` to feed into the `shingle` token filter. Default is 3.
|
||||
index.max_refresh_listeners | The maximum number of refresh listeners each shard is allowed to have.
|
||||
index.analyze.max_token_count | The maximum number of tokens that can be returned from the `_analyze` API operation. Default is 10000.
|
||||
index.highlight.max_analyzed_offset | The number of characters a highlight request can analyze. Default is 1000000.
|
||||
index.max_terms_count | The maximum number of terms a terms query can accept. Default is 65536.
|
||||
index.max_regex_length | The maximum character length of regex that can be in a regexp query. Default is 1000.
|
||||
index.query.default_field | A field or list of fields that OpenSearch uses in queries in case a field isn't specified in the parameters.
|
||||
index.routing.allocation.enable | Specifies options for the index’s shard allocation. Available options are `all` (allow allocation for all shards), `primaries` (allow allocation only for primary shards), `new_primaries` (allow allocation only for new primary shards), and `none` (do not allow allocation). Default is `all`.
|
||||
index.routing.rebalance.enable | Enables shard rebalancing for the index. Available options are `all` (allow rebalancing for all shards), `primaries` (allow rebalancing only for primary shards), `replicas` (allow rebalancing only for replicas), and `none` (do not allow rebalancing). Default is `all`.
|
||||
index.gc_deletes | The amount of time to retain a deleted document's version number. Default is `60s`.
|
||||
index.default_pipeline | The default ingest node pipeline for the index. If the default pipeline is set and the pipeline does not exist, then index requests fail. The pipeline name `_none` specifies that the index does not have an ingest pipeline.
|
||||
index.final_pipeline | The final ingest node pipeline for the index. If the final pipeline is set and the pipeline does not exist, then index requests fail. The pipeline name `_none` specifies that the index does not have an ingest pipeline.
|
||||
|
||||
## Updating a dynamic index setting
|
||||
|
||||
You can update a dynamic index setting at any time through the API. For example, to update the refresh interval, use the following request:
|
||||
|
||||
```json
|
||||
PUT /testindex/_settings
|
||||
{
|
||||
"index": {
|
||||
"refresh_interval": "2s"
|
||||
}
|
||||
}
|
||||
```
|
||||
{% include copy-curl.html %}
|
||||
|
||||
For more information about updating settings, including supported query parameters, see [Update settings]({{site.url}}{{site.baseurl}}/api-reference/index-apis/update-settings/).
|
||||
|
|
@ -151,3 +151,7 @@ GET finished_flight_job/_search
|
|||
}
|
||||
|
||||
```
|
||||
|
||||
## Index codec considerations
|
||||
|
||||
For index codec considerations, see [Index codecs]({{site.url}}{{site.baseurl}}/im-plugin/index-codecs/#index-rollups-and-transforms).
|
||||
|
|
@ -262,3 +262,7 @@ Option | Valid values | Description | Required
|
|||
:--- | :--- | :---
|
||||
`index` | String | The name of the destination index. | Yes
|
||||
`version_type` | Enum | The version type for the indexing operation. Valid values: internal, external, external_gt, external_gte. | No
|
||||
|
||||
## Index codec considerations
|
||||
|
||||
For index codec considerations, see [Index codecs]({{site.url}}{{site.baseurl}}/im-plugin/index-codecs/#reindexing).
|
||||
|
|
|
|||
|
|
@ -12,7 +12,7 @@ has_toc: false
|
|||
|
||||
# Snapshots
|
||||
|
||||
Snapshots are backups of a cluster's indexes and state. State includes cluster settings, node information, index metadata (mappings, settings, templates, etc.), and shard allocation.
|
||||
Snapshots are backups of a cluster's indexes and state. State includes cluster settings, node information, index metadata (mappings, settings, or templates), and shard allocation.
|
||||
|
||||
Snapshots have two main uses:
|
||||
|
||||
|
|
@ -25,6 +25,6 @@ Snapshots have two main uses:
|
|||
For example, if you're moving from a proof-of-concept to a production cluster, you might take a snapshot of the former and restore it on the latter.
|
||||
|
||||
|
||||
You can take and restore snapshots using the [snapshot API]({{site.url}}{{site.baseurl}}/opensearch/snapshots/snapshot-restore).
|
||||
You can take and restore snapshots using the [snapshot API]({{site.url}}{{site.baseurl}}/opensearch/snapshots/snapshot-restore/).
|
||||
|
||||
If you need to automate taking snapshots, you can use the [snapshot management]({{site.url}}{{site.baseurl}}/opensearch/snapshots/snapshot-management) feature.
|
||||
If you need to automate snapshot creation, you can use the [snapshot management]({{site.url}}{{site.baseurl}}/opensearch/snapshots/snapshot-management/) feature.
|
||||
|
|
|
|||
|
|
@ -37,7 +37,7 @@ If you need to delete a snapshot, be sure to use the OpenSearch API rather than
|
|||
|
||||
## Register repository
|
||||
|
||||
Before you can take a snapshot, you have to "register" a snapshot repository. A snapshot repository is just a storage location: a shared file system, Amazon S3, Hadoop Distributed File System (HDFS), Azure Storage, etc.
|
||||
Before you can take a snapshot, you have to "register" a snapshot repository. A snapshot repository is just a storage location: a shared file system, Amazon Simple Storage Service (Amazon S3), Hadoop Distributed File System (HDFS), or Azure Storage.
|
||||
|
||||
|
||||
### Shared file system
|
||||
|
|
@ -340,7 +340,7 @@ Request parameters | Description
|
|||
`partial` | Whether to allow the restoration of partial snapshots. Default is false.
|
||||
`rename_pattern` | If you want to rename indexes as you restore them, use this option to specify a regular expression that matches all indexes you want to restore. Use capture groups (`()`) to reuse portions of the index name.
|
||||
`rename_replacement` | If you want to rename indexes as you restore them, use this option to specify the replacement pattern. Use `$0` to include the entire matching index name, `$1` to include the content of the first capture group, and so on.
|
||||
`index_settings` | If you want to change [index settings]({{site.url}}{{site.baseurl}}/api-reference/index-apis/create-index/#index-settings) applied during restore, specify them here. You cannot change `index.number_of_shards`.
|
||||
`index_settings` | If you want to change [index settings]({{site.url}}{{site.baseurl}}/im-plugin/index-settings/) applied during the restore operation, specify them here. You cannot change `index.number_of_shards`.
|
||||
`ignore_index_settings` | Rather than explicitly specifying new settings with `index_settings`, you can ignore certain index settings in the snapshot and use the cluster defaults applied during restore. You cannot ignore `index.number_of_shards`, `index.number_of_replicas`, or `index.auto_expand_replicas`.
|
||||
`storage_type` | `local` indicates that all snapshot metadata and index data will be downloaded to local storage. <br /><br > `remote_snapshot` indicates that snapshot metadata will be downloaded to the cluster, but the remote repository will remain the authoritative store of the index data. Data will be downloaded and cached as necessary to service queries. At least one node in the cluster must be configured with the [search role]({{site.url}}{{site.baseurl}}/security/access-control/users-roles/) in order to restore a snapshot using the type `remote_snapshot`. <br /><br > Defaults to `local`.
|
||||
|
||||
|
|
@ -385,3 +385,7 @@ curl -k --cert ./kirk.pem --key ./kirk-key.pem -XPOST 'https://localhost:9200/_s
|
|||
|
||||
We strongly recommend against restoring `.opendistro_security` using an admin certificate because doing so can alter the security posture of the entire cluster. See [A word of caution]({{site.url}}{{site.baseurl}}/security-plugin/configuration/security-admin/#a-word-of-caution) for a recommended process to back up and restore your Security plugin configuration.
|
||||
{: .warning}
|
||||
|
||||
## Index codec considerations
|
||||
|
||||
For index codec considerations, see [Index codecs]({{site.url}}{{site.baseurl}}/im-plugin/index-codecs/#snapshots).
|
||||
|
|
|
|||
Loading…
Reference in New Issue