Adds documentation for ISM error prevention (#2090)
* Adds documentation for ISM error prevention Signed-off-by: ariamarble <armarble@amazon.com> * Added additional page for API. Made various corrections Signed-off-by: ariamarble <armarble@amazon.com> * made suggested changes Signed-off-by: ariamarble <armarble@amazon.com> * made editorial changes Signed-off-by: ariamarble <armarble@amazon.com> Signed-off-by: ariamarble <armarble@amazon.com>
This commit is contained in:
Aria Marble
GitHub
parent
e81a878acf
commit
cc34d69139
|
|
@ -0,0 +1,155 @@
|
|||
---
|
||||
layout: default
|
||||
title: ISM Error Prevention API
|
||||
parent: ISM Error Prevention
|
||||
nav_order: 10
|
||||
---
|
||||
|
||||
# ISM Error Prevention API
|
||||
|
||||
The ISM Error Prevention API allows you to enable Index State Management (ISM) error prevention and check the validation status and message.
|
||||
|
||||
## Enable error prevention validation
|
||||
|
||||
You can configure error prevention validation by setting the `plugins.index_state_management.validation_service.enabled` parameter.
|
||||
|
||||
#### Sample request
|
||||
|
||||
```bash
|
||||
PUT _cluster/settings
|
||||
{
|
||||
"persistent":{
|
||||
"plugins.index_state_management.validation_action.enabled": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### Sample response
|
||||
|
||||
```json
|
||||
{
|
||||
"acknowledged" : true,
|
||||
"persistent" : {
|
||||
"plugins" : {
|
||||
"index_state_management" : {
|
||||
"validation_action" : {
|
||||
"enabled" : "true"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"transient" : { }
|
||||
}
|
||||
```
|
||||
|
||||
## Check validation status and message via the Explain API
|
||||
|
||||
Pass the `validate_action=true` path parameter in the Explain API URI to see the validation status and message.
|
||||
|
||||
#### Sample request
|
||||
|
||||
```bash
|
||||
GET _plugins/_ism/explain/test-000001?validate_action=true
|
||||
```
|
||||
|
||||
#### Sample response
|
||||
|
||||
```json
|
||||
{
|
||||
"test-000001" : {
|
||||
"index.plugins.index_state_management.policy_id" : "test_rollover",
|
||||
"index.opendistro.index_state_management.policy_id" : "test_rollover",
|
||||
"index" : "test-000001",
|
||||
"index_uuid" : "CgKsxFmQSIa8dWqpbSJmyA",
|
||||
"policy_id" : "test_rollover",
|
||||
"policy_seq_no" : -2,
|
||||
"policy_primary_term" : 0,
|
||||
"rolled_over" : false,
|
||||
"index_creation_date" : 1667410460649,
|
||||
"state" : {
|
||||
"name" : "rollover",
|
||||
"start_time" : 1667410766045
|
||||
},
|
||||
"action" : {
|
||||
"name" : "rollover",
|
||||
"start_time" : 1667411127803,
|
||||
"index" : 0,
|
||||
"failed" : false,
|
||||
"consumed_retries" : 0,
|
||||
"last_retry_time" : 0
|
||||
},
|
||||
"step" : {
|
||||
"name" : "attempt_rollover",
|
||||
"start_time" : 1667411127803,
|
||||
"step_status" : "starting"
|
||||
},
|
||||
"retry_info" : {
|
||||
"failed" : true,
|
||||
"consumed_retries" : 0
|
||||
},
|
||||
"info" : {
|
||||
"message" : "Previous action was not able to update IndexMetaData."
|
||||
},
|
||||
"enabled" : false,
|
||||
"validate" : {
|
||||
"validation_message" : "Missing rollover_alias index setting [index=test-000001]",
|
||||
"validation_status" : "re_validating"
|
||||
}
|
||||
},
|
||||
"total_managed_indices" : 1
|
||||
}
|
||||
```
|
||||
|
||||
If you pass the parameter without a value or false, then it doesn't return the validation status and message. Only if you pass `validate_action=true` will the response will return the validation status and message.
|
||||
|
||||
#### Sample request
|
||||
|
||||
```bash
|
||||
GET _plugins/_ism/explain/test-000001?validate_action=false
|
||||
--- OR ---
|
||||
GET _plugins/_ism/explain/test-000001
|
||||
```
|
||||
|
||||
#### Sample response
|
||||
|
||||
```json
|
||||
{
|
||||
"test-000001" : {
|
||||
"index.plugins.index_state_management.policy_id" : "test_rollover",
|
||||
"index.opendistro.index_state_management.policy_id" : "test_rollover",
|
||||
"index" : "test-000001",
|
||||
"index_uuid" : "CgKsxFmQSIa8dWqpbSJmyA",
|
||||
"policy_id" : "test_rollover",
|
||||
"policy_seq_no" : -2,
|
||||
"policy_primary_term" : 0,
|
||||
"rolled_over" : false,
|
||||
"index_creation_date" : 1667410460649,
|
||||
"state" : {
|
||||
"name" : "rollover",
|
||||
"start_time" : 1667410766045
|
||||
},
|
||||
"action" : {
|
||||
"name" : "rollover",
|
||||
"start_time" : 1667411127803,
|
||||
"index" : 0,
|
||||
"failed" : false,
|
||||
"consumed_retries" : 0,
|
||||
"last_retry_time" : 0
|
||||
},
|
||||
"step" : {
|
||||
"name" : "attempt_rollover",
|
||||
"start_time" : 1667411127803,
|
||||
"step_status" : "starting"
|
||||
},
|
||||
"retry_info" : {
|
||||
"failed" : true,
|
||||
"consumed_retries" : 0
|
||||
},
|
||||
"info" : {
|
||||
"message" : "Previous action was not able to update IndexMetaData."
|
||||
},
|
||||
"enabled" : false
|
||||
},
|
||||
"total_managed_indices" : 1
|
||||
}
|
||||
```
|
||||
|
|
@ -0,0 +1,69 @@
|
|||
---
|
||||
layout: default
|
||||
title: ISM Error Prevention
|
||||
parent: Index State Management
|
||||
nav_order: 1
|
||||
has_children: true
|
||||
redirect_from:
|
||||
- /im-plugin/ism/error-prevention/
|
||||
---
|
||||
|
||||
# ISM error prevention
|
||||
|
||||
Error prevention validates Index State Management (ISM) actions before they are performed in order to prevent actions from failing. It also outputs additional information from the action validation results in the response of the [Index Explain API]({{site.url}}{{site.baseurl}}/im-plugin/ism/api/#explain-index). Validation rules and troubleshooting of each action are listed in the following sections.
|
||||
|
||||
---
|
||||
|
||||
#### Table of contents
|
||||
1. TOC
|
||||
{:toc}
|
||||
|
||||
|
||||
---
|
||||
|
||||
## rollover
|
||||
|
||||
ISM does not perform a `rollover` action for an index under any of these conditions:
|
||||
|
||||
- [The index is not the write index]({{site.url}}{{site.baseurl}}/im_plugin/ism/error-prevention/resolutions/#the-index-is-not-the-write-index).
|
||||
- [The index does not have an alias]({{site.url}}{{site.baseurl}}/im_plugin/ism/error-prevention/resolutions/#the-index-does-not-have-an-alias).
|
||||
- [The rollover policy does not contain a rollover_alias index setting]({{site.url}}{{site.baseurl}}/im_plugin/ism/error-prevention/resolutions/#the-rollover-policy-misses-rollover_alias-index-setting).
|
||||
- [Skipping of a rollover action has occured]({{site.url}}{{site.baseurl}}/im_plugin/ism/error-prevention/resolutions/#skipping-rollover-action-is-true).
|
||||
- [The index has already been rolled over using the alias successfully]({{site.url}}{{site.baseurl}}/im_plugin/ism/error-prevention/resolutions/#this-index-has-already-been-rolled-over-successfully).
|
||||
|
||||
## delete
|
||||
|
||||
ISM does not perform a `delete` action for an index under any of these conditions:
|
||||
|
||||
- The index does not exist.
|
||||
- The index name is invalid.
|
||||
- The index is the write index for a data stream.
|
||||
|
||||
## force_merge
|
||||
|
||||
ISM does not perform a `force_merge` action for an index if its dataset is too large and exceeds the threshold.
|
||||
|
||||
## replica_count
|
||||
|
||||
ISM does not perform a `replica_count` action for an index under any of these conditions:
|
||||
|
||||
- The amount of data exceeds the threshold.
|
||||
- The number of shards exceeds the maximum.
|
||||
|
||||
## open
|
||||
|
||||
ISM does not perform an `open` action for an index under any of these conditions:
|
||||
|
||||
- The index is blocked.
|
||||
- The number of shards exceeds the maximum.
|
||||
|
||||
## read_only
|
||||
|
||||
ISM does not perform a `read_only` action for an index under any of these conditions:
|
||||
|
||||
- The index is blocked.
|
||||
- The amount of data exceeds the threshold.
|
||||
|
||||
## read_write
|
||||
|
||||
ISM does not perform a `read_write` action for an index if the index is blocked.
|
||||
|
|
@ -0,0 +1,228 @@
|
|||
---
|
||||
layout: default
|
||||
title: ISM Error Prevention resolutions
|
||||
parent: ISM Error Prevention
|
||||
nav_order: 5
|
||||
---
|
||||
|
||||
# ISM error prevention resolutions
|
||||
|
||||
Resolutions of errors for each validation rule action are listed in the following sections.
|
||||
|
||||
---
|
||||
|
||||
#### Table of contents
|
||||
1. TOC
|
||||
{:toc}
|
||||
|
||||
|
||||
---
|
||||
|
||||
## The index is not the write index
|
||||
|
||||
To confirm that the index is a write index, run the following request:
|
||||
|
||||
```bash
|
||||
GET <index>/_alias?pretty
|
||||
```
|
||||
|
||||
If the response does not contain `"is_write_index"` : true, the index is not a write index. The following example confirms that the index is a write index:
|
||||
|
||||
```json
|
||||
{
|
||||
"<index>" : {
|
||||
"aliases" : {
|
||||
"<index_alias>" : {
|
||||
"is_write_index" : true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
To set the index as a write index, run the following request:
|
||||
|
||||
```bash
|
||||
PUT <index>
|
||||
{
|
||||
"aliases": {
|
||||
"<index_alias>" : {
|
||||
"is_write_index" : true
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## The index does not have an alias
|
||||
|
||||
If the index does not have an alias, you can add one by running the following request:
|
||||
|
||||
```bash
|
||||
POST _aliases
|
||||
{
|
||||
"actions": [
|
||||
{
|
||||
"add": {
|
||||
"index": "<target_index>",
|
||||
"alias": "<index_alias>"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Skipping rollover action is true
|
||||
|
||||
In the event that skipping a rollover action occurs, run the following request:
|
||||
|
||||
```bash
|
||||
GET <target_index>/_settings?pretty
|
||||
```
|
||||
|
||||
If you receive the response in the first example, you can reset it by running the request in the second example:
|
||||
|
||||
```json
|
||||
{
|
||||
"index": {
|
||||
"opendistro.index_state_management.rollover_skip": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
```bash
|
||||
PUT <target_index>/_settings
|
||||
{
|
||||
"index": {
|
||||
"index_state_management.rollover_skip": false
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## This index has already been rolled over successfully
|
||||
|
||||
Remove the [rollover policy from the index]({{site.url}}{{site.baseurl}}/im-plugin/ism/api/#remove-policy-from-index) to prevent this error from reoccurring.
|
||||
|
||||
## The rollover policy misses rollover_alias index setting
|
||||
|
||||
Add a `rollover_alias` index setting to the rollover policy to resolve this issue. Run the following request:
|
||||
|
||||
```bash
|
||||
PUT _index_template/ism_rollover
|
||||
{
|
||||
"index_patterns": ["<index_patterns_in_rollover_policy>"],
|
||||
"template": {
|
||||
"settings": {
|
||||
"plugins.index_state_management.rollover_alias": "<rollover_alias>"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Data too large and exceeding the threshold
|
||||
|
||||
Check the [JVM information]({{site.url}}{{site.baseurl}}/api-reference/nodes-apis/nodes-info/) and increase the heap memory.
|
||||
|
||||
## Maximum shards exceeded
|
||||
|
||||
The shard limit per node, or per index, causes this issue to occur. Check whether there is a `total_shards_per_node` limit by running the following request:
|
||||
|
||||
```bash
|
||||
GET /_cluster/settings
|
||||
```
|
||||
|
||||
If the response contains `total_shards_per_node`, increase its value temporarily by running the following request:
|
||||
|
||||
```bash
|
||||
PUT _cluster/settings
|
||||
{
|
||||
"transient":{
|
||||
"cluster.routing.allocation.total_shards_per_node":100
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
To check whether there is a shard limit for an index, run the following request:
|
||||
|
||||
```bash
|
||||
GET <index>/_settings/index.routing-
|
||||
```
|
||||
|
||||
If the response contains the setting in the first example, increase its value or set it to `-1` for unlimited shards, as shown in the second example:
|
||||
|
||||
```json
|
||||
"index" : {
|
||||
"routing" : {
|
||||
"allocation" : {
|
||||
"total_shards_per_node" : "10"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
```bash
|
||||
PUT <index>/_settings
|
||||
{"index.routing.allocation.total_shards_per_node":-1}
|
||||
```
|
||||
|
||||
## The index is a write index for some data stream
|
||||
|
||||
If you still want to delete the index, check your [data stream]({{site.url}}{{site.baseurl}}/opensearch/data-streams/) settings and change the write index.
|
||||
|
||||
## The index is blocked
|
||||
|
||||
Generally, the index is blocked because disk usage has exceeded the flood-stage watermark and the index has a `read-only-allow-delete` block. To resolve this issue, you can:
|
||||
|
||||
1. Remove the `-index.blocks.read_only_allow_delete-` parameter.
|
||||
1. Temporarily increase the disk watermarks.
|
||||
1. Temporarily disable the disk allocation threshold.
|
||||
|
||||
To prevent the issue from reoccurring, it is better to reduce the usage of the disk by increasing disk space, adding new nodes, or removing data or indexes that are no longer needed.
|
||||
|
||||
Remove `-index.blocks.read_only_allow_delete-` by running the following request:
|
||||
|
||||
```bash
|
||||
PUT <index>/_settings
|
||||
{
|
||||
"index.blocks.read_only_allow_delete": null
|
||||
}
|
||||
```
|
||||
|
||||
Increase the low disk watermarks by running the following request:
|
||||
|
||||
```bash
|
||||
PUT _cluster/settings
|
||||
{
|
||||
"transient": {
|
||||
"cluster": {
|
||||
"routing": {
|
||||
"allocation": {
|
||||
"disk": {
|
||||
"watermark": {
|
||||
"low": "25.0gb"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Disable the disk allocation threshold by running the following request:
|
||||
|
||||
```bash
|
||||
PUT _cluster/settings
|
||||
{
|
||||
"transient": {
|
||||
"cluster": {
|
||||
"routing": {
|
||||
"allocation": {
|
||||
"disk": {
|
||||
"threshold_enabled" : false
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
Loading…
Reference in New Issue