From 3d04278a675c88f2399ab9905e96e9d3f2cdc3af Mon Sep 17 00:00:00 2001 From: Stein Arne Storslett Date: Tue, 3 May 2022 09:29:10 +0200 Subject: [PATCH 1/6] Add doc on action rollover min_primary_shard_size Reference https://github.com/opensearch-project/index-management/pull/220 Signed-off-by: Stein Arne Storslett --- _im-plugin/ism/policies.md | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/_im-plugin/ism/policies.md b/_im-plugin/ism/policies.md index 79cc1ed1..d4ac427a 100644 --- a/_im-plugin/ism/policies.md +++ b/_im-plugin/ism/policies.md @@ -206,6 +206,7 @@ Set `index.plugins.index_state_management.rollover_alias` as the alias to rollov Parameter | Description | Type | Example | Required :--- | :--- |:--- |:--- | `min_size` | The minimum size of the total primary shard storage (not counting replicas) required to roll over the index. For example, if you set `min_size` to 100 GiB and your index has 5 primary shards and 5 replica shards of 20 GiB each, the total size of all primary shards is 100 GiB, so the rollover occurs. ISM doesn't check indexes continually, so it doesn't roll over indexes at exactly 100 GiB. Instead, if an index is continuously growing, ISM might check it at 99 GiB, not perform the rollover, check again when the shards reach 105 GiB, and then perform the operation. | `string` | `20gb` or `5mb` | No +`min_primary_shard_size` | The minimum size of a *single primary shard* storage (not counting replicas) required to roll over the index. For example, if you set `min_primary_shard_size` to 30 GiB and there is a *single primary shard* that has a size greater than the condition, the rollover occurs. ISM doesn't check indexes continually, so it doesn't roll over indexes at exactly 30 GiB. Instead, if an index shard is continuously growing, ISM might check it at 29 GiB, not perform the rollover, check again when the shards reach 32 GiB, and then perform the operation. | `string` | `20gb` or `5mb` | No `min_doc_count` | The minimum number of documents required to roll over the index. | `number` | `2000000` | No `min_index_age` | The minimum age required to roll over the index. Index age is the time between its creation and the present. | `string` | `5d` or `7h` | No @@ -217,6 +218,14 @@ Parameter | Description | Type | Example | Required } ``` +```json +{ + "rollover": { + "min_primary_shard_size": "30gb" + } +} +``` + ```json { "rollover": { @@ -618,7 +627,8 @@ After 30 days, the policy moves this index into a `delete` state. The service se "actions": [ { "rollover": { - "min_index_age": "1d" + "min_index_age": "1d", + "min_primary_shard_size": "30gb" } } ], @@ -666,7 +676,11 @@ After 30 days, the policy moves this index into a `delete` state. The service se } ] } - ] + ], + "ism_template": { + "index_patterns": ["log*"], + "priority": 100 + } } } ``` From c799cf43c7aa5a7faa5115a34a7346f62a742f43 Mon Sep 17 00:00:00 2001 From: Stein Arne Storslett Date: Mon, 23 May 2022 09:21:55 +0200 Subject: [PATCH 2/6] Add doc on action rollover min_primary_shard_size updated. Signed-off-by: Stein Arne Storslett --- _im-plugin/ism/policies.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/_im-plugin/ism/policies.md b/_im-plugin/ism/policies.md index d4ac427a..b6a7b639 100644 --- a/_im-plugin/ism/policies.md +++ b/_im-plugin/ism/policies.md @@ -200,15 +200,17 @@ Deletes a managed index. Rolls an alias over to a new index when the managed index meets one of the rollover conditions. +**Important**: ISM checks the conditions for operations on **every execution of the policy** based on the **set interval**, _not_ continuously. If the condition is checked and the value is **close to but not above** the configured value the rollover will not be performed before the value is above the configured limit. For example for `min_size`, with a value of 100GiB, ISM might check the index at 99 GiB, not perform the rollover, check again when the shards reach 105 GiB, and then perform the operation. + The index format must match the pattern: `^.*-\d+$`. For example, `(logs-000001)`. Set `index.plugins.index_state_management.rollover_alias` as the alias to rollover. Parameter | Description | Type | Example | Required :--- | :--- |:--- |:--- | -`min_size` | The minimum size of the total primary shard storage (not counting replicas) required to roll over the index. For example, if you set `min_size` to 100 GiB and your index has 5 primary shards and 5 replica shards of 20 GiB each, the total size of all primary shards is 100 GiB, so the rollover occurs. ISM doesn't check indexes continually, so it doesn't roll over indexes at exactly 100 GiB. Instead, if an index is continuously growing, ISM might check it at 99 GiB, not perform the rollover, check again when the shards reach 105 GiB, and then perform the operation. | `string` | `20gb` or `5mb` | No -`min_primary_shard_size` | The minimum size of a *single primary shard* storage (not counting replicas) required to roll over the index. For example, if you set `min_primary_shard_size` to 30 GiB and there is a *single primary shard* that has a size greater than the condition, the rollover occurs. ISM doesn't check indexes continually, so it doesn't roll over indexes at exactly 30 GiB. Instead, if an index shard is continuously growing, ISM might check it at 29 GiB, not perform the rollover, check again when the shards reach 32 GiB, and then perform the operation. | `string` | `20gb` or `5mb` | No -`min_doc_count` | The minimum number of documents required to roll over the index. | `number` | `2000000` | No -`min_index_age` | The minimum age required to roll over the index. Index age is the time between its creation and the present. | `string` | `5d` or `7h` | No +`min_size` | The minimum size of the total primary shard storage (not counting replicas) required to roll over the index. For example, if you set `min_size` to 100 GiB and your index has 5 primary shards and 5 replica shards of 20 GiB each, the total size of all primary shards is 100 GiB, so the rollover occurs. See **Important** note above. | `string` | `20gb` or `5mb` | No +`min_primary_shard_size` | The minimum storage size of a **single primary shard** required to roll over the index. For example, if you set `min_primary_shard_size` to 30 GiB and **one of** the primary shards in the index has a size greater than the condition, the rollover occurs. See **Important** note above. | `string` | `20gb` or `5mb` | No +`min_doc_count` | The minimum number of documents required to roll over the index. See **Important** note above. | `number` | `2000000` | No +`min_index_age` | The minimum age required to roll over the index. Index age is the time between its creation and the present. See **Important** note above. | `string` | `5d` or `7h` | No ```json { From 763f6ce0e8ec73125dba434fa760b3a047ab2e09 Mon Sep 17 00:00:00 2001 From: Stein Arne Storslett Date: Mon, 23 May 2022 09:30:50 +0200 Subject: [PATCH 3/6] Add doc on action rollover min_primary_shard_size language. Signed-off-by: Stein Arne Storslett --- _im-plugin/ism/policies.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_im-plugin/ism/policies.md b/_im-plugin/ism/policies.md index b6a7b639..eb9531d0 100644 --- a/_im-plugin/ism/policies.md +++ b/_im-plugin/ism/policies.md @@ -200,7 +200,7 @@ Deletes a managed index. Rolls an alias over to a new index when the managed index meets one of the rollover conditions. -**Important**: ISM checks the conditions for operations on **every execution of the policy** based on the **set interval**, _not_ continuously. If the condition is checked and the value is **close to but not above** the configured value the rollover will not be performed before the value is above the configured limit. For example for `min_size`, with a value of 100GiB, ISM might check the index at 99 GiB, not perform the rollover, check again when the shards reach 105 GiB, and then perform the operation. +**Important**: ISM checks the conditions for operations on **every execution of the policy** based on the **set interval**, _not_ continuously. The rollover will not be performed before the value is **above** the configured limit **when the check is performed**. For example for `min_size`, with a value of 100GiB, ISM might check the index at 99 GiB, not perform the rollover, check again when the shards reach 105 GiB, and then perform the operation. The index format must match the pattern: `^.*-\d+$`. For example, `(logs-000001)`. Set `index.plugins.index_state_management.rollover_alias` as the alias to rollover. From 2992b6cb7aacf9a53543c0c35d64c18c400d82f3 Mon Sep 17 00:00:00 2001 From: Stein Arne Storslett Date: Mon, 23 May 2022 09:33:38 +0200 Subject: [PATCH 4/6] Add doc on action rollover min_primary_shard_size language. Signed-off-by: Stein Arne Storslett --- _im-plugin/ism/policies.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_im-plugin/ism/policies.md b/_im-plugin/ism/policies.md index eb9531d0..5fc5993e 100644 --- a/_im-plugin/ism/policies.md +++ b/_im-plugin/ism/policies.md @@ -200,7 +200,7 @@ Deletes a managed index. Rolls an alias over to a new index when the managed index meets one of the rollover conditions. -**Important**: ISM checks the conditions for operations on **every execution of the policy** based on the **set interval**, _not_ continuously. The rollover will not be performed before the value is **above** the configured limit **when the check is performed**. For example for `min_size`, with a value of 100GiB, ISM might check the index at 99 GiB, not perform the rollover, check again when the shards reach 105 GiB, and then perform the operation. +**Important**: ISM checks the conditions for operations on **every execution of the policy** based on the **set interval**, _not_ continuously. The rollover will not be performed before the value **has reached** or exceeded the configured limit **when the check is performed**. For example for `min_size`, with a value of 100GiB, ISM might check the index at 99 GiB, not perform the rollover, check again when the shards reach 105 GiB, and then perform the operation. The index format must match the pattern: `^.*-\d+$`. For example, `(logs-000001)`. Set `index.plugins.index_state_management.rollover_alias` as the alias to rollover. From 53337576b718b7fee8870f5138cf96c7b5b4ba3c Mon Sep 17 00:00:00 2001 From: Stein Arne Storslett Date: Mon, 23 May 2022 09:40:38 +0200 Subject: [PATCH 5/6] Add doc on action rollover min_primary_shard_size language. Signed-off-by: Stein Arne Storslett --- _im-plugin/ism/policies.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_im-plugin/ism/policies.md b/_im-plugin/ism/policies.md index 5fc5993e..4aa75613 100644 --- a/_im-plugin/ism/policies.md +++ b/_im-plugin/ism/policies.md @@ -200,7 +200,7 @@ Deletes a managed index. Rolls an alias over to a new index when the managed index meets one of the rollover conditions. -**Important**: ISM checks the conditions for operations on **every execution of the policy** based on the **set interval**, _not_ continuously. The rollover will not be performed before the value **has reached** or exceeded the configured limit **when the check is performed**. For example for `min_size`, with a value of 100GiB, ISM might check the index at 99 GiB, not perform the rollover, check again when the shards reach 105 GiB, and then perform the operation. +**Important**: ISM checks the conditions for operations on **every execution of the policy** based on the **set interval**, _not_ continuously. The rollover will not be performed before the value **has reached** or exceeded the configured limit **when the check is performed**. For example with `min_size` configured to a value of 100GiB, ISM might check the index at 99 GiB and not perform the rollover. At the next check the index can have grown to i.e. 105 GiB, and the operation is performed. The index format must match the pattern: `^.*-\d+$`. For example, `(logs-000001)`. Set `index.plugins.index_state_management.rollover_alias` as the alias to rollover. From d81422cfc343553b5605dc3ce2091497fa9f9c74 Mon Sep 17 00:00:00 2001 From: Stein Arne Storslett Date: Tue, 24 May 2022 09:49:45 +0200 Subject: [PATCH 6/6] Add doc on action rollover min_primary_shard_size language. Signed-off-by: Stein Arne Storslett --- _im-plugin/ism/policies.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_im-plugin/ism/policies.md b/_im-plugin/ism/policies.md index 4aa75613..a3d4c88f 100644 --- a/_im-plugin/ism/policies.md +++ b/_im-plugin/ism/policies.md @@ -200,7 +200,7 @@ Deletes a managed index. Rolls an alias over to a new index when the managed index meets one of the rollover conditions. -**Important**: ISM checks the conditions for operations on **every execution of the policy** based on the **set interval**, _not_ continuously. The rollover will not be performed before the value **has reached** or exceeded the configured limit **when the check is performed**. For example with `min_size` configured to a value of 100GiB, ISM might check the index at 99 GiB and not perform the rollover. At the next check the index can have grown to i.e. 105 GiB, and the operation is performed. +**Important**: ISM checks the conditions for operations on **every execution of the policy** based on the **set interval**, _not_ continuously. The rollover will be performed if the value **has reached** or _exceeded_ the configured limit **when the check is performed**. For example with `min_size` configured to a value of 100GiB, ISM might check the index at 99 GiB and not perform the rollover. However, if the index has grown past the limit (e.g. 105GiB) by the next check, the operation is performed. The index format must match the pattern: `^.*-\d+$`. For example, `(logs-000001)`. Set `index.plugins.index_state_management.rollover_alias` as the alias to rollover.