From 30a713cc3f8a890090153e76a2afaa1080344150 Mon Sep 17 00:00:00 2001 From: Tal Levy Date: Wed, 14 Nov 2018 09:23:50 -0800 Subject: [PATCH] [ILM] add docs explaining high-level concepts (#35322) add docs explaining high-level concepts --- docs/reference/ilm/index.asciidoc | 4 +- .../reference/ilm/policy-definitions.asciidoc | 549 ++++++++++++++++++ 2 files changed, 552 insertions(+), 1 deletion(-) create mode 100644 docs/reference/ilm/policy-definitions.asciidoc diff --git a/docs/reference/ilm/index.asciidoc b/docs/reference/ilm/index.asciidoc index a542aa61094..3aa8921d8bc 100644 --- a/docs/reference/ilm/index.asciidoc +++ b/docs/reference/ilm/index.asciidoc @@ -52,10 +52,12 @@ hardware. include::getting-started-ilm.asciidoc[] -include::using-policies-rollover.asciidoc[] +include::policy-definitions.asciidoc[] include::set-up-lifecycle-policy.asciidoc[] +include::using-policies-rollover.asciidoc[] + include::update-lifecycle-policy.asciidoc[] include::get-index-lifecycle-information.asciidoc[] diff --git a/docs/reference/ilm/policy-definitions.asciidoc b/docs/reference/ilm/policy-definitions.asciidoc new file mode 100644 index 00000000000..e51d80d0261 --- /dev/null +++ b/docs/reference/ilm/policy-definitions.asciidoc @@ -0,0 +1,549 @@ +[role="xpack"] +[testenv="basic"] +[[ilm-policy-definition]] +== Policy Phases and Actions + +There are four stages in the index lifecycle, in the order +they are executed. + +[options="header"] +|====== +| Name | Description +| `hot` | The index is actively being written to +| `warm` | The index is generally not being written to, but is still queried +| `cold` | The index is no longer being updated and is seldom queried. The +information still needs to be searchable, but it's okay if those queries are +slower. +| `delete` | The index is no longer needed and can safely be deleted +|====== + +Each of these stages is called a "phase". A policy does not need to configure +each phase for an index. For example, one policy may define only the hot +phase and the delete phase, while another may define all four phases. + +=== Timing + +Indices enter phases based on a phase's `min_age` parameter. +The index will not enter the phase until the index's age is older than that +of the `min_age`. The parameter is configured using a time +duration format (see <>). + +`min_age` defaults to zero seconds `0s` for each phase if not specified. + +[source,js] +-------------------------------------------------- +PUT _ilm/policy/my_policy +{ + "policy": { + "phases": { + "warm": { + "min_age": "1d", + "actions": { + "allocate": { + "number_of_replicas": 1 + } + } + }, + "delete": { + "min_age": "30d", + "actions": { + "delete": {} + } + } + } + } +} +-------------------------------------------------- +// CONSOLE + +The Above example configures a policy that moves the index into the warm +phase after one day. Until then, the index is in a waiting state. After +moving into the warm phase, it will wait until 30 days have elapsed before +moving to the delete phase and deleting the index. + +`min_age` is usually the time elapsed from the time the index is created. If the +index is rolled over, then `min_age` is the time elapsed from the time the index +is rolled over. The intention here is to execute following phases and actions +relative to when data was written last to a rolled over index. + +=== Actions + +The below list shows the actions which are available in each phase. + +* Hot + - <> +* Warm + - <> + - <> + - <> + - <> +* Cold + - <> +* Delete + - <> + +[[ilm-allocate-action]] +==== Allocate + +Phases allowed: warm, cold. + +The Allocate action allows you to specify which nodes are allowed to host the +shards of the index and set the number of replicas. +Behind the scenes, it is modifying the index settings +for shard filtering and/or replica counts. When updating the number of replicas, +configuring allocation rules is optional. When configuring allocation rules, +setting number of replicas is optional. Although this action can be treated as +two separate index settings updates, both can be configured at once. + +Read more about index replicas <>. +Read more about shard allocation filtering in +the <>. + +[[ilm-allocate-options]] +.Allocate Options +[options="header"] +|====== +| Name | Required | Default | Description +| `number_of_replicas` | no | - | The number of replicas to + assign to the index +| `include` | no | - | assigns an index to nodes + having at least _one_ of the attributes +| `exclude` | no | - | assigns an index to nodes having + _none_ of the attributes +| `require` | no | - | assigns an index to nodes having + _all_ of the attributes +|====== + +If `number_of_replicas` is not configured, then at least one of `include`, +`exclude`, and `require` is required. An empty Allocate Action with no configuration +is invalid. + +===== Example: Change number of replicas + +In this example, the index's number of replicas is changed to `2`, while allocation +rules are unchanged. + +[source,js] +-------------------------------------------------- +PUT _ilm/policy/my_policy +{ + "policy": { + "phases": { + "warm": { + "actions": { + "allocate" : { + "number_of_replicas" : 2 + } + } + } + } + } +} +-------------------------------------------------- +// CONSOLE + +===== Example: Assign index to node with specific "box_type" attribute + +This example assigns the index to nodes with `box_type` attribute of "hot" or "warm". + +[source,js] +-------------------------------------------------- +PUT _ilm/policy/my_policy +{ + "policy": { + "phases": { + "warm": { + "actions": { + "allocate" : { + "include" : { + "box_type": "hot,warm" + } + } + } + } + } + } +} +-------------------------------------------------- +// CONSOLE + +===== Example: Assign index to a specific node and update replica settings + +This example updates the index to have one replica per shard and be allocated +to nodes with a `box_type` attribute of "cold". + +[source,js] +-------------------------------------------------- +PUT _ilm/policy/my_policy +{ + "policy": { + "phases": { + "warm": { + "actions": { + "allocate" : { + "number_of_replicas": 1, + "require" : { + "box_type": "cold" + } + } + } + } + } + } +} +-------------------------------------------------- +// CONSOLE + +[[ilm-delete-action]] +==== Delete + +Phases allowed: delete. + +The Delete Action does just that, it deletes the index. + +This action does not have any options associated with it. + +[source,js] +-------------------------------------------------- +PUT _ilm/policy/my_policy +{ + "policy": { + "phases": { + "delete": { + "actions": { + "delete" : { } + } + } + } + } +} +-------------------------------------------------- +// CONSOLE + +[[ilm-forcemerge-action]] +==== Force Merge + +Phases allowed: warm. + +NOTE: Index will be be made read-only when this action is run +(see: <>) + +The Force Merge Action <> the index into at +most a specific number of <>. + +[[ilm-forcemerge-options]] +.Force Merge Options +[options="header"] +|====== +| Name | Required | Default | Description +| `max_num_segments` | yes | - | The number of + segments to merge to. + To fully merge the + index, set it to `1` +|====== + +[source,js] +-------------------------------------------------- +PUT _ilm/policy/my_policy +{ + "policy": { + "phases": { + "warm": { + "actions": { + "forcemerge" : { + "max_num_segments": 1 + } + } + } + } + } +} +-------------------------------------------------- +// CONSOLE + +[[ilm-readonly-action]] +==== Read-Only + +Phases allowed: warm. + +This action will set the index to be read-only +(see: <>) + +This action does not have any options associated with it. + +[source,js] +-------------------------------------------------- +PUT _ilm/policy/my_policy +{ + "policy": { + "phases": { + "warm": { + "actions": { + "readonly" : { } + } + } + } + } +} +-------------------------------------------------- +// CONSOLE + +[[ilm-rollover-action]] +==== Rollover + +Phases allowed: hot. + +[WARNING] +index format must match pattern '^.*-\\d+$', for example (`logs-000001`). +[WARNING] +The managed index must set `index.lifecycle.rollover_alias` as the +alias to rollover. The index must also be the write index for the alias. + +For example, if an index to be managed has an alias `my_data`. The managed +index "my_index" must be the write index for the alias. For more information, read +<>. + +[source,js] +-------------------------------------------------- +PUT my_index +{ + "settings": { + "index.lifecycle.name": "my_policy", + "index.lifecycle.rollover_alias": "my_data" + }, + "aliases": { + "my_data": { + "is_write_index": true + } + } +} +-------------------------------------------------- +// CONSOLE + +The Rollover Action rolls an alias over to a new index when the +existing index meets one of the rollover conditions. + + +[[ilm-rollover-options]] +.Rollover Options +[options="header"] +|====== +| Name | Required | Default | Description +| `max_size` | no | - | max index storage size. + See <> + for formatting +| `max_docs` | no | - | max number of documents an + index is to contain before + rolling over. +| `max_age` | no | - | max time elapsed from index + creation. See + <> + for formatting +|====== + +At least one of `max_size`, `max_docs`, `max_age` or any combinations of the +three are required to be specified. + +===== Example: Rollover when index is too large + +This example rolls the index over when it is at least 100 gigabytes. + +[source,js] +-------------------------------------------------- +PUT _ilm/policy/my_policy +{ + "policy": { + "phases": { + "hot": { + "actions": { + "rollover" : { + "max_size": "100GB" + } + } + } + } + } +} +-------------------------------------------------- +// CONSOLE + +===== Example: Rollover when index has too many documents + +This example rolls the index over when it contains at least +1000000 documents. + +[source,js] +-------------------------------------------------- +PUT _ilm/policy/my_policy +{ + "policy": { + "phases": { + "hot": { + "actions": { + "rollover" : { + "max_docs": 1000000 + } + } + } + } + } +} +-------------------------------------------------- +// CONSOLE + +===== Example: Rollover when index is too old + +This example rolls the index over when it has been created at least +7 days ago. + +[source,js] +-------------------------------------------------- +PUT _ilm/policy/my_policy +{ + "policy": { + "phases": { + "hot": { + "actions": { + "rollover" : { + "max_age": "7d" + } + } + } + } + } +} +-------------------------------------------------- +// CONSOLE + +===== Example: Rollover when index is too old or too large + +This example rolls the index over when it has been created at least +7 days ago or it is at least 100 gigabytes. In this case, the index will be +rolled over when any of the conditions is met. + +[source,js] +-------------------------------------------------- +PUT _ilm/policy/my_policy +{ + "policy": { + "phases": { + "hot": { + "actions": { + "rollover" : { + "max_age": "7d", + "max_size": "100GB" + } + } + } + } + } +} +-------------------------------------------------- +// CONSOLE + + +[[ilm-shrink-action]] +==== Shrink + +NOTE: Index will be be made read-only when this action is run +(see: <>) + +This action shrinks an existing index into a new index with fewer primary +shards. It calls the <> to shrink the index. +Since allocating all the primary shards of the index to one node is a +prerequisite, this action will first allocate the primary shards to a valid +node. After shrinking, it will swap aliases pointing to the original index +into the new shrunken index. The new index will also have a new name: +"shrink-". So if the original index was called "logs", +then the new index will be named "shrink-logs". + +[[ilm-shrink-options]] +.Shrink Options +[options="header"] +|====== +| Name | Required | Default | Description +| `number_of_shards` | yes | - | The number of shards + to shrink to. must be + a factor of the number + of shards in the + source index. +|====== + +[source,js] +-------------------------------------------------- +PUT _ilm/policy/my_policy +{ + "policy": { + "phases": { + "warm": { + "actions": { + "shrink" : { + "number_of_shards": 1 + } + } + } + } + } +} +-------------------------------------------------- +// CONSOLE + + +=== Full Policy + +With all of these actions, we can support complex management strategies for our +indices. This policy will define an index that will start in the hot phase, +rolling over every 20g or 7 days. After 30 days it enters the warm phase +and increases the replicas to 2, force merges and shrinks. After 60 days +it enters the cold phase and allocates to "cold" nodes, and after 90 days the +index is deleted. + +[source,js] +-------------------------------------------------- +PUT _ilm/policy/full_policy +{ + "policy": { + "phases": { + "hot": { + "actions": { + "rollover": { + "max_age": "7d", + "max_size": "20G" + } + } + }, + "warm": { + "min_age": "30d", + "actions": { + "forcemerge": { + "max_num_segments": 1 + }, + "shrink": { + "number_of_shards": 1 + }, + "allocate": { + "number_of_replicas": 2 + } + } + }, + "cold": { + "min_age": "60d", + "actions": { + "allocate": { + "require": { + "type": "cold" + } + } + } + }, + "delete": { + "min_age": "90d", + "actions": { + "delete": {} + } + } + } + } +} +-------------------------------------------------- +// CONSOLE