diff --git a/docs/reference/ilm/actions/_ilm-action-template.asciidoc b/docs/reference/ilm/actions/_ilm-action-template.asciidoc new file mode 100644 index 00000000000..29ce79ceb49 --- /dev/null +++ b/docs/reference/ilm/actions/_ilm-action-template.asciidoc @@ -0,0 +1,94 @@ +//// +This is a template for ILM action reference documentation. + +To document a new action, copy this file, remove comments like this, and +replace "sample" with the appropriate action name. + +Ensure the new action docs are linked and included in +docs/reference/ilm/actions.asciidoc +//// + +[role="xpack"] +[[ilm-sample]] +=== Sample + +Phases allowed: hot, warm, cold, delete. + +//// +INTRO +Include a brief, 1-2 sentence description. +//// + +Does a cool thing. + +[[ilm-sample-options]] +==== Options + +//// +Definition list of the options that can be specified for the action: + +If there are no options: + +None. +//// + +`sample_option1`:: +(Optional, integer) +Number of something. + +`sample_option2`:: +(Required, string) +Name of something. + +[[ilm-sample-ex]] +==== Example + +//// +Basic example of configuring the action in an ILM policy. + +Additional examples are optional. +//// + +[source,console] +-------------------------------------------------- +PUT _ilm/policy/my_policy +{ + "policy": { + "phases": { + "warm": { + "actions": { + "sample" : { + "sample_option1" : 2 + } + } + } + } + } +} +-------------------------------------------------- +// TEST[skip: Replace fake actions and remove this comment.] + +[[ilm-sample2-ex]] +===== Describe example + +The sample action in the following policy does something interesting. + +[source,console] +-------------------------------------------------- +PUT _ilm/policy/my_policy +{ + "policy": { + "phases": { + "warm": { + "actions": { + "sample" : { + "sample_option1" : 100, + "sample_option2" : "interesting" + } + } + } + } + } +} +-------------------------------------------------- +// TEST[skip: Replace fake actions and remove this comment.] diff --git a/docs/reference/ilm/actions/ilm-allocate.asciidoc b/docs/reference/ilm/actions/ilm-allocate.asciidoc new file mode 100644 index 00000000000..69ca1fcce87 --- /dev/null +++ b/docs/reference/ilm/actions/ilm-allocate.asciidoc @@ -0,0 +1,130 @@ +[role="xpack"] +[[ilm-allocate]] +=== Allocate + +Phases allowed: warm, cold. + +Updates the index settings to change which nodes are allowed to host the index shards +and change the number of replicas. + +You can configure this action to modify both the allocation rules and number of replicas, +only the allocation rules, or only the number of replicas. + +For more information about how {es} uses replicas for scaling, see +<>. See <> for more information about +controlling where {es} allocates shards of a particular index. + +NOTE: The allocate action is not allowed in the hot phase. +The initial allocation for the index must be done manually or via index templates. +{ilm-init} doesn't handle index allocation during the hot phase. + +[[ilm-allocate-options]] +==== Options + +You must specify the number of replicas or at least one +`include`, `exclude`, or `require` option. +An empty allocate action is invalid. + +For more information about using custom attributes for shard allocation, +see <>. + +`number_of_replicas`:: +(Optional, integer) +Number of replicas to assign to the index. + +`include`:: +(Optional, object) +Assigns an index to nodes that have at least _one_ of the specified custom attributes. + +`exclude`:: +(Optional, object) +Assigns an index to nodes that have _none_ of the specified custom attributes. + +`require`:: +(Optional, object) +Assigns an index to nodes that have _all_ of the specified custom attributes. + +[[ilm-allocate-ex]] +==== Example + +The allocate action in the following policy changes the index's number of replicas to `2`. +The index allocation rules are not changed. + +[source,console] +-------------------------------------------------- +PUT _ilm/policy/my_policy +{ + "policy": { + "phases": { + "warm": { + "actions": { + "allocate" : { + "number_of_replicas" : 2 + } + } + } + } + } +} +-------------------------------------------------- + +[[ilm-allocate-assign-index-attribute-ex]] +===== Assign index to nodes using a custom attribute + +The allocate action in the following policy assigns the index to nodes +that have a `box_type` of _hot_ or _warm_. + +To designate a node's `box_type`, you set a custom attribute in the node configuration. +For example, set `node.attr.box_type: hot` in `elasticsearch.yml`. +For more information, see <>. + +[source,console] +-------------------------------------------------- +PUT _ilm/policy/my_policy +{ + "policy": { + "phases": { + "warm": { + "actions": { + "allocate" : { + "include" : { + "box_type": "hot,warm" + } + } + } + } + } + } +} +-------------------------------------------------- + +[[ilm-allocate-assign-index-node-ex]] +===== Assign index to a specific node and update replica settings + +The allocate action in the following policy updates the index to have one replica per shard +and be allocated to nodes that have a `box_type` of _cold_. + +To designate a node's `box_type`, you set a custom attribute in the node configuration. +For example, set `node.attr.box_type: cold` in `elasticsearch.yml`. +For more information, see <>. + +[source,console] +-------------------------------------------------- +PUT _ilm/policy/my_policy +{ + "policy": { + "phases": { + "warm": { + "actions": { + "allocate" : { + "number_of_replicas": 1, + "require" : { + "box_type": "cold" + } + } + } + } + } + } +} +-------------------------------------------------- diff --git a/docs/reference/ilm/actions/ilm-delete.asciidoc b/docs/reference/ilm/actions/ilm-delete.asciidoc new file mode 100644 index 00000000000..3de5f1e4290 --- /dev/null +++ b/docs/reference/ilm/actions/ilm-delete.asciidoc @@ -0,0 +1,36 @@ +[role="xpack"] +[[ilm-delete]] +=== Delete + +Phases allowed: delete. + +Permanently removes the index. + +[[ilm-delete-options]] +==== Options + +`delete_searchable_snapshot`:: +(Optional, boolean) +Deletes the searchable snapshot created in the cold phase. +Defaults to `true`. +This option is applicable when the <> action is used in the cold phase. + +[[ilm-delete-action-ex]] +==== Example + +[source,console] +-------------------------------------------------- +PUT _ilm/policy/my_policy +{ + "policy": { + "phases": { + "delete": { + "actions": { + "delete" : { } + } + } + } + } +} +-------------------------------------------------- diff --git a/docs/reference/ilm/actions/ilm-forcemerge.asciidoc b/docs/reference/ilm/actions/ilm-forcemerge.asciidoc new file mode 100644 index 00000000000..86f8d579a17 --- /dev/null +++ b/docs/reference/ilm/actions/ilm-forcemerge.asciidoc @@ -0,0 +1,52 @@ +[role="xpack"] +[[ilm-forcemerge]] +=== Force merge + +Phases allowed: hot, warm. + +<> the index into +the specified maximum number of <>. + +This action makes the index <>. + +To use the `forcemerge` action in the `hot` phase, the `rollover` action *must* be present. +If no rollover action is configured, {ilm-init} will reject the policy. + +[[ilm-forcemerge-options]] +==== Options + +`max_num_segments`:: +(Required, integer) +Number of segments to merge to. To fully merge the index, set to `1`. + +`codec`:: +(Optional, string) +Use the `best_compression` codec. Valid values: `best_compression`. ++ +[WARNING] +====== +Setting `"codec": "best_compression"` in the {ilm-init} forcemerge action causes {ilm-int} to +<> and then <> the index prior to the force merge. +During this time, the index is unavailable for both read and write operations. +====== + +[[ilm-forcemerge-action-ex]] +==== Example + +[source,console] +-------------------------------------------------- +PUT _ilm/policy/my_policy +{ + "policy": { + "phases": { + "warm": { + "actions": { + "forcemerge" : { + "max_num_segments": 1 + } + } + } + } + } +} +-------------------------------------------------- diff --git a/docs/reference/ilm/actions/ilm-freeze.asciidoc b/docs/reference/ilm/actions/ilm-freeze.asciidoc new file mode 100644 index 00000000000..960add463a2 --- /dev/null +++ b/docs/reference/ilm/actions/ilm-freeze.asciidoc @@ -0,0 +1,40 @@ +[role="xpack"] +[[ilm-freeze]] +=== Freeze + +Phases allowed: cold. + +<> an index to minimize its memory footprint. + +[IMPORTANT] +================================ + Freezing an index closes the index and reopens it within the same API call. + This means that for a short time no primaries are allocated. + The cluster will go red until the primaries are allocated. + This limitation might be removed in the future. +================================ + +[[ilm-freeze-options]] +==== Options + +None. + +[[ilm-freeze-ex]] +==== Example + +[source,console] +-------------------------------------------------- +PUT _ilm/policy/my_policy +{ + "policy": { + "phases": { + "cold": { + "actions": { + "freeze" : { } + } + } + } + } +} +-------------------------------------------------- + diff --git a/docs/reference/ilm/actions/ilm-readonly.asciidoc b/docs/reference/ilm/actions/ilm-readonly.asciidoc new file mode 100644 index 00000000000..3dc3cac3cdc --- /dev/null +++ b/docs/reference/ilm/actions/ilm-readonly.asciidoc @@ -0,0 +1,31 @@ +[role="xpack"] +[[ilm-readonly]] +=== Read only + +Phases allowed: warm. + +Makes the index <>. + +[[ilm-read-only-options]] +==== Options + +None. + +[[ilm-read-only-ex]] +==== Example + +[source,console] +-------------------------------------------------- +PUT _ilm/policy/my_policy +{ + "policy": { + "phases": { + "warm": { + "actions": { + "readonly" : { } + } + } + } + } +} +-------------------------------------------------- diff --git a/docs/reference/ilm/actions/ilm-rollover.asciidoc b/docs/reference/ilm/actions/ilm-rollover.asciidoc new file mode 100644 index 00000000000..acfdb362fc4 --- /dev/null +++ b/docs/reference/ilm/actions/ilm-rollover.asciidoc @@ -0,0 +1,187 @@ +[role="xpack"] +[[ilm-rollover]] +=== Rollover + +Phases allowed: hot. + +Rolls an alias over to a new index when the existing index meets one of the rollover conditions. + +IMPORTANT: If the rollover action is used on a <>, +policy execution waits until the leader index rolls over (or is +<>), +then converts the follower index into a regular index with the +<>. + +For a managed index to be rolled over: + +* The index name must match the pattern '^.*-\\d+$', for example (`my_index-000001`). +* `index.lifecycle.rollover_alias` must be configured as the alias to roll over. +* The index must be the <<<> for the alias. + +For example, if `my_index-000001` has the alias `my_data`, +the following settings must be configured. + +[source,console] +-------------------------------------------------- +PUT my_index-000001 +{ + "settings": { + "index.lifecycle.name": "my_policy", + "index.lifecycle.rollover_alias": "my_data" + }, + "aliases": { + "my_data": { + "is_write_index": true + } + } +} +-------------------------------------------------- + +[[ilm-rollover-options]] +==== Options + +You must specify at least one rollover condition. +An empty rollover action is invalid. + +`max_size`:: +(Optional, <>) +Triggers roll over after the specified maximum primary shard index storage size is reached. + +`max_docs`:: +(Optional, integer) +Triggers roll over after the specified maximum number of documents in the index is reached. + +`max_age`:: +(Optional, <>) +Triggers roll over after the maximum elapsed time from index creation is reached. + +[[ilm-rollover-ex]] +==== Example + +[[ilm-rollover-size-ex]] +===== Roll over based on index size + +This example rolls the index over when it is at least 100 gigabytes. + +[source,console] +-------------------------------------------------- +PUT _ilm/policy/my_policy +{ + "policy": { + "phases": { + "hot": { + "actions": { + "rollover" : { + "max_size": "100GB" + } + } + } + } + } +} +-------------------------------------------------- + +[ilm-rollover-documents-ex]] +===== Roll over based on document count + +This example rolls the index over when it contains at least one hundred million documents. + +[source,console] +-------------------------------------------------- +PUT _ilm/policy/my_policy +{ + "policy": { + "phases": { + "hot": { + "actions": { + "rollover" : { + "max_docs": 100000000 + } + } + } + } + } +} +-------------------------------------------------- + +[ilm-rollover-age-ex]] +===== Roll over based on index age + +This example rolls the index over if it was created at least 7 days ago. + +[source,console] +-------------------------------------------------- +PUT _ilm/policy/my_policy +{ + "policy": { + "phases": { + "hot": { + "actions": { + "rollover" : { + "max_age": "7d" + } + } + } + } + } +} +-------------------------------------------------- + +[ilm-rollover-conditions-ex]] +===== Roll over using multiple conditions + +When you specify multiple rollover conditions, +the index is rolled over when _any_ of the conditions are met. +This example rolls the index over if it is at least 7 days old or at least 100 gigabytes. + +[source,console] +-------------------------------------------------- +PUT _ilm/policy/my_policy +{ + "policy": { + "phases": { + "hot": { + "actions": { + "rollover" : { + "max_age": "7d", + "max_size": "100GB" + } + } + } + } + } +} +-------------------------------------------------- + +[ilm-rollover-block-ex]] +===== Rollover condition blocks phase transition + +The rollover action only completes if one of its conditions is met. +This means that any subsequent phases are blocked until rollover succeeds. + +For example, the following policy deletes the index one day after it rolls over. +It does not delete the index one day after it was created. + +[source,console] +-------------------------------------------------- +PUT /_ilm/policy/rollover_policy +{ + "policy": { + "phases": { + "hot": { + "actions": { + "rollover": { + "max_size": "50G" + } + } + }, + "delete": { + "min_age": "1d", + "actions": { + "delete": {} + } + } + } + } +} +-------------------------------------------------- diff --git a/docs/reference/ilm/actions/ilm-searchable-snapshot.asciidoc b/docs/reference/ilm/actions/ilm-searchable-snapshot.asciidoc new file mode 100644 index 00000000000..7f2fc7b477a --- /dev/null +++ b/docs/reference/ilm/actions/ilm-searchable-snapshot.asciidoc @@ -0,0 +1,39 @@ +[role="xpack"] +[[ilm-searchable-snapshot]] +=== Searchable snapshot + +Phases allowed: cold. + +Takes a snapshot of the managed index in the configured repository +and mounts it as a searchable snapshot. + +By default, this snapshot is deleted by the <> in the delete phase. +To keep the snapshot, set `delete_searchable_snapshot` to `false` in the delete action. + +[[ilm-searchable-snapshot-options]] +==== Options + +`snapshot_repository`:: +(Required, string) +Specifies where to store the snapshot. +See <> for more information. + +[[ilm-searchable-snapshot-ex]] +==== Examples +[source,console] +-------------------------------------------------- +PUT _ilm/policy/my_policy +{ + "policy": { + "phases": { + "cold": { + "actions": { + "searchable_snapshot" : { + "snapshot_repository" : "backing_repo" + } + } + } + } + } +} +-------------------------------------------------- diff --git a/docs/reference/ilm/actions/ilm-set-priority.asciidoc b/docs/reference/ilm/actions/ilm-set-priority.asciidoc new file mode 100644 index 00000000000..7f22bb64bdb --- /dev/null +++ b/docs/reference/ilm/actions/ilm-set-priority.asciidoc @@ -0,0 +1,44 @@ +[role="xpack"] +[[ilm-set-priority]] +=== Set priority + +Phases allowed: hot, warm, cold. + +Sets the <> of the index as +soon as the policy enters the hot, warm, or cold phase. +Higher priority indices are recovered before indices with lower priorities following a node restart. + +Generally, indexes in the hot phase should have the highest value and +indexes in the cold phase should have the lowest values. +For example: 100 for the hot phase, 50 for the warm phase, and 0 for the cold phase. +Indices that don't set this value have a default priority of 1. + +[[ilm-set-priority-options]] +==== Options + +`priority`:: +(Required, integer) +The priority for the index. +Must be 0 or greater. +Set to `null` to remove the priority. + +[[ilm-set-priority-ex]] +==== Example + +[source,console] +-------------------------------------------------- +PUT _ilm/policy/my_policy +{ + "policy": { + "phases": { + "warm": { + "actions": { + "set_priority" : { + "priority": 50 + } + } + } + } + } +} +-------------------------------------------------- diff --git a/docs/reference/ilm/actions/ilm-shrink.asciidoc b/docs/reference/ilm/actions/ilm-shrink.asciidoc new file mode 100644 index 00000000000..d13e8bf41cf --- /dev/null +++ b/docs/reference/ilm/actions/ilm-shrink.asciidoc @@ -0,0 +1,51 @@ +[role="xpack"] +[[ilm-shrink]] +=== Shrink + +Phases allowed: warm + +Sets an index to <> +and shrinks it into a new index with fewer primary shards. +The name of the new index is of the form `shrink-`. +For example, if the name of the source index is _logs_, +the name of the shrunken index is _shrink-logs_. + +The shrink action allocates all primary shards of the index to one node so it +can call the <> to shrink the index. +After shrinking, it swaps aliases that point to the original index to the new shrunken index. + +[IMPORTANT] +If the shrink action is used on a <>, +policy execution waits until the leader index rolls over (or is +<>), +then converts the follower index into a regular index with the +<> before performing the shrink operation. + +[[ilm-shrink-options]] +==== Shrink options +`number_of_shards`:: +(Required, integer) +Number of shards to shrink to. +Must be a factor of the number of shards in the source index. + + +[[ilm-shrink-ex]] +==== Example + +[source,console] +-------------------------------------------------- +PUT _ilm/policy/my_policy +{ + "policy": { + "phases": { + "warm": { + "actions": { + "shrink" : { + "number_of_shards": 1 + } + } + } + } + } +} +-------------------------------------------------- \ No newline at end of file diff --git a/docs/reference/ilm/actions/ilm-unfollow.asciidoc b/docs/reference/ilm/actions/ilm-unfollow.asciidoc new file mode 100644 index 00000000000..5607f67a2c8 --- /dev/null +++ b/docs/reference/ilm/actions/ilm-unfollow.asciidoc @@ -0,0 +1,58 @@ +[role="xpack"] +[[ilm-unfollow]] +=== Unfollow + +Phases allowed: hot, warm, cold. + +Converts a {ref}/ccr-apis.html[{ccr-init}] follower index into a regular index. +This enables the shrink, rollover, and searchable snapshot actions +to be be performed safely on follower indices. +You can also use unfollow directly when moving follower indices through the lifecycle. +Has no effect on indices that are not followers, phase execution just moves to the next action. + +[NOTE] +This action is triggered automatically by the +<>, <>, +and <> actions +when they are applied to follower indices. + +This action waits until is it safe to convert a follower index into a regular index. +The following conditions must be met: + +* The leader index must have `index.lifecycle.indexing_complete` set to `true`. +This happens automatically if the leader index is rolled over using the +<> action, and can be set manually using +the <> API. +* All operations performed on the leader index have been replicated to the follower index. +This ensures that no operations are lost when the index is converted. + +Once these conditions are met, unfollow performs the following operations: + +* Pauses indexing following for the follower index. +* Closes the follower index. +* Unfollows the leader index. +* Opens the follower index (which is at this point is a regular index). + +[[ilm-unfollow-options]] +==== Options + +None. + +[[ilm-unfollow-ex]] +==== Example + +[source,console] +-------------------------------------------------- +PUT _ilm/policy/my_policy +{ + "policy": { + "phases": { + "hot": { + "actions": { + "unfollow" : {} + } + } + } + } +} +-------------------------------------------------- \ No newline at end of file diff --git a/docs/reference/ilm/actions/ilm-wait-for-snapshot.asciidoc b/docs/reference/ilm/actions/ilm-wait-for-snapshot.asciidoc new file mode 100644 index 00000000000..5a953417a64 --- /dev/null +++ b/docs/reference/ilm/actions/ilm-wait-for-snapshot.asciidoc @@ -0,0 +1,36 @@ +[role="xpack"] +[[ilm-wait-for-snapshot]] +=== Wait for snapshot + +Phases allowed: delete. + +Waits for the specified {slm-init} policy to be executed before removing the index. +This ensures that a snapshot of the deleted index is available. + +[[ilm-wait-for-snapshot-options]] +==== Options + +`policy`:: +(Required, string) +Name of the {slm-init} policy that the delete action should wait for. + +[[ilm-wait-for-snapshot-ex]] +==== Example + +[source,console] +-------------------------------------------------- +PUT _ilm/policy/my_policy +{ + "policy": { + "phases": { + "delete": { + "actions": { + "wait_for_snapshot" : { + "policy": "slm-policy-name" + } + } + } + } + } +} +-------------------------------------------------- \ No newline at end of file diff --git a/docs/reference/ilm/apis/explain.asciidoc b/docs/reference/ilm/apis/explain.asciidoc index b95357c06b9..8bfd9e7a26f 100644 --- a/docs/reference/ilm/apis/explain.asciidoc +++ b/docs/reference/ilm/apis/explain.asciidoc @@ -39,11 +39,11 @@ about any failures. `only_managed`:: (Optional, boolean) Filters the returned indices to only indices that are managed by - ILM. + {ilm-init}. `only_errors`:: (Optional, boolean) Filters the returned indices to only indices that are managed by - ILM and are in an error state, either due to an encountering an error while + {ilm-init} and are in an error state, either due to an encountering an error while executing the policy, or attempting to use a policy that does not exist. include::{docdir}/rest-api/common-parms.asciidoc[tag=timeoutparms] @@ -99,7 +99,7 @@ GET my_index/_ilm/explain -------------------------------------------------- // TEST[continued] -When management of the index is first taken over by ILM, `explain` shows +When management of the index is first taken over by {ilm-init}, `explain` shows that the index is managed and in the `new` phase: [source,console-result] @@ -124,9 +124,9 @@ that the index is managed and in the `new` phase: -------------------------------------------------- // TESTRESPONSE[skip:no way to know if we will get this response immediately] -<1> Shows if the index is being managed by ILM. If the index is not managed by -ILM the other fields will not be shown -<2> The name of the policy which ILM is using for this index +<1> Shows if the index is being managed by {ilm-init}. If the index is not managed by +{ilm-init} the other fields will not be shown +<2> The name of the policy which {ilm-init} is using for this index <3> The timestamp used for the `min_age` <4> The age of the index (used for calculating when to enter the next phase) <5> When the index entered the current phase @@ -302,7 +302,7 @@ the case. <1> The step that caused the error <2> Indicates if retrying the failed step can overcome the error. If this -is true, ILM will retry the failed step automatically. +is true, {ilm-init} will retry the failed step automatically. <3> Shows the number of attempted automatic retries to execute the failed step. <4> What went wrong diff --git a/docs/reference/ilm/apis/get-status.asciidoc b/docs/reference/ilm/apis/get-status.asciidoc index e5970e7e05a..600496da222 100644 --- a/docs/reference/ilm/apis/get-status.asciidoc +++ b/docs/reference/ilm/apis/get-status.asciidoc @@ -28,7 +28,7 @@ Retrieves the current {ilm} ({ilm-init}) status. Returns the status of the {ilm-init} plugin. The `operation_mode` field in the response shows one of three states: `STARTED`, `STOPPING`, or `STOPPED`. You can change the status of the {ilm-init} plugin with the -<> and <> APIs. +<> and <> APIs. [[ilm-get-status-query-params]] ==== {api-query-parms-title} diff --git a/docs/reference/ilm/apis/ilm-api.asciidoc b/docs/reference/ilm/apis/ilm-api.asciidoc index e2b1f74020b..94f6db52366 100644 --- a/docs/reference/ilm/apis/ilm-api.asciidoc +++ b/docs/reference/ilm/apis/ilm-api.asciidoc @@ -4,7 +4,7 @@ You use the following APIs to set up policies to automatically manage the index lifecycle. For more information about {ilm} ({ilm-init}), see <>. -[float] +[discrete] [[ilm-api-policy-endpoint]] === Policy management APIs @@ -12,7 +12,7 @@ For more information about {ilm} ({ilm-init}), see < * <> * <> -[float] +[discrete] [[ilm-api-index-endpoint]] === Index management APIs @@ -20,13 +20,13 @@ For more information about {ilm} ({ilm-init}), see < * <> * <> -[float] +[discrete] [[ilm-api-management-endpoint]] === Operation management APIs -* <> -* <> -* <> +* <> +* <> +* <> * <> diff --git a/docs/reference/ilm/apis/retry-policy.asciidoc b/docs/reference/ilm/apis/retry-policy.asciidoc index 28f341a2feb..41f0e82675f 100644 --- a/docs/reference/ilm/apis/retry-policy.asciidoc +++ b/docs/reference/ilm/apis/retry-policy.asciidoc @@ -24,7 +24,7 @@ see <>. ==== {api-description-title} Sets the policy back to the step where the error occurred and executes the step. -Use the <> to determine if an index is in the ERROR +Use the <> to determine if an index is in the ERROR step. [[ilm-retry-policy-path-params]] diff --git a/docs/reference/ilm/apis/start.asciidoc b/docs/reference/ilm/apis/start.asciidoc index 1a620924766..fac7ddcbbb1 100644 --- a/docs/reference/ilm/apis/start.asciidoc +++ b/docs/reference/ilm/apis/start.asciidoc @@ -37,7 +37,7 @@ include::{docdir}/rest-api/common-parms.asciidoc[tag=timeoutparms] [[ilm-start-example]] ==== {api-examples-title} -The following example starts the ILM plugin. +The following example starts the {ilm-init} plugin. ////////////////////////// diff --git a/docs/reference/ilm/apis/stop.asciidoc b/docs/reference/ilm/apis/stop.asciidoc index f2f99db3d9a..8f6e71558be 100644 --- a/docs/reference/ilm/apis/stop.asciidoc +++ b/docs/reference/ilm/apis/stop.asciidoc @@ -31,7 +31,7 @@ is useful when you are performing maintenance on the cluster and need to prevent The API returns as soon as the stop request has been acknowledged, but the plugin might continue to run until in-progress operations complete and the plugin -can be safely stopped. Use the <> API to see +can be safely stopped. Use the <> API to see if {ilm-init} is running. [[ilm-stop-query-params]] @@ -42,7 +42,7 @@ include::{docdir}/rest-api/common-parms.asciidoc[tag=timeoutparms] [[ilm-stop-example]] ==== {api-examples-title} -The following example stops the ILM plugin. +The following example stops the {ilm-init} plugin. ////////////////////////// diff --git a/docs/reference/ilm/error-handling.asciidoc b/docs/reference/ilm/error-handling.asciidoc index d3c6c9a18d1..381f2b922cd 100644 --- a/docs/reference/ilm/error-handling.asciidoc +++ b/docs/reference/ilm/error-handling.asciidoc @@ -5,7 +5,7 @@ During Index Lifecycle Management's execution of the policy for an index, it's possible for a step to encounter an error during its execution. When this -happens, ILM will move the management state into an "error" step. This halts +happens, {ilm-init} will move the management state into an "error" step. This halts further execution of the policy and gives an administrator the chance to address any issues with the policy, index, or cluster. @@ -50,11 +50,11 @@ PUT /myindex -------------------------------------------------- // TEST[continued] -After five days have passed, ILM will attempt to shrink this index from 2 +After five days have passed, {ilm-init} will attempt to shrink this index from 2 shards to 4, which is invalid since the shrink action cannot increase the -number of shards. When this occurs, ILM will move this +number of shards. When this occurs, {ilm-init} will move this index to the "error" step. Once an index is in this step, information about the -reason for the error can be retrieved from the <>: +reason for the error can be retrieved from the <>: [source,console] -------------------------------------------------- @@ -104,7 +104,7 @@ Which returns the following information: -------------------------------------------------- // TESTRESPONSE[skip:no way to know if we will get this response immediately] -<1> this index is managed by ILM +<1> this index is managed by {ilm-init} <2> the policy in question, in this case, "shrink-the-index" <3> the current age for the index <4> what phase the index is currently in @@ -140,7 +140,7 @@ PUT _ilm/policy/shrink-the-index -------------------------------------------------- // TEST[continued] -[float] +[discrete] === Retrying failed index lifecycle management steps Once the underlying issue that caused an index to move to the error step has @@ -155,5 +155,5 @@ POST /myindex/_ilm/retry Once this has been issue, index lifecycle management will asynchronously pick up on the step that is in a failed state, attempting to re-run it. The -<> can again be used to monitor the status of +<> can again be used to monitor the status of re-running the step. diff --git a/docs/reference/ilm/ilm-actions.asciidoc b/docs/reference/ilm/ilm-actions.asciidoc new file mode 100644 index 00000000000..422c51ff422 --- /dev/null +++ b/docs/reference/ilm/ilm-actions.asciidoc @@ -0,0 +1,56 @@ +[role="xpack"] +[testenv="basic"] +[[ilm-actions]] +== Index lifecycle actions + +[[ilm-allocate-action]] +<>:: +Move shards to nodes with different performance characteristics +and reduce the number of replicas. + +[[ilm-delete-action]]<>:: +Permanently remove the index. + +[[ilm-forcemerge-action]]<>:: +Reduce the number of index segments and purge deleted documents. +Makes the index read-only. + +[[ilm-freeze-action]]<>:: +Freeze the index to minimize its memory footprint. + +[[ilm-readonly-action]]<>:: +Block write operations to the index. + +[[ilm-rollover-action]]<>:: +Remove the index as the write index for the rollover alias and +start indexing to a new index. + +[[ilm-searchable-snapshot-action]]<>:: +Take a snapshot of the managed index in the configured repository +and mount it as a searchable snapshot. + +[[ilm-set-priority-action]]<>:: +Lower the priority of an index as it moves through the lifecycle +to ensure that hot indices are recovered first. + +[[ilm-shrink-action]]<>:: +Reduce the number of primary shards by shrinking the index into a new index. + +[[ilm-unfollow-action]]<>:: +Convert a follower index to a regular index. +Performed automatically before a rollover, shrink, or searchable snapshot action. + +[[ilm-wait-for-snapshot-action]]<>:: +Ensure that a snapshot exists before deleting the index. + +include::actions/ilm-allocate.asciidoc[] +include::actions/ilm-delete.asciidoc[] +include::actions/ilm-forcemerge.asciidoc[] +include::actions/ilm-freeze.asciidoc[] +include::actions/ilm-readonly.asciidoc[] +include::actions/ilm-rollover.asciidoc[] +include::actions/ilm-searchable-snapshot.asciidoc[] +include::actions/ilm-set-priority.asciidoc[] +include::actions/ilm-shrink.asciidoc[] +include::actions/ilm-unfollow.asciidoc[] +include::actions/ilm-wait-for-snapshot.asciidoc[] diff --git a/docs/reference/ilm/ilm-and-snapshots.asciidoc b/docs/reference/ilm/ilm-and-snapshots.asciidoc index 5de30f11382..7a4f2efa40d 100644 --- a/docs/reference/ilm/ilm-and-snapshots.asciidoc +++ b/docs/reference/ilm/ilm-and-snapshots.asciidoc @@ -25,9 +25,9 @@ paused>> to allow the policy to be removed. For example, the following workflow can be used in the above situation to prevent the execution of the lifecycle policy for an index: -1. Pause execution of all lifecycle policies using the <> +1. Pause execution of all lifecycle policies using the <> 2. Restore the snapshot. 3. Perform whatever operations you wish before resuming lifecycle execution, or remove the lifecycle policy from the index using the <> -4. Resume execution of lifecycle policies using the <> +4. Resume execution of lifecycle policies using the <> diff --git a/docs/reference/ilm/ilm-concepts.asciidoc b/docs/reference/ilm/ilm-concepts.asciidoc new file mode 100644 index 00000000000..5ff31b5b34b --- /dev/null +++ b/docs/reference/ilm/ilm-concepts.asciidoc @@ -0,0 +1,16 @@ +[role="xpack"] +[testenv="basic"] +[[ilm-concepts]] +== {ilm-init} concepts + +++++ +Concepts +++++ + +* <> +* <> +* <> + +include::ilm-index-lifecycle.asciidoc[] +include::index-rollover.asciidoc[] +include::update-lifecycle-policy.asciidoc[] \ No newline at end of file diff --git a/docs/reference/ilm/ilm-index-lifecycle.asciidoc b/docs/reference/ilm/ilm-index-lifecycle.asciidoc new file mode 100644 index 00000000000..1369418a54e --- /dev/null +++ b/docs/reference/ilm/ilm-index-lifecycle.asciidoc @@ -0,0 +1,77 @@ +[role="xpack"] +[testenv="basic"] +[[ilm-index-lifecycle]] +=== Index lifecycle +++++ +Index lifecycle +++++ + +{ilm-init} defines four index lifecycle _phases_: + +* Hot: The index is actively being updated and queried. +* Warm: The index is no longer being updated but is still being queried. +* Cold: The index is no longer being updated and is seldom queried. The +information still needs to be searchable, but it's okay if those queries are +slower. +* Delete: The index is no longer needed and can safely be removed. + +An index's _lifecycle policy_ specifies which phases +are applicable, what actions are performed in each phase, +and when it transitions between phases. + +You can manually apply a lifecycle policy when you create an index. +For time series indices, you need to associate the lifecycle policy with +the index template used to create new indices in the series. +When an index rolls over, a manually-applied policy isn't automatically applied to the new index. + +[discrete] +[[ilm-phase-transitions]] +=== Phase transitions + +{ilm-init} moves indices through the lifecycle according to their age. +To control the timing of these transitions, you set a _minimum age_ for each phase. +For an index to move to the next phase, all actions in the current phase must be complete and +the index must be older than the minimum age of the next phase. + +The minimum age defaults to zero, which causes {ilm-init} to move indices to the next phase +as soon as all actions in the current phase complete. + +[discrete] +[[ilm-phase-execution]] +=== Phase execution + +{ilm-init} controls the order in which the actions in a phase are executed and +what _steps_ are executed to perform the necessary index operations for each action. + +When an index enters a phase, {ilm-init} caches the phase definition in the index metadata. +This ensures that policy updates don't put the index into a state where it can never exit the phase. +If changes can be safely applied, {ilm-init} updates the cached phase definition. +If they cannot, phase execution continues using the cached definition. + +[discrete] +[[ilm-phase-actions]] +=== Phase actions + +{ilm-init} supports the following actions in each phase. + +* Hot + - <> + - <> + - <> +* Warm + - <> + - <> + - <> + - <> + - <> + - <> +* Cold + - <> + - <> + - <> + - <> + - <> +* Delete + - <> + - <> + diff --git a/docs/reference/ilm/overview-ilm.asciidoc b/docs/reference/ilm/ilm-overview.asciidoc similarity index 67% rename from docs/reference/ilm/overview-ilm.asciidoc rename to docs/reference/ilm/ilm-overview.asciidoc index af3797587fb..cd7cd179de7 100644 --- a/docs/reference/ilm/overview-ilm.asciidoc +++ b/docs/reference/ilm/ilm-overview.asciidoc @@ -1,7 +1,7 @@ [role="xpack"] [testenv="basic"] [[overview-index-lifecycle-management]] -== Index lifecycle management overview +== {ilm-init} overview ++++ Overview @@ -21,29 +21,15 @@ include::../glossary.asciidoc[tag=force-merge-def-short] include::../glossary.asciidoc[tag=freeze-def-short] * **Delete** - Permanently remove an index, including all of its data and metadata. -Typically, you associate a lifecycle policy with an index template so it is automatically applied -to new indices. -You can also apply a policy manually when you create an index. - -{ilm-init} simplifies managing indices in hot-warm-cold architectures, +{ilm-init} makes it easier to manage indices in hot-warm-cold architectures, which are common when you're working with time-series data such as logs and metrics. -As an index ages, it moves through four possible phases: +You can specify: -* Hot--the index is actively being updated and queried. -* Warm--the index is no longer being updated, but is still being queried. -* Cold--the index is no longer being updated and is seldom queried. The -information still needs to be searchable, but it's okay if those queries are -slower. -* Delete--the index is no longer needed and can safely be deleted. - -A lifecycle policy controls how an index moves between phases and -what actions to perform during each phase. You can specify: - -* The maximum size or age at which you want to roll over to a new index. +* The maximum size, number of documents, or age at which you want to roll over to a new index. * The point at which the index is no longer being updated and the number of primary shards can be reduced. -* When to force a merge to permanently delete documents marked for deletion. +* When to force a merge to permanently remove documents marked for deletion. * The point at which the index can be moved to less performant hardware. * The point at which the availability is not as critical and the number of replicas can be reduced. diff --git a/docs/reference/ilm/getting-started-ilm.asciidoc b/docs/reference/ilm/ilm-tutorial.asciidoc similarity index 75% rename from docs/reference/ilm/getting-started-ilm.asciidoc rename to docs/reference/ilm/ilm-tutorial.asciidoc index efd033be7dd..0ac1978e921 100644 --- a/docs/reference/ilm/getting-started-ilm.asciidoc +++ b/docs/reference/ilm/ilm-tutorial.asciidoc @@ -1,15 +1,14 @@ [role="xpack"] [testenv="basic"] - [[getting-started-index-lifecycle-management]] -== Get started: Automate rollover with {ilm-init} +== Tutorial: Automate rollover with {ilm-init} ++++ Automate rollover ++++ -This tutorial demonstrates how to use {ilm} ({ilm-init}) -to manage indices that contain time-series data. +This tutorial demonstrates how to use {ilm} +({ilm-init}) to manage indices that contain time-series data. When you continuously index timestamped documents into {es} using Filebeat, Logstash, or some other mechanism, @@ -20,27 +19,42 @@ and still get the most out of your data. To automate rollover and management of time-series indices with {ilm-init}, you: -. <> with the {ilm-init} put policy API. +. <> that defines the appropriate +phases and actions. . <> to apply the policy to each new index. . <> as the initial write index. -. <> -as expected with the {ilm-init} explain API. +. <> +as expected. -[float] +For an introduction to rolling indices, see <>. + +NOTE: {filebeat} includes a default {ilm-init} policy that initiates the rollover action when +the index size reaches 50GB or becomes 30 days old. +You can use this policy as a starting point, or replace it with a custom policy. +See +{kib}/example-using-index-lifecycle-policy.html[Use {ilm-init} to manage Filebeat time-based indices]. + + +[discrete] [[ilm-gs-create-policy]] === Create a lifecycle policy A lifecycle policy specifies the phases in the index lifecycle and the actions to perform in each phase. A lifecycle can have up to four phases: -`hot`, `warm`, `cold`, and `delete`. Policies are defined in JSON -and added through the {ilm-init} put policy API. +`hot`, `warm`, `cold`, and `delete`. -For example, the following request creates a `timeseries_policy` with two phases: +You can define and manage policies through the {kib} Management UI, +which invokes the {ilm-init} <> API to create policies +according to the options you specify. -* The `hot` phase defines a `rollover` action to specify that an index rolls over when it +For example, you might define a `timeseries_policy` that has two phases: + +* A `hot` phase that defines a rollover action to specify that an index rolls over when it reaches either a `max_size` of 50 gigabytes or a `max_age` of 30 days. -* The `delete` phase uses `min_age` to remove the index 90 days after rollover. -Note that this value is relative to the rollover time, not the index creation time. +* A `delete` phase that sets `min_age` to remove the index 90 days after rollover. +Note that this value is relative to the rollover time, not the index creation time. + +The underlying put policy request looks like this: [source,console] ------------------------ @@ -71,24 +85,32 @@ PUT _ilm/policy/timeseries_policy <3> Move the index into the `delete` phase 90 days after rollover. <4> Trigger the `delete` action when the index enters the delete phase. -See <<_actions>> for the complete list of actions available in each phase. +You can also invoke this API directly to add lifecycle policies. -[float] +For the complete list of actions that {ilm} can perform, see <>. + +[discrete] [[ilm-gs-apply-policy]] === Create an index template to apply the lifecycle policy -To automaticaly apply a lifecycle policy to the new write index on rollover, +To automatically apply a lifecycle policy to the new write index on rollover, specify the policy in the index template used to create new indices. -For example, the following request creates a `timeseries_template` that is applied to new indices -whose names match the `timeseries-*` index pattern. -The template configures two {ilm-init} settings: +For example, you might create a `timeseries_template` that is applied to new indices +whose names match the `timeseries-*` index pattern. -* `index.lifecycle.name` specifies the name of the lifecycle policy to apply to all new indices that match -the index pattern. +To enable automatic rollover, the template configures two {ilm-init} settings: + +* `index.lifecycle.name` specifies the name of the lifecycle policy to apply to new indices +that match the index pattern. * `index.lifecycle.rollover_alias` specifies the index alias to be rolled over when the rollover action is triggered for an index. +You can use the {kib} Create template wizard to add the template. +This wizard invokes the put template API to create the template with the options you specify. + +The underlying request looks like this: + [source,console] ----------------------- PUT _template/timeseries_template @@ -109,6 +131,9 @@ PUT _template/timeseries_template <3> The name of the alias used to reference these indices. Required for policies that use the rollover action. +You can also invoke this API directly to add templates. + + ////////////////////////// [source,console] @@ -119,7 +144,7 @@ DELETE /_template/timeseries_template ////////////////////////// -[float] +[discrete] [[ilm-gs-bootstrap]] === Bootstrap the initial time-series index @@ -154,11 +179,9 @@ This process repeats each time rollover conditions are met. You can search across all of the indices managed by the `timeseries_policy` with the `timeseries` alias. Write operations are routed to the current write index. -For more information about write indices and rollover, see the <>. - -[float] +[discrete] [[ilm-gs-check-progress]] -=== Checking progress +=== Check lifecycle progress To get status information for managed indices, you use the {ilm-init} explain API. This lets you find out things like: @@ -179,6 +202,7 @@ The response below shows that the bootstrap index is waiting in the `hot` phase' It remains in this state and {ilm-init} continues to call `attempt-rollover` until the rollover conditions are met. +// [[36818c6d9f434d387819c30bd9addb14]] [source,console-result] -------------------------------------------------- { @@ -219,5 +243,3 @@ until the rollover conditions are met. <2> The age of the index <3> The step {ilm-init} is performing on the index <4> The definition of the current phase (the `hot` phase) - -See the <> for more information. diff --git a/docs/reference/ilm/ilm-with-existing-indices.asciidoc b/docs/reference/ilm/ilm-with-existing-indices.asciidoc index f2e2f5e7665..bdc258d1845 100644 --- a/docs/reference/ilm/ilm-with-existing-indices.asciidoc +++ b/docs/reference/ilm/ilm-with-existing-indices.asciidoc @@ -77,7 +77,7 @@ DELETE _template/mylogs_template ////////////////////////// Now that we have these indices, we'll look at a few different ways of migrating -these indices to ILM. +these indices to {ilm-init}. [[ilm-with-existing-periodic-indices]] === Managing existing periodic indices with {ilm-init} @@ -91,12 +91,11 @@ The simplest way to manage existing indices while transitioning to fully all new documents should be directed to {ilm-init}-managed indices - if you are using Beats or Logstash data shippers, upgrading all of those shippers to version 7.0.0 or higher will take care of that part for you. If you are not -using Beats or Logstash, you may need to set up ILM for new indices yourself as -demonstrated in the <>. +using Beats or Logstash, you may need to set up {ilm-init} for new indices yourself as +demonstrated in the <>. NOTE: If you are using Beats through Logstash, you may need to change your -Logstash output configuration and invoke the Beats setup to use ILM for new +Logstash output configuration and invoke the Beats setup to use {ilm-init} for new data. Once all new documents are being written to fully {ilm-init}-managed indices, it @@ -341,7 +340,7 @@ All new documents should be indexed via the `mylogs` alias at this point. Adding new data to the old indices during the reindexing process can cause data to be added to the old indices, but not be reindexed into the new indices. -NOTE: If you do not want to mix new data and old data in the new ILM-managed +NOTE: If you do not want to mix new data and old data in the new {ilm-init}-managed indices, indexing of new data should be paused entirely while the reindex completes. Mixing old and new data within one index is safe, but keep in mind that the indices with mixed data should be retained in their entirety until you @@ -364,7 +363,7 @@ PUT _cluster/settings ----------------------- // TEST[skip:don't want to overwrite this setting for other tests] -<1> This tells ILM to check for rollover conditions every minute +<1> This tells {ilm-init} to check for rollover conditions every minute We're now ready to reindex our data using the <>. If you have a timestamp or date field in your documents, as in this example, it may diff --git a/docs/reference/ilm/index-rollover.asciidoc b/docs/reference/ilm/index-rollover.asciidoc new file mode 100644 index 00000000000..f8680251585 --- /dev/null +++ b/docs/reference/ilm/index-rollover.asciidoc @@ -0,0 +1,38 @@ +[[index-rollover]] +=== Rollover + +When indexing time-series data like logs or metrics, you can't write to a single index indefinitely. +To meet your indexing and search performance requirements and manage resource usage, +you write to an index until some threshold is met and +then create a new index and start writing to it instead. +Using rolling indices enables you to: + +* Optimize the active index for high ingest rates on high-performance _hot_ nodes. +* Optimize for search performance on _warm_ nodes. +* Shift older, less frequently accessed data to less expensive _cold_ nodes, +* Delete data according to your retention policies by removing entire indices. + +Rollover relies on three things: + +* An _index template_ that specifies the settings for each new index in the series. +You optimize this configuration for ingestion, typically using as many shards as you have hot nodes. +* An _index alias_ that references the entire set of indices. +* A single index designated as the _write index_. +This is the active index that handles all write requests. +On each rollover, the new index becomes the write index. + +[discrete] +[role="xpack"] +[testenv="basic"] +[[ilm-automatic-rollover]] +=== Automatic rollover + +{ilm-init} enables you to automatically roll over to a new index based +on the index size, document count, or age. When a rollover is triggered, a new +index is created, the write alias is updated to point to the new index, and all +subsequent updates are written to the new index. + +TIP: Rolling over to a new index based on size, document count, or age is preferable +to time-based rollovers. Rolling over at an arbitrary time often results in +many small indices, which can have a negative impact on performance and +resource usage. \ No newline at end of file diff --git a/docs/reference/ilm/index.asciidoc b/docs/reference/ilm/index.asciidoc index a398f56d1ce..434e803e887 100644 --- a/docs/reference/ilm/index.asciidoc +++ b/docs/reference/ilm/index.asciidoc @@ -1,7 +1,7 @@ [role="xpack"] [testenv="basic"] [[index-lifecycle-management]] -= Manage the index lifecycle += {ilm-init}: Manage the index lifecycle [partintro] -- @@ -9,8 +9,8 @@ You can configure {ilm} ({ilm-init}) policies to automatically manage indices ac your performance, resiliency, and retention requirements. For example, you could use {ilm-init} to: +* Spin up a new index when an index reaches a certain size or number of documents * Create a new index each day, week, or month and archive previous ones -* Spin up a new index when an index reaches a certain size * Delete stale indices to enforce data retention standards [TIP] @@ -18,27 +18,27 @@ To automatically back up your indices and manage snapshots, use <>. * <> +* <> * <> -* <> +* <> * <> -* <> * <> * <> -* <> +* <> * <> * <> -- -include::overview-ilm.asciidoc[] +include::ilm-overview.asciidoc[] -include::getting-started-ilm.asciidoc[] +include::ilm-concepts.asciidoc[] -include::policy-definitions.asciidoc[] +include::ilm-tutorial.asciidoc[] + +include::ilm-actions.asciidoc[] include::set-up-lifecycle-policy.asciidoc[] -include::update-lifecycle-policy.asciidoc[] - include::error-handling.asciidoc[] include::start-stop-ilm.asciidoc[] diff --git a/docs/reference/ilm/policy-definitions.asciidoc b/docs/reference/ilm/policy-definitions.asciidoc deleted file mode 100644 index 6d467e02526..00000000000 --- a/docs/reference/ilm/policy-definitions.asciidoc +++ /dev/null @@ -1,836 +0,0 @@ -[role="xpack"] -[testenv="basic"] -[[ilm-policy-definition]] -== Policy phases and actions - -There are four stages in the index lifecycle, in the order -they are executed. - -[options="header"] -|====== -| Name | Description -| `hot` | The index is actively being written to -| `warm` | The index is generally not being written to, but is still queried -| `cold` | The index is no longer being updated and is seldom queried. The -information still needs to be searchable, but it's okay if those queries are -slower. -| `delete` | The index is no longer needed and can safely be deleted -|====== - -Each of these stages is called a "phase". A policy does not need to configure -each phase for an index. For example, one policy may define only the hot -phase and the delete phase, while another may define all four phases. - -=== Timing - -Indices enter phases based on a phase's `min_age` parameter. -The index will not enter the phase until the index's age is older than that -of the `min_age`. The parameter is configured using a time -duration format (see <>). - -`min_age` defaults to zero seconds `0s` for each phase if not specified. - -[source,console] --------------------------------------------------- -PUT _ilm/policy/my_policy -{ - "policy": { - "phases": { - "warm": { - "min_age": "1d", - "actions": { - "allocate": { - "number_of_replicas": 1 - } - } - }, - "delete": { - "min_age": "30d", - "actions": { - "delete": {} - } - } - } - } -} --------------------------------------------------- - -The above example configures a policy that moves the index into the warm -phase after one day. Until then, the index is in a waiting state. After -moving into the warm phase, it will wait until 30 days have elapsed before -moving to the delete phase and deleting the index. - -`min_age` is usually the time elapsed from the time the index is created, unless -the `index.lifecycle.origination_date` index setting is configured, in which -case the `min_age` will be the time elapsed since that specified date. If the -index is rolled over, then `min_age` is the time elapsed from the time the -index is rolled over. The intention here is to execute following phases and -actions relative to when data was written last to a rolled over index. - -The previous phase's actions must complete before {ilm} will check `min_age` and -transition into the next phase. By default, {ilm} checks for indices that meet -policy criteria, like `min_age`, every 10 minutes. You can use the -`indices.lifecycle.poll_interval` cluster setting to control how often this -check occurs. - -=== Phase Execution - -The current phase definition, of an index's policy being executed, is stored -in the index's metadata. This phase definition is cached to prevent changes to -the policy from putting the index in a state where it cannot proceed from its -current step. When the policy is updated we check to see if this phase -definition can be safely updated, and if so, update the cached definition in -indices using the updated policy. The phase and its actions are compiled into a -series of discrete steps that are executed sequentially. Since some {ilm-init} -actions are more complex and involve multiple operations against an index, each -of these operations are done in isolation in a unit called a "step". The -<> exposes this information to us -to see which step our index is either to execute next, or is currently -executing. - -=== Actions - -The below list shows the actions which are available in each phase. - -NOTE: The order that configured actions are performed in within each phase is -determined automatically by {ilm-init}, and cannot be changed by changing the -policy definition. - -* Hot - - <> - - <> - - <> -* Warm - - <> - - <> - - <> - - <> - - <> - - <> -* Cold - - <> - - <> - - <> - - <> - - <> -* Delete - - <> - - <> - -[[ilm-allocate-action]] -==== Allocate - -Phases allowed: warm, cold. - -The Allocate action allows you to specify which nodes are allowed to host the -shards of the index and set the number of replicas. -Behind the scenes, it is modifying the index settings -for shard filtering and/or replica counts. When updating the number of replicas, -configuring allocation rules is optional. When configuring allocation rules, -setting number of replicas is optional. Although this action can be treated as -two separate index settings updates, both can be configured at once. - -For more information about how {es} uses replicas for scaling, see -<>. See <> for more information about -controlling where Elasticsearch allocates shards of a particular index. - -[[ilm-allocate-options]] -.Allocate Options -[options="header"] -|====== -| Name | Required | Default | Description -| `number_of_replicas` | no | - | The number of replicas to - assign to the index -| `include` | no | - | assigns an index to nodes - having at least _one_ of the attributes -| `exclude` | no | - | assigns an index to nodes having - _none_ of the attributes -| `require` | no | - | assigns an index to nodes having - _all_ of the attributes -|====== - -If `number_of_replicas` is not configured, then at least one of `include`, -`exclude`, and `require` is required. An empty Allocate Action with no configuration -is invalid. - -===== Example: Change number of replicas - -In this example, the index's number of replicas is changed to `2`, while allocation -rules are unchanged. - -[source,console] --------------------------------------------------- -PUT _ilm/policy/my_policy -{ - "policy": { - "phases": { - "warm": { - "actions": { - "allocate" : { - "number_of_replicas" : 2 - } - } - } - } - } -} --------------------------------------------------- - -===== Example: Assign index to node with specific "box_type" attribute - -This example assigns the index to nodes with `box_type` attribute of "hot" or "warm". - -[source,console] --------------------------------------------------- -PUT _ilm/policy/my_policy -{ - "policy": { - "phases": { - "warm": { - "actions": { - "allocate" : { - "include" : { - "box_type": "hot,warm" - } - } - } - } - } - } -} --------------------------------------------------- - -===== Example: Assign index to a specific node and update replica settings - -This example updates the index to have one replica per shard and be allocated -to nodes with a `box_type` attribute of "cold". - -[source,console] --------------------------------------------------- -PUT _ilm/policy/my_policy -{ - "policy": { - "phases": { - "warm": { - "actions": { - "allocate" : { - "number_of_replicas": 1, - "require" : { - "box_type": "cold" - } - } - } - } - } - } -} --------------------------------------------------- - -[[ilm-wait-for-snapshot-action]] -==== Wait For Snapshot - -Phases allowed: delete. - -The Wait For Snapshot Action waits for defined SLM policy to be executed to ensure that snapshot of index exists before -deletion. - -[[ilm-wait-for-snapshot-options]] -.Wait For Snapshot -[options="header"] -|====== -| Name | Required | Default | Description -| `policy` | yes | - | SLM policy name that this action should wait for -|====== - -[source,console] --------------------------------------------------- -PUT _ilm/policy/my_policy -{ - "policy": { - "phases": { - "delete": { - "actions": { - "wait_for_snapshot" : { - "policy": "slm-policy-name" - } - } - } - } - } -} --------------------------------------------------- - -[[ilm-delete-action]] -==== Delete - -Phases allowed: delete. - -The Delete Action does just that, it deletes the index. - -[[ilm-delete-action-options]] -.Delete -[options="header"] -|====== -| Name | Required | Default | Description -| `delete_searchable_snapshot` | no | true | Deletes the searchable snapshot created in the cold phase, if a - snapshot was created (ie. if the <> was used in the cold phase) -|====== - -[source,console] --------------------------------------------------- -PUT _ilm/policy/my_policy -{ - "policy": { - "phases": { - "delete": { - "actions": { - "delete" : { } - } - } - } - } -} --------------------------------------------------- - -[[ilm-forcemerge-action]] -==== Force Merge - -Phases allowed: hot, warm. - -NOTE: Index will be be made read-only when this action is run -(see: <>) - -NOTE: The `forcemerge` action is best effort. It might happen that some of the -shards are relocating, in which case they will not be merged. - -NOTE: If the `forcemerge` action is used in the `hot` phase, the `rollover` action *must* be present. -ILM validates this predicate and will refuse a policy with a forcemerge in the hot phase without a -rollover action. - -The Force Merge Action <> the index into at -most a specific number of <>. - -[[ilm-forcemerge-options]] -.Force Merge Options -[options="header"] -|====== -| Name | Required | Default | Description -| `max_num_segments` | yes | - | The number of segments to merge to. To fully merge the index, set it to `1` -| `codec` | no | - | Optional specification of the `best_compression` codec -|====== - -[WARNING] -====== -When using the `"codec": "best_compression"` configuration in the ILM forcemerge action, ILM will -<> and then <> the index prior to a forcemerge. -During this time the index will be unavailable for either read or write operations. -====== - -[source,console] --------------------------------------------------- -PUT _ilm/policy/my_policy -{ - "policy": { - "phases": { - "warm": { - "actions": { - "forcemerge" : { - "max_num_segments": 1 - } - } - } - } - } -} --------------------------------------------------- - -[[ilm-freeze-action]] -==== Freeze - -Phases allowed: cold. - -This action will <> the index -by calling the <>. - -[source,console] --------------------------------------------------- -PUT _ilm/policy/my_policy -{ - "policy": { - "phases": { - "cold": { - "actions": { - "freeze" : { } - } - } - } - } -} --------------------------------------------------- - -[IMPORTANT] -================================ - Freezing an index will close the index and reopen it within the same API call. - This causes primaries to not be allocated for a short amount of time and - causes the cluster to go red until the primaries are allocated again. - This limitation might be removed in the future. -================================ - -[[ilm-readonly-action]] -==== Read-Only - -Phases allowed: warm. - -This action will set the index to be read-only -(see: <>) - -This action does not have any options associated with it. - -[source,console] --------------------------------------------------- -PUT _ilm/policy/my_policy -{ - "policy": { - "phases": { - "warm": { - "actions": { - "readonly" : { } - } - } - } - } -} --------------------------------------------------- - -[[ilm-rollover-action]] -==== Rollover - -Phases allowed: hot. - -[WARNING] -index format must match pattern '^.*-\\d+$', for example (`logs-000001`). -[WARNING] -The managed index must set `index.lifecycle.rollover_alias` as the -alias to rollover. The index must also be the write index for the alias. - -[IMPORTANT] -If a policy using the Rollover action is used on a <>, policy execution will wait until the leader index rolls over (or has -<>), then convert the -follower index into a regular index as if <> had been used instead of rolling over. - -For example, if an index to be managed has an alias `my_data`. The managed -index "my_index-000001" must be the write index for the alias. For more information, read -<>. - -[source,console] --------------------------------------------------- -PUT my_index-000001 -{ - "settings": { - "index.lifecycle.name": "my_policy", - "index.lifecycle.rollover_alias": "my_data" - }, - "aliases": { - "my_data": { - "is_write_index": true - } - } -} --------------------------------------------------- - -The Rollover Action rolls an alias over to a new index when the -existing index meets one of the rollover conditions. - - -[[ilm-rollover-options]] -.Rollover Options -[options="header"] -|====== -| Name | Required | Default | Description -| `max_size` | no | - | max primary shard index storage size. - See <> - for formatting -| `max_docs` | no | - | max number of documents an - index is to contain before - rolling over. -| `max_age` | no | - | max time elapsed from index - creation. See - <> - for formatting -|====== - -At least one of `max_size`, `max_docs`, `max_age` or any combinations of the -three are required to be specified. - -===== Example: Rollover when index is too large - -This example rolls the index over when it is at least 100 gigabytes. - -[source,console] --------------------------------------------------- -PUT _ilm/policy/my_policy -{ - "policy": { - "phases": { - "hot": { - "actions": { - "rollover" : { - "max_size": "100GB" - } - } - } - } - } -} --------------------------------------------------- - -===== Example: Rollover when index has too many documents - -This example rolls the index over when it contains at least -100000000 documents. - -[source,console] --------------------------------------------------- -PUT _ilm/policy/my_policy -{ - "policy": { - "phases": { - "hot": { - "actions": { - "rollover" : { - "max_docs": 100000000 - } - } - } - } - } -} --------------------------------------------------- - -===== Example: Rollover when index is too old - -This example rolls the index over when it has been created at least -7 days ago. - -[source,console] --------------------------------------------------- -PUT _ilm/policy/my_policy -{ - "policy": { - "phases": { - "hot": { - "actions": { - "rollover" : { - "max_age": "7d" - } - } - } - } - } -} --------------------------------------------------- - -===== Example: Rollover when index is too old or too large - -This example rolls the index over when it has been created at least -7 days ago or it is at least 100 gigabytes. In this case, the index will be -rolled over when any of the conditions is met. - -[source,console] --------------------------------------------------- -PUT _ilm/policy/my_policy -{ - "policy": { - "phases": { - "hot": { - "actions": { - "rollover" : { - "max_age": "7d", - "max_size": "100GB" - } - } - } - } - } -} --------------------------------------------------- - -===== Example: Rollover condition stalls phase transition - -The Rollover action will only complete once one of its conditions is -met. This means that any proceeding phases will be blocked until Rollover -succeeds. - -[source,console] --------------------------------------------------- -PUT /_ilm/policy/rollover_policy -{ - "policy": { - "phases": { - "hot": { - "actions": { - "rollover": { - "max_size": "50G" - } - } - }, - "delete": { - "min_age": "1d", - "actions": { - "delete": {} - } - } - } - } -} --------------------------------------------------- - -The above example illustrates a policy which attempts to delete an -index one day after the index has been rolled over. It does not -delete the index one day after it has been created. - -[[ilm-searchable-snapshot-action]] -==== Searchable Snapshot - -Phases allowed: cold. - -This action takes a snapshot of the managed index in the configured repository -and mounts it as a searchable snapshot. - -[[ilm-searchable-snapshot-options]] -.Searchable Snapshot Options -[options="header"] -|====== -| Name | Required | Default | Description -| `snapshot_repository` | yes | - | Repository used to store the snapshot created by this action. The snapshot will be, - by default, deleted by the <> in the delete phase, if - configured, but the user can configure the <> to keep - the snapshot using the `delete_searchable_snapshot` option. - -|====== - -[source,console] --------------------------------------------------- -PUT _ilm/policy/my_policy -{ - "policy": { - "phases": { - "cold": { - "actions": { - "searchable_snapshot" : { - "snapshot_repository" : "backing_repo" - } - } - } - } - } -} --------------------------------------------------- -[[ilm-set-priority-action]] -==== Set Priority - -Phases allowed: hot, warm, cold. - -This action sets the <> on the index as -soon as the policy enters the hot, warm, or cold phase. Indices with a higher -priority will be recovered before indices with lower priorities following a node -restart. Generally, indexes in the hot phase should have the highest value and -indexes in the cold phase should have the lowest values. For example: -100 for the hot phase, 50 for the warm phase, and 0 for the cold phase. -Indicies that don't set this value have an implicit default priority of 1. - -[[ilm-set-priority-options]] -.Set Priority Options -[options="header"] -|====== -| Name | Required | Default | Description -| `priority` | yes | - | The priority for the index. Must be 0 or greater. - The value may also be set to null to remove the priority. - -|====== - -[source,console] --------------------------------------------------- -PUT _ilm/policy/my_policy -{ - "policy": { - "phases": { - "warm": { - "actions": { - "set_priority" : { - "priority": 50 - } - } - } - } - } -} --------------------------------------------------- - -[[ilm-shrink-action]] -==== Shrink - -NOTE: Index will be be made read-only when this action is run -(see: <>) -[IMPORTANT] -If a policy using the Shrink action is used on a <>, policy execution will wait until the leader index rolls over (or has -<>), then convert the -follower index into a regular index as if <> had been used before shrink is applied, as shrink cannot be safely -applied to follower indices. - -This action shrinks an existing index into a new index with fewer primary -shards. It calls the <> to shrink the index. -Since allocating all the primary shards of the index to one node is a -prerequisite, this action will first allocate the primary shards to a valid -node. After shrinking, it will swap aliases pointing to the original index -into the new shrunken index. The new index will also have a new name: -"shrink-". So if the original index was called "logs", -then the new index will be named "shrink-logs". - -[[ilm-shrink-options]] -.Shrink Options -[options="header"] -|====== -| Name | Required | Default | Description -| `number_of_shards` | yes | - | The number of shards - to shrink to. must be - a factor of the number - of shards in the - source index. -|====== - -[source,console] --------------------------------------------------- -PUT _ilm/policy/my_policy -{ - "policy": { - "phases": { - "warm": { - "actions": { - "shrink" : { - "number_of_shards": 1 - } - } - } - } - } -} --------------------------------------------------- - -[[ilm-unfollow-action]] -==== Unfollow - -[IMPORTANT] -This action may be used explicitly, as shown below, but this action is also run -before <> and <> as described in the documentation for those actions. This is -expected and safe for non-CCR indices to run, as the steps are skipped when CCR -is not in use. - -This action turns a {ref}/ccr-apis.html[ccr] follower index -into a regular index. This can be desired when moving follower -indices into the next phase. Also certain actions like shrink -and rollover can then be performed safely on follower indices. - -This action will wait until is it safe to convert a follower index into a -regular index. In particular, the following conditions must be met: - -* The leader index must have `index.lifecycle.indexing_complete` set to `true`. -This happens automatically if the leader index is rolled over using -<>, or may be set manually using -the <>. -* All operations performed on the leader index must have been replicated to the -follower index. This ensures that no operations will be lost when the index is -converted into a regular index. - -If the unfollow action encounters a follower index then -the following operations will be performed on it: - -* Pauses indexing following for the follower index. -* Closes the follower index. -* Unfollows the follower index. -* Opens the follower index (which is at this point is a regular index). - -The unfollow action does not have any options and -if it encounters a non follower index, then the -unfollow action leaves that index untouched and -lets the next action operate on this index. - -[source,console] --------------------------------------------------- -PUT _ilm/policy/my_policy -{ - "policy": { - "phases": { - "hot": { - "actions": { - "unfollow" : {} - } - } - } - } -} --------------------------------------------------- - -=== Full Policy - -With all of these actions, we can support complex management strategies for our -indices. This policy will define an index that will start in the hot phase, -rolling over every 50 GB or 7 days. After 30 days it enters the warm phase -and increases the replicas to 2, force merges and shrinks. After 60 days -it enters the cold phase and allocates to "cold" nodes, and after 90 days the -index is deleted. - -[source,console] --------------------------------------------------- -PUT _ilm/policy/full_policy -{ - "policy": { - "phases": { - "hot": { - "actions": { - "rollover": { - "max_age": "7d", - "max_size": "50G" - } - } - }, - "warm": { - "min_age": "30d", - "actions": { - "forcemerge": { - "max_num_segments": 1 - }, - "shrink": { - "number_of_shards": 1 - }, - "allocate": { - "number_of_replicas": 2 - } - } - }, - "cold": { - "min_age": "60d", - "actions": { - "allocate": { - "require": { - "type": "cold" - } - } - } - }, - "delete": { - "min_age": "90d", - "actions": { - "delete": {} - } - } - } - } -} --------------------------------------------------- diff --git a/docs/reference/ilm/set-up-lifecycle-policy.asciidoc b/docs/reference/ilm/set-up-lifecycle-policy.asciidoc index 5da5fc3e464..f0517c334f1 100644 --- a/docs/reference/ilm/set-up-lifecycle-policy.asciidoc +++ b/docs/reference/ilm/set-up-lifecycle-policy.asciidoc @@ -1,12 +1,26 @@ [role="xpack"] [testenv="basic"] [[set-up-lifecycle-policy]] -== Create lifecycle policy +== Configure lifecycle policy [[ilm-policy-definition]] -In order for an index to use an {ilm} policy to manage its lifecycle we must -first define a lifecycle policy for it to use. The following request creates a -policy called `my_policy` in Elasticsearch which we can later use to manage our -indexes. +For {ilm-init} to manage an index, a valid policy +must be specified in the `index.lifecycle.name` index setting. + +To configure a lifecycle policy for rolling indices, +you create the policy and add it to the index template. + +To use a policy to manage an index that doesn't roll over, +you can specify the policy directly when you create it. + +[discrete] +[[ilm-create-policy]] +=== Create lifecycle policy + +You use the <> to define a new lifecycle policy. +For example, the following request creates `my_policy`, a +policy that defines a hot and and delete phase. +When the index reaches 25GB, it rolls over directly to the delete phase. +The index is deleted 30 days after rollover. [source,console] ------------------------ @@ -32,28 +46,24 @@ PUT _ilm/policy/my_policy } ------------------------ -<1> Rollover the index when it reaches 25GB in size -<2> Delete the index when its 30 days old +<1> Roll over the index when it reaches 25GB in size +<2> Delete the index 30 days after rollover -{ilm} will manage an index using the policy defined in the -`index.lifecycle.name` index setting. If this setting does not exist in the -settings for a particular index, {ilm} will not manage that index. +NOTE: {ilm-init} policies are stored in the global cluster state and can be included in snapshots by +setting `include_global_state` to `true` when you <>. +Restoring {ilm-init} policies from a snapshot is all-or-nothing. +The entire global state, including all policies, is overwritten when you restore the snapshot. -To set the policy for an index there are two options: +[discrete] +[[apply-policy-template]] +=== Apply lifecycle policy with an index template -1. Apply the policy to an index template and bootstrap creating the first index -2. Apply the policy to a new index in a create index request - -NOTE: ILM policies are stored in global cluster state which can be backed up using Snapshot with the -`include_global_state` parameter set to `true`. -Restoring ILM policies from global state in a Snapshot is all-or-nothing that will overwrite the entire global -state using the snapshot's point in time copy including all ILM policies. - -[[applying-policy-to-template]] -=== Applying a policy to an index template - -The `index.lifecycle.name` setting can be set in an index template so that it -is automatically applied to indexes matching the templates index pattern: +To use a policy that triggers the rollover action, +you need to configure the policy in the index template used to create each new index. + +In addition to specifying the name of the policy in the `index.lifecycle.name` setting, +you specify a `index.lifecycle.rollover_alias` for referencing +the indices managed by this policy. [source,console] ----------------------- @@ -64,14 +74,14 @@ PUT _template/my_template "number_of_shards": 1, "number_of_replicas": 1, "index.lifecycle.name": "my_policy", <2> - "index.lifecycle.rollover_alias": "test-alias" + "index.lifecycle.rollover_alias": "test-alias" <3> } } ----------------------- -<1> This template will be applied to all indexes which have a name starting -with `test-` -<2> The template will set the policy to be used to `my_policy` +<1> Use this template for all new indices whose names begin with `test-` +<2> Apply `my_policy` to new indices created with this template +<3> Define an index alias for referencing indices managed by `my_policy` ////////////////////////// @@ -83,8 +93,18 @@ DELETE /_template/my_template ////////////////////////// -Now that a policy exists and is used in an index template we can create an -initial index which will be managed by our policy: +[discrete] +[[create-initial-index]] +==== Create an initial managed index + +You need to manually create the first index managed by a policy that uses the rollover action +and designate it as the write index. +The name of the index must match the pattern defined in the index template and end with a number. +This number is incremented to generate the name of indices created by the rollover action. + +For example, the following request creates the `test-00001` index. +Because it matches the index pattern specified in `my_template`, +{es} automatically applies the settings from that template. [source,console] ----------------------- @@ -100,14 +120,18 @@ PUT test-000001 <1> Set this initial index to be the write index for this alias. -We can now write data to the `test-alias` alias. Because we have a rollover -action defined in our policy, when the index grows larger than 25GB {ilm} will -create a new index and roll the alias over to use the new index automatically. +Now you can start indexing data to the rollover alias specified in the lifecycle policy. +With the sample `my_policy` policy, the rollover action is triggered once the initial +index exceeds 25GB. +{ilm-init} then creates a new index that becomes the write index for the `test-alias`. -=== Apply a policy to a create index request +[discrete] +[[apply-policy-manually]] +=== Apply lifecycle policy manually -The `index.lifecycle.name` setting can be set on an individual create index -request so {ilm} immediately starts managing the index: +When you create an index directly, you can apply a lifecycle policy +by specifying the `index.lifecycle.name` setting. +This causes {ilm-init} to immediately start managing the index. [source,console] ----------------------- @@ -121,8 +145,6 @@ PUT test-index } ----------------------- - -IMPORTANT: Its recommended not to use the create index API with a policy that -defines a rollover action. If you do so, the new index as the result of the -rollover will not carry forward the policy. Always use index templates to -define policies with rollover actions. +IMPORTANT: Do not manually apply a policy that uses the rollover action. +Policies that use rollover must be applied by the <>. +Otherwise, the policy is not carried forward when the rollover action creates a new index. diff --git a/docs/reference/ilm/start-stop-ilm.asciidoc b/docs/reference/ilm/start-stop-ilm.asciidoc index dc9ecb817c4..c813792a804 100644 --- a/docs/reference/ilm/start-stop-ilm.asciidoc +++ b/docs/reference/ilm/start-stop-ilm.asciidoc @@ -3,20 +3,20 @@ [[start-stop-ilm]] == Start and stop {ilm} -All indices that are managed by ILM will continue to execute +All indices that are managed by {ilm-init} will continue to execute their policies. There may be times when this is not desired on certain indices, or maybe even all the indices in a cluster. For example, maybe there are scheduled maintenance windows when cluster topology -changes are desired that may impact running ILM actions. For this reason, -ILM has two ways to disable operations. +changes are desired that may impact running {ilm-init} actions. For this reason, +{ilm-init} has two ways to disable operations. -When stopping ILM, snapshot lifecycle management operations are also stopped, +When stopping {ilm-init}, snapshot lifecycle management operations are also stopped, this means that no scheduled snapshots are created (currently ongoing snapshots are unaffected). -Normally, ILM will be running by default. -To see the current operating status of ILM, use the <> -to see the current state of ILM. +Normally, {ilm-init} will be running by default. +To see the current operating status of {ilm-init}, use the <> +to see the current state of {ilm-init}. //// [source,console] @@ -66,22 +66,22 @@ If the request does not encounter errors, you receive the following result: } -------------------------------------------------- -The operating modes of ILM: +The operating modes of {ilm-init}: [[ilm-operating-modes]] -.ILM Operating Modes +.{ilm-init} Operating Modes [options="header"] |=== |Name |Description |RUNNING |Normal operation where all policies are executed as normal -|STOPPING|ILM has received a request to stop but is still processing some policies +|STOPPING|{ilm-init} has received a request to stop but is still processing some policies |STOPPED |This represents a state where no policies are executed |=== -[float] -=== Stopping ILM +[discrete] +=== Stopping {ilm-init} -The ILM service can be paused such that no further steps will be executed +The {ilm-init} service can be paused such that no further steps will be executed using the <>. [source,console] @@ -109,8 +109,8 @@ GET _ilm/status -------------------------------------------------- // TESTRESPONSE[s/"STOPPING"/$body.operation_mode/] -The ILM service will then, asynchronously, run all policies to a point -where it is safe to stop. After ILM verifies that it is safe, it will +The {ilm-init} service will then, asynchronously, run all policies to a point +where it is safe to stop. After {ilm-init} verifies that it is safe, it will move to the `STOPPED` mode. //// @@ -131,10 +131,10 @@ GET _ilm/status -------------------------------------------------- // TESTRESPONSE[s/"STOPPED"/$body.operation_mode/] -[float] -=== Starting ILM +[discrete] +=== Starting {ilm-init} -To start ILM and continue executing policies, use the <>. +To start {ilm-init} and continue executing policies, use the <>. [source,console] @@ -151,7 +151,7 @@ GET _ilm/status // TEST[continued] //// -The Start API will send a request to the ILM service to immediately begin +The Start API will send a request to the {ilm-init} service to immediately begin normal operations. [source,console-result] diff --git a/docs/reference/ilm/update-lifecycle-policy.asciidoc b/docs/reference/ilm/update-lifecycle-policy.asciidoc index 3b19794e8cb..75f3a6a7cb9 100644 --- a/docs/reference/ilm/update-lifecycle-policy.asciidoc +++ b/docs/reference/ilm/update-lifecycle-policy.asciidoc @@ -1,506 +1,41 @@ [role="xpack"] [testenv="basic"] [[update-lifecycle-policy]] -== Update lifecycle policy +=== Lifecycle policy updates ++++ -Update policy +Policy updates ++++ -You can update an existing lifecycle policy to fix mistakes or change -strategies for newly created indices. It is possible to update policy definitions -and an index's `index.lifecycle.name` settings independently. To prevent the situation -that phase definitions are modified while currently being executed on an index, each index -will keep the version of the current phase definition it began execution with until it completes. -This also means that changes to `min_age` will not be propagated. If a new policy is set that -introduces a later `min_age` for the currently executing phase, that new `min_age` will not -be picked up by the update. +You can change how the lifecycle of an index or collection of rolling indices is managed +by modifying the current policy or switching to a different policy. -There are three scenarios for examining the behavior updating policies and -their effects on policy execution on indices. +To ensure that policy updates don't put an index into a state where it can't exit the current phase, +the phase definition is cached in the index metadata when it enters the phase. +This cached definition is used to complete the phase. -=== Updates to policies not managing indices +When the index advances to the next phase, it uses the phase definition from the updated policy. -Indices not referencing an existing policy that is updated will not be affected. -If an index is assigned to the policy, it will be assigned the latest version of that policy +[discrete] +[[ilm-apply-changes]] +=== How changes are applied -To show this, let's create a policy `my_policy`. +When a policy is initially applied to an index, the index gets the latest version of the policy. +If you update the policy, the policy version is bumped and {ilm-init} can detect that the index +is using an earlier version that needs to be updated. -[source,console] ------------------------- -PUT _ilm/policy/my_policy -{ - "policy": { - "phases": { - "hot": { - "actions": { - "rollover": { - "max_size": "25GB" - } - } - }, - "delete": { - "min_age": "30d", - "actions": { - "delete": {} - } - } - } - } -} ------------------------- +Changes to `min_age` are not propagated to the cached definition. +Changing a phase's `min_age` does not affect indices that are currently executing that phase. -This newly defined policy will be created and assigned to have a version equal -to 1. Since we haven't assigned any indices to this policy, any updates that -occur will be reflected completely on indices that are newly set to be managed -by this policy. +For example, if you create a policy that has a hot phase that does not specify a `min_age`, +indices immediately enter the hot phase when the policy is applied. +If you then update the policy to specify a `min_age` of 1 day for the hot phase, +that has no effect on indices that are already in the hot phase. +Indices created _after_ the policy update won't enter the hot phase until they are a day old. -Updating the Delete phase's minimum age can be done in an update request. +[discrete] +[[ilm-apply-new-policy]] +=== How new policies are applied -[source,console] ------------------------- -PUT _ilm/policy/my_policy -{ - "policy": { - "phases": { - "hot": { - "actions": { - "rollover": { - "max_size": "25GB" - } - } - }, - "delete": { - "min_age": "10d", <1> - "actions": { - "delete": {} - } - } - } - } -} ------------------------- -// TEST[continued] - -<1> update `min_age` to 10 days - -////////// -[source,console] --------------------------------------------------- -GET _ilm/policy/my_policy --------------------------------------------------- -// TEST[continued] -////////// - -When we get the policy, we will see it reflect our latest changes, but -with its version bumped to 2. - -[source,console-result] --------------------------------------------------- -{ - "my_policy": { - "version": 2, <1> - "modified_date": 82392349, <2> - "policy": { - "phases": { - "hot": { - "min_age": "0ms", - "actions": { - "rollover": { - "max_size": "25gb" - } - } - }, - "delete": { - "min_age": "10d", - "actions": { - "delete": { - "delete_searchable_snapshot": true - } - } - } - } - } - } -} --------------------------------------------------- -// TESTRESPONSE[s/"modified_date": 82392349/"modified_date": $body.my_policy.modified_date/] - -<1> The updated version value -<2> The timestamp when this policy was updated last. - -Afterwords, any indices set to `my_policy` will execute against version 2 of -the policy. - -=== Updates to executing policies - -Indices preserve the phase definition from the latest policy version that existed -at the time that it entered that phase. Changes to the currently-executing phase within policy updates will -not be reflected during execution. This means that updates to the `hot` phase, for example, will not affect -indices that are currently executing the corresponding `hot` phase. - -Let's say we have an index `my_index` managed by the below `my_executing_policy` definition. - -[source,console] ------------------------- -PUT _ilm/policy/my_executing_policy -{ - "policy": { - "phases": { - "hot": { - "actions": { - "rollover": { - "max_docs": 1 - } - } - }, - "delete": { - "min_age": "10d", - "actions": { - "delete": {} - } - } - } - } -} ------------------------- - -//// -[source,console] ------------------------- -PUT my_index -{ - "settings": { - "index.lifecycle.name": "my_executing_policy" - } -} ------------------------- -// TEST[continued] -//// - -The <> is useful to introspect managed indices to see which phase definition they are currently executing. -Using this API, we can find out that `my_index` is currently checking if it is ready to be rolled over. - -[source,console] --------------------------------------------------- -GET my_index/_ilm/explain --------------------------------------------------- -// TEST[continued] - -[source,console-result] --------------------------------------------------- -{ - "indices": { - "my_index": { - "index": "my_index", - "managed": true, - "policy": "my_executing_policy", - "lifecycle_date_millis": 1538475653281, - "age": "30s", - "phase": "hot", - "phase_time_millis": 1538475653317, - "action": "rollover", - "action_time_millis": 1538475653317, - "step": "check-rollover-ready", - "step_time_millis": 1538475653317, - "phase_execution": { - "policy": "my_executing_policy", - "modified_date_in_millis": 1538475653317, - "version": 1, - "phase_definition": { - "min_age": "0ms", - "actions": { - "rollover": { - "max_docs": 1 - } - } - } - } - } - } -} --------------------------------------------------- -// TESTRESPONSE[skip:no way to know if we will get this response immediately] - -We can update `my_executing_policy` to enter the hot phase after one day. - -[source,console] ------------------------- -PUT _ilm/policy/my_executing_policy -{ - "policy": { - "phases": { - "hot": { - "min_age": "1d", <1> - "actions": { - "rollover": { - "max_docs": 1 - } - } - }, - "delete": { - "min_age": "10d", - "actions": { - "delete": {} - } - } - } - } -} ------------------------- -// TEST[continued] - -<1> updated `min_age` from "0ms" to "1d" - -The index `my_index` has already entered the hot phase, so it will still -use version 1 of the policy until it completes the hot phase. - -//// -[source,console] --------------------------------------------------- -GET my_index/_ilm/explain --------------------------------------------------- -// TEST[continued] -//// - -[source,console-result] --------------------------------------------------- -{ - "indices": { - "my_index": { - "index": "my_index", - "managed": true, - "policy": "my_executing_policy", - "lifecycle_date_millis": 1538475653281, - "age": "30s", - "phase": "hot", - "phase_time_millis": 1538475653317, - "action": "rollover", - "action_time_millis": 1538475653317, - "step": "check-rollover-ready", - "step_time_millis": 1538475653317, - "phase_execution": { - "policy": "my_executing_policy", - "modified_date_in_millis": 1538475653317, - "version": 1, <1> - "phase_definition": { - "min_age": "0ms", - "actions": { - "rollover": { - "max_docs": 1 - } - } - } - } - } - } -} --------------------------------------------------- -// TESTRESPONSE[skip:no way to know if we will get this response immediately] - -<1> the version of the policy used for executing the hot phase - -We can also update `my_executing_policy` to have no rollover action and, -instead, go directly into a newly introduced `warm` phase. - -[source,console] ------------------------- -PUT _ilm/policy/my_executing_policy -{ - "policy": { - "phases": { - "warm": { - "min_age": "1d", - "actions": { - "forcemerge": { - "max_num_segments": 1 - } - } - }, - "delete": { - "min_age": "10d", - "actions": { - "delete": {} - } - } - } - } -} ------------------------- -// TEST[continued] - -Now, version 3 of this policy has no `hot` phase, but if we run the -Explain API again, we will see that nothing has changed. The index -`my_index` is still executing version 1 of the policy. - -//// -[source,console] --------------------------------------------------- -GET my_index/_ilm/explain --------------------------------------------------- -// TEST[continued] -//// - -[source,console-result] --------------------------------------------------- -{ - "indices": { - "my_index": { - "index": "my_index", - "managed": true, - "policy": "my_executing_policy", - "lifecycle_date_millis": 1538475653281, - "age": "30s", - "phase": "hot", - "phase_time_millis": 1538475653317, - "action": "rollover", - "action_time_millis": 1538475653317, - "step": "check-rollover-ready", - "step_time_millis": 1538475653317, - "phase_execution": { - "policy": "my_executing_policy", - "modified_date_in_millis": 1538475653317, - "version": 1, <1> - "phase_definition": { - "min_age": "0ms", - "actions": { - "rollover": { - "max_docs": 1 - } - } - } - } - } - } -} --------------------------------------------------- -// TESTRESPONSE[skip:no way to know if we will get this response immediately] - -<1> the version of the policy used for executing the hot phase - -After indexing one document into `my_index` so that rollover succeeds and -moves onto the next phase, we will notice something new. The index will -move into the next phase in the updated version 3 of its policy. - -//// -[source,console] --------------------------------------------------- -PUT my_index/_doc/1 -{ - "foo": "bar" -} - -GET my_index/_ilm/explain --------------------------------------------------- -// TEST[continued] -//// - -[source,console-result] --------------------------------------------------- -{ - "indices": { - "my_index": { - "index": "my_index", - "managed": true, - "policy": "my_executing_policy", - "lifecycle_date_millis": 1538475653281, - "age": "30s", - "phase": "warm", - "phase_time_millis": 1538475653317, - "action": "forcemerge", - "action_time_millis": 1538475653317, - "step": "forcemerge", - "step_time_millis": 1538475653317, - "phase_execution": { - "policy": "my_executing_policy", - "modified_date_in_millis": 1538475653317, - "version": 3, <1> - "phase_definition": { - "min_age": "1d", - "actions": { - "forcemerge": { - "max_num_segments": 1 - } - } - } - } - } - } -} --------------------------------------------------- -// TESTRESPONSE[skip:There is no way to force the index to move to the next step in a timely manner] - -<1> The index has moved to using version 3 of the policy - -`my_index` will move to the next phase in the latest policy definition, which is the newly added `warm` phase. - -=== Switching policies for an index - -Setting `index.lifecycle.name` to a different policy behaves much like a policy update, but instead of just -switching to a different version, it switches to a different policy. - -After setting a policy for an index, we can switch out `my_policy` with -`my_other_policy` by just updating the index's `index.lifecycle.name` -setting to the new policy. After completing its currently executed phase, -it will move on to the next phase in `my_other_policy`. So if it was on the -`hot` phase before, it will move to the `delete` phase after the `hot` phase concluded. - -//// -[source,console] ------------------------- -PUT _ilm/policy/my_policy -{ - "policy": { - "phases": { - "hot": { - "actions": { - "rollover": { - "max_size": "25GB" - } - } - }, - "delete": { - "min_age": "10d", - "actions": { - "delete": {} - } - } - } - } -} - -PUT _ilm/policy/my_other_policy -{ - "policy": { - "phases": { - "delete": { - "min_age": "1d", - "actions": { - "delete": {} - } - } - } - } -} - -PUT my_index -{ - "settings": { - "index.lifecycle.name": "my_policy" - } -} ------------------------- - -//// - -[source,console] --------------------------------------------------- -PUT my_index/_settings -{ - "lifecycle.name": "my_other_policy" -} --------------------------------------------------- -// TEST[continued] - -The change to the new policy will not happen immediately. The currently executing phase -of the existing policy for `my_index` will continue to execute until it completes. Once -completed, `my_index` will move to being managed by the `my_other_policy`. +When you apply a different policy to a managed index, +the index completes the current phase using the cached definition from the previous policy. +The index starts using the new policy when it moves to the next phase. diff --git a/docs/reference/ilm/using-policies-rollover.asciidoc b/docs/reference/ilm/using-policies-rollover.asciidoc index a9e42fbee5b..c430f6d7c9b 100644 --- a/docs/reference/ilm/using-policies-rollover.asciidoc +++ b/docs/reference/ilm/using-policies-rollover.asciidoc @@ -1,18 +1,8 @@ [role="xpack"] [testenv="basic"] +[[ilm-configure-rollover]] +== Configure rollover [[using-policies-rollover]] -== Roll over automatically - -The rollover action enables you to automatically roll over to a new index based -on the index size, document count, or age. When a rollover is triggered, a new -index is created, the write alias is updated to point to the new index, and all -subsequent updates are written to the new index. - -Rolling over to a new index based on size, document count, or age is preferable -to time-based rollovers. Rolling over at an arbitrary time often results in -many small indices, which can have a negative impact on performance and -resource usage. - You control when the rollover action is triggered by specifying one or more rollover parameters. The rollover is performed once any of the criteria are met. Because the criteria are checked periodically, the index might grow @@ -22,7 +12,7 @@ checked, specify the `indices.lifecycle.poll_interval` cluster setting. IMPORTANT: New indices created via rollover will not automatically inherit the policy used by the old index, and will not use any policy by default. Therefore, it is highly recommended to apply the policy via -<>, including a Rollover alias +<>, including a Rollover alias setting, for your indices which specifies the policy you wish to use for each new index. @@ -146,7 +136,7 @@ an exception to your normal Lifecycle Policy and switching the alias to a different index by hand, but do not want to remove the index from {ilm} completely. -This setting is set to `true` automatically by ILM upon the successful +This setting is set to `true` automatically by {ilm-init} upon the successful completion of a Rollover Action. However, it will be removed if <> from the index. diff --git a/docs/reference/redirects.asciidoc b/docs/reference/redirects.asciidoc index 46de6ae62bc..2fe44ac964c 100644 --- a/docs/reference/redirects.asciidoc +++ b/docs/reference/redirects.asciidoc @@ -697,3 +697,13 @@ This section was deleted. === Assertions This page was deleted. + +[role="exclude",id="_actions"] +=== {ilm-init} actions + +See <>. + +[role="exclude",id="ilm-policy-definition"] +=== {ilm-init} policy definition + +See <>.