honeymoose
/
OpenSearch
mirror of https://github.com/honeymoose/OpenSearch.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

ILM update backports (#55902) 

* [DOCS] Rework conceptual info for ILM. (#52181)

* [DOCS] Rework conceptual info for ILM.

* Split the actions out of concepts.

* Added xpack role to actions.

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Apply suggestions from code review

* Edit actions for consistency and add action template. (#55632)

* Edit actions for consistency and add action template.

* Update docs/reference/ilm/actions/ilm-readonly.asciidoc

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>

* Apply suggestions from code review
This commit is contained in:
debadair 2020-04-28 16:38:01 -07:00 committed by GitHub
parent 8d1595698b
commit 8a662c7e62
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
34 changed files with 1207 additions and 1494 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

94
docs/reference/ilm/actions/_ilm-action-template.asciidoc Normal file
View File

@ -0,0 +1,94 @@
////
This is a template for ILM action reference documentation.
To document a new action, copy this file, remove comments like this, and
replace "sample" with the appropriate action name.
Ensure the new action docs are linked and included in
docs/reference/ilm/actions.asciidoc
////
[role="xpack"]
[[ilm-sample]]
=== Sample
Phases allowed: hot, warm, cold, delete.
////
INTRO
Include a brief, 1-2 sentence description.
////
Does a cool thing.
[[ilm-sample-options]]
==== Options
////
Definition list of the options that can be specified for the action:
If there are no options:
None.
////
`sample_option1`::
(Optional, integer)
Number of something.
`sample_option2`::
(Required, string)
Name of something.
[[ilm-sample-ex]]
==== Example
////
Basic example of configuring the action in an ILM policy.
Additional examples are optional.
////
[source,console]
--------------------------------------------------
PUT _ilm/policy/my_policy
{
"policy": {
"phases": {
"warm": {
"actions": {
"sample" : {
"sample_option1" : 2
}
}
}
}
}
}
--------------------------------------------------
// TEST[skip: Replace fake actions and remove this comment.]
[[ilm-sample2-ex]]
===== Describe example
The sample action in the following policy does something interesting.
[source,console]
--------------------------------------------------
PUT _ilm/policy/my_policy
{
"policy": {
"phases": {
"warm": {
"actions": {
"sample" : {
"sample_option1" : 100,
"sample_option2" : "interesting"
}
}
}
}
}
}
--------------------------------------------------
// TEST[skip: Replace fake actions and remove this comment.]

130
docs/reference/ilm/actions/ilm-allocate.asciidoc Normal file
View File

@ -0,0 +1,130 @@
[role="xpack"]
[[ilm-allocate]]
=== Allocate
Phases allowed: warm, cold.
Updates the index settings to change which nodes are allowed to host the index shards
and change the number of replicas.
You can configure this action to modify both the allocation rules and number of replicas,
only the allocation rules, or only the number of replicas.
For more information about how {es} uses replicas for scaling, see
<<scalability>>. See <<shard-allocation-filtering>> for more information about
controlling where {es} allocates shards of a particular index.
NOTE: The allocate action is not allowed in the hot phase.
The initial allocation for the index must be done manually or via index templates.
{ilm-init} doesn't handle index allocation during the hot phase.
[[ilm-allocate-options]]
==== Options
You must specify the number of replicas or at least one
`include`, `exclude`, or `require` option.
An empty allocate action is invalid.
For more information about using custom attributes for shard allocation,
see <<shard-allocation-filtering>>.
`number_of_replicas`::
(Optional, integer)
Number of replicas to assign to the index.
`include`::
(Optional, object)
Assigns an index to nodes that have at least _one_ of the specified custom attributes.
`exclude`::
(Optional, object)
Assigns an index to nodes that have _none_ of the specified custom attributes.
`require`::
(Optional, object)
Assigns an index to nodes that have _all_ of the specified custom attributes.
[[ilm-allocate-ex]]
==== Example
The allocate action in the following policy changes the index's number of replicas to `2`.
The index allocation rules are not changed.
[source,console]
--------------------------------------------------
PUT _ilm/policy/my_policy
{
"policy": {
"phases": {
"warm": {
"actions": {
"allocate" : {
"number_of_replicas" : 2
}
}
}
}
}
}
--------------------------------------------------
[[ilm-allocate-assign-index-attribute-ex]]
===== Assign index to nodes using a custom attribute
The allocate action in the following policy assigns the index to nodes
that have a `box_type` of _hot_ or _warm_.
To designate a node's `box_type`, you set a custom attribute in the node configuration.
For example, set `node.attr.box_type: hot` in `elasticsearch.yml`.
For more information, see <<index-allocation-filters>>.
[source,console]
--------------------------------------------------
PUT _ilm/policy/my_policy
{
"policy": {
"phases": {
"warm": {
"actions": {
"allocate" : {
"include" : {
"box_type": "hot,warm"
}
}
}
}
}
}
}
--------------------------------------------------
[[ilm-allocate-assign-index-node-ex]]
===== Assign index to a specific node and update replica settings
The allocate action in the following policy updates the index to have one replica per shard
and be allocated to nodes that have a `box_type` of _cold_.
To designate a node's `box_type`, you set a custom attribute in the node configuration.
For example, set `node.attr.box_type: cold` in `elasticsearch.yml`.
For more information, see <<index-allocation-filters>>.
[source,console]
--------------------------------------------------
PUT _ilm/policy/my_policy
{
"policy": {
"phases": {
"warm": {
"actions": {
"allocate" : {
"number_of_replicas": 1,
"require" : {
"box_type": "cold"
}
}
}
}
}
}
}
--------------------------------------------------

36
docs/reference/ilm/actions/ilm-delete.asciidoc Normal file
View File

@ -0,0 +1,36 @@
[role="xpack"]
[[ilm-delete]]
=== Delete
Phases allowed: delete.
Permanently removes the index.
[[ilm-delete-options]]
==== Options
`delete_searchable_snapshot`::
(Optional, boolean)
Deletes the searchable snapshot created in the cold phase.
Defaults to `true`.
This option is applicable when the <<ilm-searchable-snapshot-action,searchable
snapshot>> action is used in the cold phase.
[[ilm-delete-action-ex]]
==== Example
[source,console]
--------------------------------------------------
PUT _ilm/policy/my_policy
{
"policy": {
"phases": {
"delete": {
"actions": {
"delete" : { }
}
}
}
}
}
--------------------------------------------------

52
docs/reference/ilm/actions/ilm-forcemerge.asciidoc Normal file
View File

@ -0,0 +1,52 @@
[role="xpack"]
[[ilm-forcemerge]]
=== Force merge
Phases allowed: hot, warm.
<<indices-forcemerge,Force merges>> the index into
the specified maximum number of <<indices-segments,segments>>.
This action makes the index <<dynamic-index-settings,read-only>>.
To use the `forcemerge` action in the `hot` phase, the `rollover` action *must* be present.
If no rollover action is configured, {ilm-init} will reject the policy.
[[ilm-forcemerge-options]]
==== Options
`max_num_segments`::
(Required, integer)
Number of segments to merge to. To fully merge the index, set to `1`.
`codec`::
(Optional, string)
Use the `best_compression` codec. Valid values: `best_compression`.
+
[WARNING]
======
Setting `"codec": "best_compression"` in the {ilm-init} forcemerge action causes {ilm-int} to
<<indices-close,close>> and then <<indices-open-close,re-open>> the index prior to the force merge.
During this time, the index is unavailable for both read and write operations.
======
[[ilm-forcemerge-action-ex]]
==== Example
[source,console]
--------------------------------------------------
PUT _ilm/policy/my_policy
{
"policy": {
"phases": {
"warm": {
"actions": {
"forcemerge" : {
"max_num_segments": 1
}
}
}
}
}
}
--------------------------------------------------

40
docs/reference/ilm/actions/ilm-freeze.asciidoc Normal file
View File

@ -0,0 +1,40 @@
[role="xpack"]
[[ilm-freeze]]
=== Freeze
Phases allowed: cold.
<<frozen-indices, Freezes>> an index to minimize its memory footprint.
[IMPORTANT]
================================
Freezing an index closes the index and reopens it within the same API call.
This means that for a short time no primaries are allocated.
The cluster will go red until the primaries are allocated.
This limitation might be removed in the future.
================================
[[ilm-freeze-options]]
==== Options
None.
[[ilm-freeze-ex]]
==== Example
[source,console]
--------------------------------------------------
PUT _ilm/policy/my_policy
{
"policy": {
"phases": {
"cold": {
"actions": {
"freeze" : { }
}
}
}
}
}
--------------------------------------------------

31
docs/reference/ilm/actions/ilm-readonly.asciidoc Normal file
View File

@ -0,0 +1,31 @@
[role="xpack"]
[[ilm-readonly]]
=== Read only
Phases allowed: warm.
Makes the index <<dynamic-index-settings,read-only>>.
[[ilm-read-only-options]]
==== Options
None.
[[ilm-read-only-ex]]
==== Example
[source,console]
--------------------------------------------------
PUT _ilm/policy/my_policy
{
"policy": {
"phases": {
"warm": {
"actions": {
"readonly" : { }
}
}
}
}
}
--------------------------------------------------

187
docs/reference/ilm/actions/ilm-rollover.asciidoc Normal file
View File

@ -0,0 +1,187 @@
[role="xpack"]
[[ilm-rollover]]
=== Rollover
Phases allowed: hot.
Rolls an alias over to a new index when the existing index meets one of the rollover conditions.
IMPORTANT: If the rollover action is used on a <<ccr-put-follow,follower index>>,
policy execution waits until the leader index rolls over (or is
<<skipping-rollover, otherwise marked complete>>),
then converts the follower index into a regular index with the
<<ilm-unfollow-action,the Unfollow action>>.
For a managed index to be rolled over:
* The index name must match the pattern '^.*-\\d+$', for example (`my_index-000001`).
* `index.lifecycle.rollover_alias` must be configured as the alias to roll over.
* The index must be the <<<<indices-rollover-is-write-index, write index>> for the alias.
For example, if `my_index-000001` has the alias `my_data`,
the following settings must be configured.
[source,console]
--------------------------------------------------
PUT my_index-000001
{
"settings": {
"index.lifecycle.name": "my_policy",
"index.lifecycle.rollover_alias": "my_data"
},
"aliases": {
"my_data": {
"is_write_index": true
}
}
}
--------------------------------------------------
[[ilm-rollover-options]]
==== Options
You must specify at least one rollover condition.
An empty rollover action is invalid.
`max_size`::
(Optional, <<byte-units, byte units>>)
Triggers roll over after the specified maximum primary shard index storage size is reached.
`max_docs`::
(Optional, integer)
Triggers roll over after the specified maximum number of documents in the index is reached.
`max_age`::
(Optional, <<time-units, time units>>)
Triggers roll over after the maximum elapsed time from index creation is reached.
[[ilm-rollover-ex]]
==== Example
[[ilm-rollover-size-ex]]
===== Roll over based on index size
This example rolls the index over when it is at least 100 gigabytes.
[source,console]
--------------------------------------------------
PUT _ilm/policy/my_policy
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover" : {
"max_size": "100GB"
}
}
}
}
}
}
--------------------------------------------------
[ilm-rollover-documents-ex]]
===== Roll over based on document count
This example rolls the index over when it contains at least one hundred million documents.
[source,console]
--------------------------------------------------
PUT _ilm/policy/my_policy
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover" : {
"max_docs": 100000000
}
}
}
}
}
}
--------------------------------------------------
[ilm-rollover-age-ex]]
===== Roll over based on index age
This example rolls the index over if it was created at least 7 days ago.
[source,console]
--------------------------------------------------
PUT _ilm/policy/my_policy
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover" : {
"max_age": "7d"
}
}
}
}
}
}
--------------------------------------------------
[ilm-rollover-conditions-ex]]
===== Roll over using multiple conditions
When you specify multiple rollover conditions,
the index is rolled over when _any_ of the conditions are met.
This example rolls the index over if it is at least 7 days old or at least 100 gigabytes.
[source,console]
--------------------------------------------------
PUT _ilm/policy/my_policy
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover" : {
"max_age": "7d",
"max_size": "100GB"
}
}
}
}
}
}
--------------------------------------------------
[ilm-rollover-block-ex]]
===== Rollover condition blocks phase transition
The rollover action only completes if one of its conditions is met.
This means that any subsequent phases are blocked until rollover succeeds.
For example, the following policy deletes the index one day after it rolls over.
It does not delete the index one day after it was created.
[source,console]
--------------------------------------------------
PUT /_ilm/policy/rollover_policy
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": {
"max_size": "50G"
}
}
},
"delete": {
"min_age": "1d",
"actions": {
"delete": {}
}
}
}
}
}
--------------------------------------------------

39
docs/reference/ilm/actions/ilm-searchable-snapshot.asciidoc Normal file
View File

@ -0,0 +1,39 @@
[role="xpack"]
[[ilm-searchable-snapshot]]
=== Searchable snapshot
Phases allowed: cold.
Takes a snapshot of the managed index in the configured repository
and mounts it as a searchable snapshot.
By default, this snapshot is deleted by the <<ilm-delete-action, delete action>> in the delete phase.
To keep the snapshot, set `delete_searchable_snapshot` to `false` in the delete action.
[[ilm-searchable-snapshot-options]]
==== Options
`snapshot_repository`::
(Required, string)
Specifies where to store the snapshot.
See <<snapshots-register-repository>> for more information.
[[ilm-searchable-snapshot-ex]]
==== Examples
[source,console]
--------------------------------------------------
PUT _ilm/policy/my_policy
{
"policy": {
"phases": {
"cold": {
"actions": {
"searchable_snapshot" : {
"snapshot_repository" : "backing_repo"
}
}
}
}
}
}
--------------------------------------------------

44
docs/reference/ilm/actions/ilm-set-priority.asciidoc Normal file
View File

@ -0,0 +1,44 @@
[role="xpack"]
[[ilm-set-priority]]
=== Set priority
Phases allowed: hot, warm, cold.
Sets the <<recovery-prioritization, priority>> of the index as
soon as the policy enters the hot, warm, or cold phase.
Higher priority indices are recovered before indices with lower priorities following a node restart.
Generally, indexes in the hot phase should have the highest value and
indexes in the cold phase should have the lowest values.
For example: 100 for the hot phase, 50 for the warm phase, and 0 for the cold phase.
Indices that don't set this value have a default priority of 1.
[[ilm-set-priority-options]]
==== Options
`priority`::
(Required, integer)
The priority for the index.
Must be 0 or greater.
Set to `null` to remove the priority.
[[ilm-set-priority-ex]]
==== Example
[source,console]
--------------------------------------------------
PUT _ilm/policy/my_policy
{
"policy": {
"phases": {
"warm": {
"actions": {
"set_priority" : {
"priority": 50
}
}
}
}
}
}
--------------------------------------------------

51
docs/reference/ilm/actions/ilm-shrink.asciidoc Normal file
View File

@ -0,0 +1,51 @@
[role="xpack"]
[[ilm-shrink]]
=== Shrink
Phases allowed: warm
Sets an index to <<dynamic-index-settings, read-only>>
and shrinks it into a new index with fewer primary shards.
The name of the new index is of the form `shrink-<original-index-name>`.
For example, if the name of the source index is _logs_,
the name of the shrunken index is _shrink-logs_.
The shrink action allocates all primary shards of the index to one node so it
can call the <<indices-shrink-index,Shrink API>> to shrink the index.
After shrinking, it swaps aliases that point to the original index to the new shrunken index.
[IMPORTANT]
If the shrink action is used on a <<ccr-put-follow,follower index>>,
policy execution waits until the leader index rolls over (or is
<<skipping-rollover, otherwise marked complete>>),
then converts the follower index into a regular index with the
<<ilm-unfollow-action,the Unfollow action>> before performing the shrink operation.
[[ilm-shrink-options]]
==== Shrink options
`number_of_shards`::
(Required, integer)
Number of shards to shrink to.
Must be a factor of the number of shards in the source index.
[[ilm-shrink-ex]]
==== Example
[source,console]
--------------------------------------------------
PUT _ilm/policy/my_policy
{
"policy": {
"phases": {
"warm": {
"actions": {
"shrink" : {
"number_of_shards": 1
}
}
}
}
}
}
--------------------------------------------------

58
docs/reference/ilm/actions/ilm-unfollow.asciidoc Normal file
View File

@ -0,0 +1,58 @@
[role="xpack"]
[[ilm-unfollow]]
=== Unfollow
Phases allowed: hot, warm, cold.
Converts a {ref}/ccr-apis.html[{ccr-init}] follower index into a regular index.
This enables the shrink, rollover, and searchable snapshot actions
to be be performed safely on follower indices.
You can also use unfollow directly when moving follower indices through the lifecycle.
Has no effect on indices that are not followers, phase execution just moves to the next action.
[NOTE]
This action is triggered automatically by the
<<ilm-rollover-action, rollover>>, <<ilm-shrink-action, shrink>>,
and <<ilm-searchable-snapshot-action, searchable snapshot>> actions
when they are applied to follower indices.
This action waits until is it safe to convert a follower index into a regular index.
The following conditions must be met:
* The leader index must have `index.lifecycle.indexing_complete` set to `true`.
This happens automatically if the leader index is rolled over using the
<<ilm-rollover-action, rollover>> action, and can be set manually using
the <<indices-update-settings,index settings>> API.
* All operations performed on the leader index have been replicated to the follower index.
This ensures that no operations are lost when the index is converted.
Once these conditions are met, unfollow performs the following operations:
* Pauses indexing following for the follower index.
* Closes the follower index.
* Unfollows the leader index.
* Opens the follower index (which is at this point is a regular index).
[[ilm-unfollow-options]]
==== Options
None.
[[ilm-unfollow-ex]]
==== Example
[source,console]
--------------------------------------------------
PUT _ilm/policy/my_policy
{
"policy": {
"phases": {
"hot": {
"actions": {
"unfollow" : {}
}
}
}
}
}
--------------------------------------------------

36
docs/reference/ilm/actions/ilm-wait-for-snapshot.asciidoc Normal file
View File

@ -0,0 +1,36 @@
[role="xpack"]
[[ilm-wait-for-snapshot]]
=== Wait for snapshot
Phases allowed: delete.
Waits for the specified {slm-init} policy to be executed before removing the index.
This ensures that a snapshot of the deleted index is available.
[[ilm-wait-for-snapshot-options]]
==== Options
`policy`::
(Required, string)
Name of the {slm-init} policy that the delete action should wait for.
[[ilm-wait-for-snapshot-ex]]
==== Example
[source,console]
--------------------------------------------------
PUT _ilm/policy/my_policy
{
"policy": {
"phases": {
"delete": {
"actions": {
"wait_for_snapshot" : {
"policy": "slm-policy-name"
}
}
}
}
}
}
--------------------------------------------------

14
docs/reference/ilm/apis/explain.asciidoc
View File

@ -39,11 +39,11 @@ about any failures.
`only_managed`::
(Optional, boolean) Filters the returned indices to only indices that are managed by
ILM.
{ilm-init}.
`only_errors`::
(Optional, boolean) Filters the returned indices to only indices that are managed by
ILM and are in an error state, either due to an encountering an error while
{ilm-init} and are in an error state, either due to an encountering an error while
executing the policy, or attempting to use a policy that does not exist.
include::{docdir}/rest-api/common-parms.asciidoc[tag=timeoutparms]
@ -99,7 +99,7 @@ GET my_index/_ilm/explain
--------------------------------------------------
// TEST[continued]
When management of the index is first taken over by ILM, `explain` shows
When management of the index is first taken over by {ilm-init}, `explain` shows
that the index is managed and in the `new` phase:
[source,console-result]
@ -124,9 +124,9 @@ that the index is managed and in the `new` phase:
--------------------------------------------------
// TESTRESPONSE[skip:no way to know if we will get this response immediately]
<1> Shows if the index is being managed by ILM. If the index is not managed by
ILM the other fields will not be shown
<2> The name of the policy which ILM is using for this index
<1> Shows if the index is being managed by {ilm-init}. If the index is not managed by
{ilm-init} the other fields will not be shown
<2> The name of the policy which {ilm-init} is using for this index
<3> The timestamp used for the `min_age`
<4> The age of the index (used for calculating when to enter the next phase)
<5> When the index entered the current phase
@ -302,7 +302,7 @@ the case.
<1> The step that caused the error
<2> Indicates if retrying the failed step can overcome the error. If this
is true, ILM will retry the failed step automatically.
is true, {ilm-init} will retry the failed step automatically.
<3> Shows the number of attempted automatic retries to execute the failed
step.
<4> What went wrong

2
docs/reference/ilm/apis/get-status.asciidoc
View File

@ -28,7 +28,7 @@ Retrieves the current {ilm} ({ilm-init}) status.
Returns the status of the {ilm-init} plugin. The `operation_mode` field in the
response shows one of three states: `STARTED`, `STOPPING`,
or `STOPPED`. You can change the status of the {ilm-init} plugin with the
<<ilm-start,start ILM>> and <<ilm-stop,stop ILM>> APIs.
<<ilm-start,start {ilm-init}>> and <<ilm-stop,stop {ilm-init}>> APIs.
[[ilm-get-status-query-params]]
==== {api-query-parms-title}

12
docs/reference/ilm/apis/ilm-api.asciidoc
View File

@ -4,7 +4,7 @@
You use the following APIs to set up policies to automatically manage the index lifecycle.
For more information about {ilm} ({ilm-init}), see <<index-lifecycle-management>>.
[float]
[discrete]
[[ilm-api-policy-endpoint]]
=== Policy management APIs
@ -12,7 +12,7 @@ For more information about {ilm} ({ilm-init}), see <<index-lifecycle-management>
* <<ilm-get-lifecycle,Get lifecycle policy>>
* <<ilm-delete-lifecycle,Delete lifecycle policy>>
[float]
[discrete]
[[ilm-api-index-endpoint]]
=== Index management APIs
@ -20,13 +20,13 @@ For more information about {ilm} ({ilm-init}), see <<index-lifecycle-management>
* <<ilm-retry-policy,Retry policy on indices>>
* <<ilm-remove-policy,Remove policy from index>>
[float]
[discrete]
[[ilm-api-management-endpoint]]
=== Operation management APIs
* <<ilm-get-status,Get ILM operation mode>>
* <<ilm-start,Start ILM>>
* <<ilm-stop,Stop ILM>>
* <<ilm-get-status,Get {ilm-init} operation mode>>
* <<ilm-start,Start {ilm-init}>>
* <<ilm-stop,Stop {ilm-init}>>
* <<ilm-explain-lifecycle,Explain API>>

2
docs/reference/ilm/apis/retry-policy.asciidoc
View File

@ -24,7 +24,7 @@ see <<security-privileges>>.
==== {api-description-title}
Sets the policy back to the step where the error occurred and executes the step.
Use the <<ilm-explain-lifecycle, ILM Explain API>> to determine if an index is in the ERROR
Use the <<ilm-explain-lifecycle, {ilm-init} Explain API>> to determine if an index is in the ERROR
step.
[[ilm-retry-policy-path-params]]

2
docs/reference/ilm/apis/start.asciidoc
View File

@ -37,7 +37,7 @@ include::{docdir}/rest-api/common-parms.asciidoc[tag=timeoutparms]
[[ilm-start-example]]
==== {api-examples-title}
The following example starts the ILM plugin.
The following example starts the {ilm-init} plugin.
//////////////////////////

4
docs/reference/ilm/apis/stop.asciidoc
View File

@ -31,7 +31,7 @@ is useful when you are performing maintenance on the cluster and need to prevent
The API returns as soon as the stop request has been acknowledged, but the
plugin might continue to run until in-progress operations complete and the plugin
can be safely stopped. Use the <<ilm-get-status, Get ILM Status>> API to see
can be safely stopped. Use the <<ilm-get-status, Get {ilm-init} Status>> API to see
if {ilm-init} is running.
[[ilm-stop-query-params]]
@ -42,7 +42,7 @@ include::{docdir}/rest-api/common-parms.asciidoc[tag=timeoutparms]
[[ilm-stop-example]]
==== {api-examples-title}
The following example stops the ILM plugin.
The following example stops the {ilm-init} plugin.
//////////////////////////

14
docs/reference/ilm/error-handling.asciidoc
View File

@ -5,7 +5,7 @@
During Index Lifecycle Management's execution of the policy for an index, it's
possible for a step to encounter an error during its execution. When this
happens, ILM will move the management state into an "error" step. This halts
happens, {ilm-init} will move the management state into an "error" step. This halts
further execution of the policy and gives an administrator the chance to address
any issues with the policy, index, or cluster.
@ -50,11 +50,11 @@ PUT /myindex
--------------------------------------------------
// TEST[continued]
After five days have passed, ILM will attempt to shrink this index from 2
After five days have passed, {ilm-init} will attempt to shrink this index from 2
shards to 4, which is invalid since the shrink action cannot increase the
number of shards. When this occurs, ILM will move this
number of shards. When this occurs, {ilm-init} will move this
index to the "error" step. Once an index is in this step, information about the
reason for the error can be retrieved from the <<ilm-explain-lifecycle,ILM Explain API>>:
reason for the error can be retrieved from the <<ilm-explain-lifecycle,{ilm-init} Explain API>>:
[source,console]
--------------------------------------------------
@ -104,7 +104,7 @@ Which returns the following information:
--------------------------------------------------
// TESTRESPONSE[skip:no way to know if we will get this response immediately]
<1> this index is managed by ILM
<1> this index is managed by {ilm-init}
<2> the policy in question, in this case, "shrink-the-index"
<3> the current age for the index
<4> what phase the index is currently in
@ -140,7 +140,7 @@ PUT _ilm/policy/shrink-the-index
--------------------------------------------------
// TEST[continued]
[float]
[discrete]
=== Retrying failed index lifecycle management steps
Once the underlying issue that caused an index to move to the error step has
@ -155,5 +155,5 @@ POST /myindex/_ilm/retry
Once this has been issue, index lifecycle management will asynchronously pick up
on the step that is in a failed state, attempting to re-run it. The
<<ilm-explain-lifecycle,ILM Explain API>> can again be used to monitor the status of
<<ilm-explain-lifecycle,{ilm-init} Explain API>> can again be used to monitor the status of
re-running the step.

56
docs/reference/ilm/ilm-actions.asciidoc Normal file
View File

@ -0,0 +1,56 @@
[role="xpack"]
[testenv="basic"]
[[ilm-actions]]
== Index lifecycle actions
[[ilm-allocate-action]]
<<ilm-allocate,Allocate>>::
Move shards to nodes with different performance characteristics
and reduce the number of replicas.
[[ilm-delete-action]]<<ilm-delete,Delete>>::
Permanently remove the index.
[[ilm-forcemerge-action]]<<ilm-forcemerge,Force Merge>>::
Reduce the number of index segments and purge deleted documents.
Makes the index read-only.
[[ilm-freeze-action]]<<ilm-freeze,Freeze>>::
Freeze the index to minimize its memory footprint.
[[ilm-readonly-action]]<<ilm-readonly,Read-Only>>::
Block write operations to the index.
[[ilm-rollover-action]]<<ilm-rollover,Rollover>>::
Remove the index as the write index for the rollover alias and
start indexing to a new index.
[[ilm-searchable-snapshot-action]]<<ilm-searchable-snapshot, Searchable Snapshot>>::
Take a snapshot of the managed index in the configured repository
and mount it as a searchable snapshot.
[[ilm-set-priority-action]]<<ilm-set-priority,Set Priority>>::
Lower the priority of an index as it moves through the lifecycle
to ensure that hot indices are recovered first.
[[ilm-shrink-action]]<<ilm-shrink,Shrink>>::
Reduce the number of primary shards by shrinking the index into a new index.
[[ilm-unfollow-action]]<<ilm-unfollow,Unfollow>>::
Convert a follower index to a regular index.
Performed automatically before a rollover, shrink, or searchable snapshot action.
[[ilm-wait-for-snapshot-action]]<<ilm-wait-for-snapshot,Wait For Snapshot>>::
Ensure that a snapshot exists before deleting the index.
include::actions/ilm-allocate.asciidoc[]
include::actions/ilm-delete.asciidoc[]
include::actions/ilm-forcemerge.asciidoc[]
include::actions/ilm-freeze.asciidoc[]
include::actions/ilm-readonly.asciidoc[]
include::actions/ilm-rollover.asciidoc[]
include::actions/ilm-searchable-snapshot.asciidoc[]
include::actions/ilm-set-priority.asciidoc[]
include::actions/ilm-shrink.asciidoc[]
include::actions/ilm-unfollow.asciidoc[]
include::actions/ilm-wait-for-snapshot.asciidoc[]

4
docs/reference/ilm/ilm-and-snapshots.asciidoc
View File

@ -25,9 +25,9 @@ paused>> to allow the policy to be removed.
For example, the following workflow can be used in the above situation to
prevent the execution of the lifecycle policy for an index:
1. Pause execution of all lifecycle policies using the <<ilm-stop,Stop ILM API>>
1. Pause execution of all lifecycle policies using the <<ilm-stop,Stop {ilm-init} API>>
2. Restore the snapshot.
3. Perform whatever operations you wish before resuming lifecycle execution, or
remove the lifecycle policy from the index using the
<<ilm-remove-policy,Remove Policy from Index API>>
4. Resume execution of lifecycle policies using the <<ilm-start,Start ILM API>>
4. Resume execution of lifecycle policies using the <<ilm-start,Start {ilm-init} API>>

16
docs/reference/ilm/ilm-concepts.asciidoc Normal file
View File

@ -0,0 +1,16 @@
[role="xpack"]
[testenv="basic"]
[[ilm-concepts]]
== {ilm-init} concepts
++++
<titleabbrev>Concepts</titleabbrev>
++++
* <<ilm-index-lifecycle>>
* <<index-rollover>>
* <<update-lifecycle-policy>>
include::ilm-index-lifecycle.asciidoc[]
include::index-rollover.asciidoc[]
include::update-lifecycle-policy.asciidoc[]

77
docs/reference/ilm/ilm-index-lifecycle.asciidoc Normal file
View File

@ -0,0 +1,77 @@
[role="xpack"]
[testenv="basic"]
[[ilm-index-lifecycle]]
=== Index lifecycle
++++
<titleabbrev>Index lifecycle</titleabbrev>
++++
{ilm-init} defines four index lifecycle _phases_:
* Hot: The index is actively being updated and queried.
* Warm: The index is no longer being updated but is still being queried.
* Cold: The index is no longer being updated and is seldom queried. The
information still needs to be searchable, but it's okay if those queries are
slower.
* Delete: The index is no longer needed and can safely be removed.
An index's _lifecycle policy_ specifies which phases
are applicable, what actions are performed in each phase,
and when it transitions between phases.
You can manually apply a lifecycle policy when you create an index.
For time series indices, you need to associate the lifecycle policy with
the index template used to create new indices in the series.
When an index rolls over, a manually-applied policy isn't automatically applied to the new index.
[discrete]
[[ilm-phase-transitions]]
=== Phase transitions
{ilm-init} moves indices through the lifecycle according to their age.
To control the timing of these transitions, you set a _minimum age_ for each phase.
For an index to move to the next phase, all actions in the current phase must be complete and
the index must be older than the minimum age of the next phase.
The minimum age defaults to zero, which causes {ilm-init} to move indices to the next phase
as soon as all actions in the current phase complete.
[discrete]
[[ilm-phase-execution]]
=== Phase execution
{ilm-init} controls the order in which the actions in a phase are executed and
what _steps_ are executed to perform the necessary index operations for each action.
When an index enters a phase, {ilm-init} caches the phase definition in the index metadata.
This ensures that policy updates don't put the index into a state where it can never exit the phase.
If changes can be safely applied, {ilm-init} updates the cached phase definition.
If they cannot, phase execution continues using the cached definition.
[discrete]
[[ilm-phase-actions]]
=== Phase actions
{ilm-init} supports the following actions in each phase.
* Hot
- <<ilm-set-priority,Set Priority>>
- <<ilm-unfollow,Unfollow>>
- <<ilm-rollover,Rollover>>
* Warm
- <<ilm-set-priority,Set Priority>>
- <<ilm-unfollow,Unfollow>>
- <<ilm-readonly,Read-Only>>
- <<ilm-allocate,Allocate>>
- <<ilm-shrink,Shrink>>
- <<ilm-forcemerge,Force Merge>>
* Cold
- <<ilm-set-priority-action,Set Priority>>
- <<ilm-unfollow-action,Unfollow>>
- <<ilm-allocate,Allocate>>
- <<ilm-freeze,Freeze>>
- <<ilm-searchable-snapshot, Searchable Snapshot>>
* Delete
- <<ilm-wait-for-snapshot-action,Wait For Snapshot>>
- <<ilm-delete,Delete>>

24
docs/reference/ilm/overview-ilm.asciidoc → docs/reference/ilm/ilm-overview.asciidoc
View File

@ -1,7 +1,7 @@
[role="xpack"]
[testenv="basic"]
[[overview-index-lifecycle-management]]
== Index lifecycle management overview
== {ilm-init} overview
++++
<titleabbrev>Overview</titleabbrev>
@ -21,29 +21,15 @@ include::../glossary.asciidoc[tag=force-merge-def-short]
include::../glossary.asciidoc[tag=freeze-def-short]
* **Delete** - Permanently remove an index, including all of its data and metadata.
Typically, you associate a lifecycle policy with an index template so it is automatically applied
to new indices.
You can also apply a policy manually when you create an index.
{ilm-init} simplifies managing indices in hot-warm-cold architectures,
{ilm-init} makes it easier to manage indices in hot-warm-cold architectures,
which are common when you're working with time-series data such as logs and metrics.
As an index ages, it moves through four possible phases:
You can specify:
* Hot--the index is actively being updated and queried.
* Warm--the index is no longer being updated, but is still being queried.
* Cold--the index is no longer being updated and is seldom queried. The
information still needs to be searchable, but it's okay if those queries are
slower.
* Delete--the index is no longer needed and can safely be deleted.
A lifecycle policy controls how an index moves between phases and
what actions to perform during each phase. You can specify:
* The maximum size or age at which you want to roll over to a new index.
* The maximum size, number of documents, or age at which you want to roll over to a new index.
* The point at which the index is no longer being updated and the number of
primary shards can be reduced.
* When to force a merge to permanently delete documents marked for deletion.
* When to force a merge to permanently remove documents marked for deletion.
* The point at which the index can be moved to less performant hardware.
* The point at which the availability is not as critical and the number of
replicas can be reduced.

80
docs/reference/ilm/getting-started-ilm.asciidoc → docs/reference/ilm/ilm-tutorial.asciidoc
View File

@ -1,15 +1,14 @@
[role="xpack"]
[testenv="basic"]
[[getting-started-index-lifecycle-management]]
== Get started: Automate rollover with {ilm-init}
== Tutorial: Automate rollover with {ilm-init}
++++
<titleabbrev>Automate rollover</titleabbrev>
++++
This tutorial demonstrates how to use {ilm} ({ilm-init})
to manage indices that contain time-series data.
This tutorial demonstrates how to use {ilm}
({ilm-init}) to manage indices that contain time-series data.
When you continuously index timestamped documents into {es} using
Filebeat, Logstash, or some other mechanism,
@ -20,27 +19,42 @@ and still get the most out of your data.
To automate rollover and management of time-series indices with {ilm-init}, you:
. <<ilm-gs-create-policy, Create a lifecycle policy>> with the {ilm-init} put policy API.
. <<ilm-gs-create-policy, Create a lifecycle policy>> that defines the appropriate
phases and actions.
. <<ilm-gs-apply-policy, Create an index template>> to apply the policy to each new index.
. <<ilm-gs-bootstrap, Bootstrap an index>> as the initial write index.
. <<ilm-gs-check-progress, Verify indices are moving through the lifecycle phases>>
as expected with the {ilm-init} explain API.
. <<ilm-gs-check-progress, Verify indices are moving through the lifecycle phases>>
as expected.
[float]
For an introduction to rolling indices, see <<index-rollover>>.
NOTE: {filebeat} includes a default {ilm-init} policy that initiates the rollover action when
the index size reaches 50GB or becomes 30 days old.
You can use this policy as a starting point, or replace it with a custom policy.
See
{kib}/example-using-index-lifecycle-policy.html[Use {ilm-init} to manage Filebeat time-based indices].
[discrete]
[[ilm-gs-create-policy]]
=== Create a lifecycle policy
A lifecycle policy specifies the phases in the index lifecycle
and the actions to perform in each phase. A lifecycle can have up to four phases:
`hot`, `warm`, `cold`, and `delete`. Policies are defined in JSON
and added through the {ilm-init} put policy API.
`hot`, `warm`, `cold`, and `delete`.
For example, the following request creates a `timeseries_policy` with two phases:
You can define and manage policies through the {kib} Management UI,
which invokes the {ilm-init} <<ilm-put-lifecycle, put policy>> API to create policies
according to the options you specify.
* The `hot` phase defines a `rollover` action to specify that an index rolls over when it
For example, you might define a `timeseries_policy` that has two phases:
* A `hot` phase that defines a rollover action to specify that an index rolls over when it
reaches either a `max_size` of 50 gigabytes or a `max_age` of 30 days.
* The `delete` phase uses `min_age` to remove the index 90 days after rollover.
Note that this value is relative to the rollover time, not the index creation time.
* A `delete` phase that sets `min_age` to remove the index 90 days after rollover.
Note that this value is relative to the rollover time, not the index creation time.
The underlying put policy request looks like this:
[source,console]
------------------------
@ -71,24 +85,32 @@ PUT _ilm/policy/timeseries_policy
<3> Move the index into the `delete` phase 90 days after rollover.
<4> Trigger the `delete` action when the index enters the delete phase.
See <<_actions>> for the complete list of actions available in each phase.
You can also invoke this API directly to add lifecycle policies.
[float]
For the complete list of actions that {ilm} can perform, see <<ilm-actions>>.
[discrete]
[[ilm-gs-apply-policy]]
=== Create an index template to apply the lifecycle policy
To automaticaly apply a lifecycle policy to the new write index on rollover,
To automatically apply a lifecycle policy to the new write index on rollover,
specify the policy in the index template used to create new indices.
For example, the following request creates a `timeseries_template` that is applied to new indices
whose names match the `timeseries-*` index pattern.
The template configures two {ilm-init} settings:
For example, you might create a `timeseries_template` that is applied to new indices
whose names match the `timeseries-*` index pattern.
* `index.lifecycle.name` specifies the name of the lifecycle policy to apply to all new indices that match
the index pattern.
To enable automatic rollover, the template configures two {ilm-init} settings:
* `index.lifecycle.name` specifies the name of the lifecycle policy to apply to new indices
that match the index pattern.
* `index.lifecycle.rollover_alias` specifies the index alias to be rolled over
when the rollover action is triggered for an index.
You can use the {kib} Create template wizard to add the template.
This wizard invokes the put template API to create the template with the options you specify.
The underlying request looks like this:
[source,console]
-----------------------
PUT _template/timeseries_template
@ -109,6 +131,9 @@ PUT _template/timeseries_template
<3> The name of the alias used to reference these indices.
Required for policies that use the rollover action.
You can also invoke this API directly to add templates.
//////////////////////////
[source,console]
@ -119,7 +144,7 @@ DELETE /_template/timeseries_template
//////////////////////////
[float]
[discrete]
[[ilm-gs-bootstrap]]
=== Bootstrap the initial time-series index
@ -154,11 +179,9 @@ This process repeats each time rollover conditions are met.
You can search across all of the indices managed by the `timeseries_policy` with the `timeseries` alias.
Write operations are routed to the current write index.
For more information about write indices and rollover, see the <<rollover-index-api-desc, rollover API>>.
[float]
[discrete]
[[ilm-gs-check-progress]]
=== Checking progress
=== Check lifecycle progress
To get status information for managed indices, you use the {ilm-init} explain API.
This lets you find out things like:
@ -179,6 +202,7 @@ The response below shows that the bootstrap index is waiting in the `hot` phase'
It remains in this state and {ilm-init} continues to call `attempt-rollover`
until the rollover conditions are met.
// [[36818c6d9f434d387819c30bd9addb14]]
[source,console-result]
--------------------------------------------------
{
@ -219,5 +243,3 @@ until the rollover conditions are met.
<2> The age of the index
<3> The step {ilm-init} is performing on the index
<4> The definition of the current phase (the `hot` phase)
See the <<index-lifecycle-management-api,ILM APIs>> for more information.

13
docs/reference/ilm/ilm-with-existing-indices.asciidoc
View File

@ -77,7 +77,7 @@ DELETE _template/mylogs_template
//////////////////////////
Now that we have these indices, we'll look at a few different ways of migrating
these indices to ILM.
these indices to {ilm-init}.
[[ilm-with-existing-periodic-indices]]
=== Managing existing periodic indices with {ilm-init}
@ -91,12 +91,11 @@ The simplest way to manage existing indices while transitioning to fully
all new documents should be directed to {ilm-init}-managed indices - if you are
using Beats or Logstash data shippers, upgrading all of those shippers to
version 7.0.0 or higher will take care of that part for you. If you are not
using Beats or Logstash, you may need to set up ILM for new indices yourself as
demonstrated in the <<getting-started-index-lifecycle-management,getting started
guide>>.
using Beats or Logstash, you may need to set up {ilm-init} for new indices yourself as
demonstrated in the <<getting-started-index-lifecycle-management,{ilm-init} tutorial>>.
NOTE: If you are using Beats through Logstash, you may need to change your
Logstash output configuration and invoke the Beats setup to use ILM for new
Logstash output configuration and invoke the Beats setup to use {ilm-init} for new
data.
Once all new documents are being written to fully {ilm-init}-managed indices, it
@ -341,7 +340,7 @@ All new documents should be indexed via the `mylogs` alias at this point. Adding
new data to the old indices during the reindexing process can cause data to be
added to the old indices, but not be reindexed into the new indices.
NOTE: If you do not want to mix new data and old data in the new ILM-managed
NOTE: If you do not want to mix new data and old data in the new {ilm-init}-managed
indices, indexing of new data should be paused entirely while the reindex
completes. Mixing old and new data within one index is safe, but keep in mind
that the indices with mixed data should be retained in their entirety until you
@ -364,7 +363,7 @@ PUT _cluster/settings
-----------------------
// TEST[skip:don't want to overwrite this setting for other tests]
<1> This tells ILM to check for rollover conditions every minute
<1> This tells {ilm-init} to check for rollover conditions every minute
We're now ready to reindex our data using the <<docs-reindex,reindex API>>. If
you have a timestamp or date field in your documents, as in this example, it may

38
docs/reference/ilm/index-rollover.asciidoc Normal file
View File

@ -0,0 +1,38 @@
[[index-rollover]]
=== Rollover
When indexing time-series data like logs or metrics, you can't write to a single index indefinitely.
To meet your indexing and search performance requirements and manage resource usage,
you write to an index until some threshold is met and
then create a new index and start writing to it instead.
Using rolling indices enables you to:
* Optimize the active index for high ingest rates on high-performance _hot_ nodes.
* Optimize for search performance on _warm_ nodes.
* Shift older, less frequently accessed data to less expensive _cold_ nodes,
* Delete data according to your retention policies by removing entire indices.
Rollover relies on three things:
* An _index template_ that specifies the settings for each new index in the series.
You optimize this configuration for ingestion, typically using as many shards as you have hot nodes.
* An _index alias_ that references the entire set of indices.
* A single index designated as the _write index_.
This is the active index that handles all write requests.
On each rollover, the new index becomes the write index.
[discrete]
[role="xpack"]
[testenv="basic"]
[[ilm-automatic-rollover]]
=== Automatic rollover
{ilm-init} enables you to automatically roll over to a new index based
on the index size, document count, or age. When a rollover is triggered, a new
index is created, the write alias is updated to point to the new index, and all
subsequent updates are written to the new index.
TIP: Rolling over to a new index based on size, document count, or age is preferable
to time-based rollovers. Rolling over at an arbitrary time often results in
many small indices, which can have a negative impact on performance and
resource usage.

20
docs/reference/ilm/index.asciidoc
View File

@ -1,7 +1,7 @@
[role="xpack"]
[testenv="basic"]
[[index-lifecycle-management]]
= Manage the index lifecycle
= {ilm-init}: Manage the index lifecycle
[partintro]
--
@ -9,8 +9,8 @@ You can configure {ilm} ({ilm-init}) policies to automatically manage indices ac
your performance, resiliency, and retention requirements.
For example, you could use {ilm-init} to:
* Spin up a new index when an index reaches a certain size or number of documents
* Create a new index each day, week, or month and archive previous ones
* Spin up a new index when an index reaches a certain size
* Delete stale indices to enforce data retention standards
[TIP]
@ -18,27 +18,27 @@ To automatically back up your indices and manage snapshots,
use <<getting-started-snapshot-lifecycle-management,snapshot lifecycle policies>>.
* <<overview-index-lifecycle-management>>
* <<ilm-concepts>>
* <<getting-started-index-lifecycle-management>>
* <<ilm-policy-definition>>
* <<ilm-actions>>
* <<set-up-lifecycle-policy>>
* <<update-lifecycle-policy>>
* <<index-lifecycle-error-handling>>
* <<start-stop-ilm>>
* <<using-policies-rollover>>
* <<ilm-configure-rollover>>
* <<ilm-with-existing-indices>>
* <<index-lifecycle-and-snapshots>>
--
include::overview-ilm.asciidoc[]
include::ilm-overview.asciidoc[]
include::getting-started-ilm.asciidoc[]
include::ilm-concepts.asciidoc[]
include::policy-definitions.asciidoc[]
include::ilm-tutorial.asciidoc[]
include::ilm-actions.asciidoc[]
include::set-up-lifecycle-policy.asciidoc[]
include::update-lifecycle-policy.asciidoc[]
include::error-handling.asciidoc[]
include::start-stop-ilm.asciidoc[]

836
docs/reference/ilm/policy-definitions.asciidoc
View File

@ -1,836 +0,0 @@
[role="xpack"]
[testenv="basic"]
[[ilm-policy-definition]]
== Policy phases and actions
There are four stages in the index lifecycle, in the order
they are executed.
[options="header"]
|======
| Name | Description
| `hot` | The index is actively being written to
| `warm` | The index is generally not being written to, but is still queried
| `cold` | The index is no longer being updated and is seldom queried. The
information still needs to be searchable, but it's okay if those queries are
slower.
| `delete` | The index is no longer needed and can safely be deleted
|======
Each of these stages is called a "phase". A policy does not need to configure
each phase for an index. For example, one policy may define only the hot
phase and the delete phase, while another may define all four phases.
=== Timing
Indices enter phases based on a phase's `min_age` parameter.
The index will not enter the phase until the index's age is older than that
of the `min_age`. The parameter is configured using a time
duration format (see <<time-units, Time Units>>).
`min_age` defaults to zero seconds `0s` for each phase if not specified.
[source,console]
--------------------------------------------------
PUT _ilm/policy/my_policy
{
"policy": {
"phases": {
"warm": {
"min_age": "1d",
"actions": {
"allocate": {
"number_of_replicas": 1
}
}
},
"delete": {
"min_age": "30d",
"actions": {
"delete": {}
}
}
}
}
}
--------------------------------------------------
The above example configures a policy that moves the index into the warm
phase after one day. Until then, the index is in a waiting state. After
moving into the warm phase, it will wait until 30 days have elapsed before
moving to the delete phase and deleting the index.
`min_age` is usually the time elapsed from the time the index is created, unless
the `index.lifecycle.origination_date` index setting is configured, in which
case the `min_age` will be the time elapsed since that specified date. If the
index is rolled over, then `min_age` is the time elapsed from the time the
index is rolled over. The intention here is to execute following phases and
actions relative to when data was written last to a rolled over index.
The previous phase's actions must complete before {ilm} will check `min_age` and
transition into the next phase. By default, {ilm} checks for indices that meet
policy criteria, like `min_age`, every 10 minutes. You can use the
`indices.lifecycle.poll_interval` cluster setting to control how often this
check occurs.
=== Phase Execution
The current phase definition, of an index's policy being executed, is stored
in the index's metadata. This phase definition is cached to prevent changes to
the policy from putting the index in a state where it cannot proceed from its
current step. When the policy is updated we check to see if this phase
definition can be safely updated, and if so, update the cached definition in
indices using the updated policy. The phase and its actions are compiled into a
series of discrete steps that are executed sequentially. Since some {ilm-init}
actions are more complex and involve multiple operations against an index, each
of these operations are done in isolation in a unit called a "step". The
<<ilm-explain-lifecycle,Explain Lifecycle API>> exposes this information to us
to see which step our index is either to execute next, or is currently
executing.
=== Actions
The below list shows the actions which are available in each phase.
NOTE: The order that configured actions are performed in within each phase is
determined automatically by {ilm-init}, and cannot be changed by changing the
policy definition.
* Hot
- <<ilm-set-priority-action,Set Priority>>
- <<ilm-unfollow-action,Unfollow>>
- <<ilm-rollover-action,Rollover>>
* Warm
- <<ilm-set-priority-action,Set Priority>>
- <<ilm-unfollow-action,Unfollow>>
- <<ilm-readonly-action,Read-Only>>
- <<ilm-allocate-action,Allocate>>
- <<ilm-shrink-action,Shrink>>
- <<ilm-forcemerge-action,Force Merge>>
* Cold
- <<ilm-set-priority-action,Set Priority>>
- <<ilm-unfollow-action,Unfollow>>
- <<ilm-allocate-action,Allocate>>
- <<ilm-freeze-action,Freeze>>
- <<ilm-searchable-snapshot-action, Searchable Snapshot>>
* Delete
- <<ilm-wait-for-snapshot-action,Wait For Snapshot>>
- <<ilm-delete-action,Delete>>
[[ilm-allocate-action]]
==== Allocate
Phases allowed: warm, cold.
The Allocate action allows you to specify which nodes are allowed to host the
shards of the index and set the number of replicas.
Behind the scenes, it is modifying the index settings
for shard filtering and/or replica counts. When updating the number of replicas,
configuring allocation rules is optional. When configuring allocation rules,
setting number of replicas is optional. Although this action can be treated as
two separate index settings updates, both can be configured at once.
For more information about how {es} uses replicas for scaling, see
<<scalability>>. See <<shard-allocation-filtering>> for more information about
controlling where Elasticsearch allocates shards of a particular index.
[[ilm-allocate-options]]
.Allocate Options
[options="header"]
|======
| Name | Required | Default | Description
| `number_of_replicas` | no | - | The number of replicas to
assign to the index
| `include` | no | - | assigns an index to nodes
having at least _one_ of the attributes
| `exclude` | no | - | assigns an index to nodes having
_none_ of the attributes
| `require` | no | - | assigns an index to nodes having
_all_ of the attributes
|======
If `number_of_replicas` is not configured, then at least one of `include`,
`exclude`, and `require` is required. An empty Allocate Action with no configuration
is invalid.
===== Example: Change number of replicas
In this example, the index's number of replicas is changed to `2`, while allocation
rules are unchanged.
[source,console]
--------------------------------------------------
PUT _ilm/policy/my_policy
{
"policy": {
"phases": {
"warm": {
"actions": {
"allocate" : {
"number_of_replicas" : 2
}
}
}
}
}
}
--------------------------------------------------
===== Example: Assign index to node with specific "box_type" attribute
This example assigns the index to nodes with `box_type` attribute of "hot" or "warm".
[source,console]
--------------------------------------------------
PUT _ilm/policy/my_policy
{
"policy": {
"phases": {
"warm": {
"actions": {
"allocate" : {
"include" : {
"box_type": "hot,warm"
}
}
}
}
}
}
}
--------------------------------------------------
===== Example: Assign index to a specific node and update replica settings
This example updates the index to have one replica per shard and be allocated
to nodes with a `box_type` attribute of "cold".
[source,console]
--------------------------------------------------
PUT _ilm/policy/my_policy
{
"policy": {
"phases": {
"warm": {
"actions": {
"allocate" : {
"number_of_replicas": 1,
"require" : {
"box_type": "cold"
}
}
}
}
}
}
}
--------------------------------------------------
[[ilm-wait-for-snapshot-action]]
==== Wait For Snapshot
Phases allowed: delete.
The Wait For Snapshot Action waits for defined SLM policy to be executed to ensure that snapshot of index exists before
deletion.
[[ilm-wait-for-snapshot-options]]
.Wait For Snapshot
[options="header"]
|======
| Name | Required | Default | Description
| `policy` | yes | - | SLM policy name that this action should wait for
|======
[source,console]
--------------------------------------------------
PUT _ilm/policy/my_policy
{
"policy": {
"phases": {
"delete": {
"actions": {
"wait_for_snapshot" : {
"policy": "slm-policy-name"
}
}
}
}
}
}
--------------------------------------------------
[[ilm-delete-action]]
==== Delete
Phases allowed: delete.
The Delete Action does just that, it deletes the index.
[[ilm-delete-action-options]]
.Delete
[options="header"]
|======
| Name | Required | Default | Description
| `delete_searchable_snapshot` | no | true | Deletes the searchable snapshot created in the cold phase, if a
snapshot was created (ie. if the <<ilm-searchable-snapshot-action,
Searchable Snapshot Action>> was used in the cold phase)
|======
[source,console]
--------------------------------------------------
PUT _ilm/policy/my_policy
{
"policy": {
"phases": {
"delete": {
"actions": {
"delete" : { }
}
}
}
}
}
--------------------------------------------------
[[ilm-forcemerge-action]]
==== Force Merge
Phases allowed: hot, warm.
NOTE: Index will be be made read-only when this action is run
(see: <<dynamic-index-settings,index.blocks.write>>)
NOTE: The `forcemerge` action is best effort. It might happen that some of the
shards are relocating, in which case they will not be merged.
NOTE: If the `forcemerge` action is used in the `hot` phase, the `rollover` action *must* be present.
ILM validates this predicate and will refuse a policy with a forcemerge in the hot phase without a
rollover action.
The Force Merge Action <<indices-forcemerge,force merges>> the index into at
most a specific number of <<indices-segments,segments>>.
[[ilm-forcemerge-options]]
.Force Merge Options
[options="header"]
|======
| Name | Required | Default | Description
| `max_num_segments` | yes | - | The number of segments to merge to. To fully merge the index, set it to `1`
| `codec` | no | - | Optional specification of the `best_compression` codec
|======
[WARNING]
======
When using the `"codec": "best_compression"` configuration in the ILM forcemerge action, ILM will
<<indices-close,close>> and then <<indices-open-close,re-open>> the index prior to a forcemerge.
During this time the index will be unavailable for either read or write operations.
======
[source,console]
--------------------------------------------------
PUT _ilm/policy/my_policy
{
"policy": {
"phases": {
"warm": {
"actions": {
"forcemerge" : {
"max_num_segments": 1
}
}
}
}
}
}
--------------------------------------------------
[[ilm-freeze-action]]
==== Freeze
Phases allowed: cold.
This action will <<frozen-indices, freeze>> the index
by calling the <<freeze-index-api, Freeze Index API>>.
[source,console]
--------------------------------------------------
PUT _ilm/policy/my_policy
{
"policy": {
"phases": {
"cold": {
"actions": {
"freeze" : { }
}
}
}
}
}
--------------------------------------------------
[IMPORTANT]
================================
Freezing an index will close the index and reopen it within the same API call.
This causes primaries to not be allocated for a short amount of time and
causes the cluster to go red until the primaries are allocated again.
This limitation might be removed in the future.
================================
[[ilm-readonly-action]]
==== Read-Only
Phases allowed: warm.
This action will set the index to be read-only
(see: <<dynamic-index-settings,index.blocks.write>>)
This action does not have any options associated with it.
[source,console]
--------------------------------------------------
PUT _ilm/policy/my_policy
{
"policy": {
"phases": {
"warm": {
"actions": {
"readonly" : { }
}
}
}
}
}
--------------------------------------------------
[[ilm-rollover-action]]
==== Rollover
Phases allowed: hot.
[WARNING]
index format must match pattern '^.*-\\d+$', for example (`logs-000001`).
[WARNING]
The managed index must set `index.lifecycle.rollover_alias` as the
alias to rollover. The index must also be the write index for the alias.
[IMPORTANT]
If a policy using the Rollover action is used on a <<ccr-put-follow,follower
index>>, policy execution will wait until the leader index rolls over (or has
<<skipping-rollover, otherwise been marked as complete>>), then convert the
follower index into a regular index as if <<ilm-unfollow-action,the Unfollow
action>> had been used instead of rolling over.
For example, if an index to be managed has an alias `my_data`. The managed
index "my_index-000001" must be the write index for the alias. For more information, read
<<indices-rollover-is-write-index,Write Index Alias Behavior>>.
[source,console]
--------------------------------------------------
PUT my_index-000001
{
"settings": {
"index.lifecycle.name": "my_policy",
"index.lifecycle.rollover_alias": "my_data"
},
"aliases": {
"my_data": {
"is_write_index": true
}
}
}
--------------------------------------------------
The Rollover Action rolls an alias over to a new index when the
existing index meets one of the rollover conditions.
[[ilm-rollover-options]]
.Rollover Options
[options="header"]
|======
| Name | Required | Default | Description
| `max_size` | no | - | max primary shard index storage size.
See <<byte-units, Byte Units>>
for formatting
| `max_docs` | no | - | max number of documents an
index is to contain before
rolling over.
| `max_age` | no | - | max time elapsed from index
creation. See
<<time-units, Time Units>>
for formatting
|======
At least one of `max_size`, `max_docs`, `max_age` or any combinations of the
three are required to be specified.
===== Example: Rollover when index is too large
This example rolls the index over when it is at least 100 gigabytes.
[source,console]
--------------------------------------------------
PUT _ilm/policy/my_policy
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover" : {
"max_size": "100GB"
}
}
}
}
}
}
--------------------------------------------------
===== Example: Rollover when index has too many documents
This example rolls the index over when it contains at least
100000000 documents.
[source,console]
--------------------------------------------------
PUT _ilm/policy/my_policy
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover" : {
"max_docs": 100000000
}
}
}
}
}
}
--------------------------------------------------
===== Example: Rollover when index is too old
This example rolls the index over when it has been created at least
7 days ago.
[source,console]
--------------------------------------------------
PUT _ilm/policy/my_policy
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover" : {
"max_age": "7d"
}
}
}
}
}
}
--------------------------------------------------
===== Example: Rollover when index is too old or too large
This example rolls the index over when it has been created at least
7 days ago or it is at least 100 gigabytes. In this case, the index will be
rolled over when any of the conditions is met.
[source,console]
--------------------------------------------------
PUT _ilm/policy/my_policy
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover" : {
"max_age": "7d",
"max_size": "100GB"
}
}
}
}
}
}
--------------------------------------------------
===== Example: Rollover condition stalls phase transition
The Rollover action will only complete once one of its conditions is
met. This means that any proceeding phases will be blocked until Rollover
succeeds.
[source,console]
--------------------------------------------------
PUT /_ilm/policy/rollover_policy
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": {
"max_size": "50G"
}
}
},
"delete": {
"min_age": "1d",
"actions": {
"delete": {}
}
}
}
}
}
--------------------------------------------------
The above example illustrates a policy which attempts to delete an
index one day after the index has been rolled over. It does not
delete the index one day after it has been created.
[[ilm-searchable-snapshot-action]]
==== Searchable Snapshot
Phases allowed: cold.
This action takes a snapshot of the managed index in the configured repository
and mounts it as a searchable snapshot.
[[ilm-searchable-snapshot-options]]
.Searchable Snapshot Options
[options="header"]
|======
| Name | Required | Default | Description
| `snapshot_repository` | yes | - | Repository used to store the snapshot created by this action. The snapshot will be,
by default, deleted by the <<ilm-delete-action, Delete Action>> in the delete phase, if
configured, but the user can configure the <<ilm-delete-action, Delete Action>> to keep
the snapshot using the `delete_searchable_snapshot` option.
|======
[source,console]
--------------------------------------------------
PUT _ilm/policy/my_policy
{
"policy": {
"phases": {
"cold": {
"actions": {
"searchable_snapshot" : {
"snapshot_repository" : "backing_repo"
}
}
}
}
}
}
--------------------------------------------------
[[ilm-set-priority-action]]
==== Set Priority
Phases allowed: hot, warm, cold.
This action sets the <<recovery-prioritization, index priority>> on the index as
soon as the policy enters the hot, warm, or cold phase. Indices with a higher
priority will be recovered before indices with lower priorities following a node
restart. Generally, indexes in the hot phase should have the highest value and
indexes in the cold phase should have the lowest values. For example:
100 for the hot phase, 50 for the warm phase, and 0 for the cold phase.
Indicies that don't set this value have an implicit default priority of 1.
[[ilm-set-priority-options]]
.Set Priority Options
[options="header"]
|======
| Name | Required | Default | Description
| `priority` | yes | - | The priority for the index. Must be 0 or greater.
The value may also be set to null to remove the priority.
|======
[source,console]
--------------------------------------------------
PUT _ilm/policy/my_policy
{
"policy": {
"phases": {
"warm": {
"actions": {
"set_priority" : {
"priority": 50
}
}
}
}
}
}
--------------------------------------------------
[[ilm-shrink-action]]
==== Shrink
NOTE: Index will be be made read-only when this action is run
(see: <<dynamic-index-settings,index.blocks.write>>)
[IMPORTANT]
If a policy using the Shrink action is used on a <<ccr-put-follow,follower
index>>, policy execution will wait until the leader index rolls over (or has
<<skipping-rollover, otherwise been marked as complete>>), then convert the
follower index into a regular index as if <<ilm-unfollow-action,the Unfollow
action>> had been used before shrink is applied, as shrink cannot be safely
applied to follower indices.
This action shrinks an existing index into a new index with fewer primary
shards. It calls the <<indices-shrink-index,Shrink API>> to shrink the index.
Since allocating all the primary shards of the index to one node is a
prerequisite, this action will first allocate the primary shards to a valid
node. After shrinking, it will swap aliases pointing to the original index
into the new shrunken index. The new index will also have a new name:
"shrink-<origin-index-name>". So if the original index was called "logs",
then the new index will be named "shrink-logs".
[[ilm-shrink-options]]
.Shrink Options
[options="header"]
|======
| Name | Required | Default | Description
| `number_of_shards` | yes | - | The number of shards
to shrink to. must be
a factor of the number
of shards in the
source index.
|======
[source,console]
--------------------------------------------------
PUT _ilm/policy/my_policy
{
"policy": {
"phases": {
"warm": {
"actions": {
"shrink" : {
"number_of_shards": 1
}
}
}
}
}
}
--------------------------------------------------
[[ilm-unfollow-action]]
==== Unfollow
[IMPORTANT]
This action may be used explicitly, as shown below, but this action is also run
before <<ilm-rollover-action,the Rollover action>> and <<ilm-shrink-action,the
Shrink action>> as described in the documentation for those actions. This is
expected and safe for non-CCR indices to run, as the steps are skipped when CCR
is not in use.
This action turns a {ref}/ccr-apis.html[ccr] follower index
into a regular index. This can be desired when moving follower
indices into the next phase. Also certain actions like shrink
and rollover can then be performed safely on follower indices.
This action will wait until is it safe to convert a follower index into a
regular index. In particular, the following conditions must be met:
* The leader index must have `index.lifecycle.indexing_complete` set to `true`.
This happens automatically if the leader index is rolled over using
<<ilm-rollover-action,the Rollover action>>, or may be set manually using
the <<indices-update-settings,Index Settings API>>.
* All operations performed on the leader index must have been replicated to the
follower index. This ensures that no operations will be lost when the index is
converted into a regular index.
If the unfollow action encounters a follower index then
the following operations will be performed on it:
* Pauses indexing following for the follower index.
* Closes the follower index.
* Unfollows the follower index.
* Opens the follower index (which is at this point is a regular index).
The unfollow action does not have any options and
if it encounters a non follower index, then the
unfollow action leaves that index untouched and
lets the next action operate on this index.
[source,console]
--------------------------------------------------
PUT _ilm/policy/my_policy
{
"policy": {
"phases": {
"hot": {
"actions": {
"unfollow" : {}
}
}
}
}
}
--------------------------------------------------
=== Full Policy
With all of these actions, we can support complex management strategies for our
indices. This policy will define an index that will start in the hot phase,
rolling over every 50 GB or 7 days. After 30 days it enters the warm phase
and increases the replicas to 2, force merges and shrinks. After 60 days
it enters the cold phase and allocates to "cold" nodes, and after 90 days the
index is deleted.
[source,console]
--------------------------------------------------
PUT _ilm/policy/full_policy
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": {
"max_age": "7d",
"max_size": "50G"
}
}
},
"warm": {
"min_age": "30d",
"actions": {
"forcemerge": {
"max_num_segments": 1
},
"shrink": {
"number_of_shards": 1
},
"allocate": {
"number_of_replicas": 2
}
}
},
"cold": {
"min_age": "60d",
"actions": {
"allocate": {
"require": {
"type": "cold"
}
}
}
},
"delete": {
"min_age": "90d",
"actions": {
"delete": {}
}
}
}
}
}
--------------------------------------------------

104
docs/reference/ilm/set-up-lifecycle-policy.asciidoc
View File

@ -1,12 +1,26 @@
[role="xpack"]
[testenv="basic"]
[[set-up-lifecycle-policy]]
== Create lifecycle policy
== Configure lifecycle policy [[ilm-policy-definition]]
In order for an index to use an {ilm} policy to manage its lifecycle we must
first define a lifecycle policy for it to use. The following request creates a
policy called `my_policy` in Elasticsearch which we can later use to manage our
indexes.
For {ilm-init} to manage an index, a valid policy
must be specified in the `index.lifecycle.name` index setting.
To configure a lifecycle policy for rolling indices,
you create the policy and add it to the index template.
To use a policy to manage an index that doesn't roll over,
you can specify the policy directly when you create it.
[discrete]
[[ilm-create-policy]]
=== Create lifecycle policy
You use the <<ilm-put-lifecycle,create policy API>> to define a new lifecycle policy.
For example, the following request creates `my_policy`, a
policy that defines a hot and and delete phase.
When the index reaches 25GB, it rolls over directly to the delete phase.
The index is deleted 30 days after rollover.
[source,console]
------------------------
@ -32,28 +46,24 @@ PUT _ilm/policy/my_policy
}
------------------------
<1> Rollover the index when it reaches 25GB in size
<2> Delete the index when its 30 days old
<1> Roll over the index when it reaches 25GB in size
<2> Delete the index 30 days after rollover
{ilm} will manage an index using the policy defined in the
`index.lifecycle.name` index setting. If this setting does not exist in the
settings for a particular index, {ilm} will not manage that index.
NOTE: {ilm-init} policies are stored in the global cluster state and can be included in snapshots by
setting `include_global_state` to `true` when you <<snapshots-take-snapshot, take the snapshot>>.
Restoring {ilm-init} policies from a snapshot is all-or-nothing.
The entire global state, including all policies, is overwritten when you restore the snapshot.
To set the policy for an index there are two options:
[discrete]
[[apply-policy-template]]
=== Apply lifecycle policy with an index template
1. Apply the policy to an index template and bootstrap creating the first index
2. Apply the policy to a new index in a create index request
NOTE: ILM policies are stored in global cluster state which can be backed up using Snapshot with the
`include_global_state` parameter set to `true`.
Restoring ILM policies from global state in a Snapshot is all-or-nothing that will overwrite the entire global
state using the snapshot's point in time copy including all ILM policies.
[[applying-policy-to-template]]
=== Applying a policy to an index template
The `index.lifecycle.name` setting can be set in an index template so that it
is automatically applied to indexes matching the templates index pattern:
To use a policy that triggers the rollover action,
you need to configure the policy in the index template used to create each new index.
In addition to specifying the name of the policy in the `index.lifecycle.name` setting,
you specify a `index.lifecycle.rollover_alias` for referencing
the indices managed by this policy.
[source,console]
-----------------------
@ -64,14 +74,14 @@ PUT _template/my_template
"number_of_shards": 1,
"number_of_replicas": 1,
"index.lifecycle.name": "my_policy", <2>
"index.lifecycle.rollover_alias": "test-alias"
"index.lifecycle.rollover_alias": "test-alias" <3>
}
}
-----------------------
<1> This template will be applied to all indexes which have a name starting
with `test-`
<2> The template will set the policy to be used to `my_policy`
<1> Use this template for all new indices whose names begin with `test-`
<2> Apply `my_policy` to new indices created with this template
<3> Define an index alias for referencing indices managed by `my_policy`
//////////////////////////
@ -83,8 +93,18 @@ DELETE /_template/my_template
//////////////////////////
Now that a policy exists and is used in an index template we can create an
initial index which will be managed by our policy:
[discrete]
[[create-initial-index]]
==== Create an initial managed index
You need to manually create the first index managed by a policy that uses the rollover action
and designate it as the write index.
The name of the index must match the pattern defined in the index template and end with a number.
This number is incremented to generate the name of indices created by the rollover action.
For example, the following request creates the `test-00001` index.
Because it matches the index pattern specified in `my_template`,
{es} automatically applies the settings from that template.
[source,console]
-----------------------
@ -100,14 +120,18 @@ PUT test-000001
<1> Set this initial index to be the write index for this alias.
We can now write data to the `test-alias` alias. Because we have a rollover
action defined in our policy, when the index grows larger than 25GB {ilm} will
create a new index and roll the alias over to use the new index automatically.
Now you can start indexing data to the rollover alias specified in the lifecycle policy.
With the sample `my_policy` policy, the rollover action is triggered once the initial
index exceeds 25GB.
{ilm-init} then creates a new index that becomes the write index for the `test-alias`.
=== Apply a policy to a create index request
[discrete]
[[apply-policy-manually]]
=== Apply lifecycle policy manually
The `index.lifecycle.name` setting can be set on an individual create index
request so {ilm} immediately starts managing the index:
When you create an index directly, you can apply a lifecycle policy
by specifying the `index.lifecycle.name` setting.
This causes {ilm-init} to immediately start managing the index.
[source,console]
-----------------------
@ -121,8 +145,6 @@ PUT test-index
}
-----------------------
IMPORTANT: Its recommended not to use the create index API with a policy that
defines a rollover action. If you do so, the new index as the result of the
rollover will not carry forward the policy. Always use index templates to
define policies with rollover actions.
IMPORTANT: Do not manually apply a policy that uses the rollover action.
Policies that use rollover must be applied by the <<apply-policy-template, index template>>.
Otherwise, the policy is not carried forward when the rollover action creates a new index.

38
docs/reference/ilm/start-stop-ilm.asciidoc
View File

@ -3,20 +3,20 @@
[[start-stop-ilm]]
== Start and stop {ilm}
All indices that are managed by ILM will continue to execute
All indices that are managed by {ilm-init} will continue to execute
their policies. There may be times when this is not desired on certain
indices, or maybe even all the indices in a cluster. For example,
maybe there are scheduled maintenance windows when cluster topology
changes are desired that may impact running ILM actions. For this reason,
ILM has two ways to disable operations.
changes are desired that may impact running {ilm-init} actions. For this reason,
{ilm-init} has two ways to disable operations.
When stopping ILM, snapshot lifecycle management operations are also stopped,
When stopping {ilm-init}, snapshot lifecycle management operations are also stopped,
this means that no scheduled snapshots are created (currently ongoing snapshots
are unaffected).
Normally, ILM will be running by default.
To see the current operating status of ILM, use the <<ilm-get-status,Get Status API>>
to see the current state of ILM.
Normally, {ilm-init} will be running by default.
To see the current operating status of {ilm-init}, use the <<ilm-get-status,Get Status API>>
to see the current state of {ilm-init}.
////
[source,console]
@ -66,22 +66,22 @@ If the request does not encounter errors, you receive the following result:
}
--------------------------------------------------
The operating modes of ILM:
The operating modes of {ilm-init}:
[[ilm-operating-modes]]
.ILM Operating Modes
.{ilm-init} Operating Modes
[options="header"]
|===
|Name |Description
|RUNNING |Normal operation where all policies are executed as normal
|STOPPING|ILM has received a request to stop but is still processing some policies
|STOPPING|{ilm-init} has received a request to stop but is still processing some policies
|STOPPED |This represents a state where no policies are executed
|===
[float]
=== Stopping ILM
[discrete]
=== Stopping {ilm-init}
The ILM service can be paused such that no further steps will be executed
The {ilm-init} service can be paused such that no further steps will be executed
using the <<ilm-stop,Stop API>>.
[source,console]
@ -109,8 +109,8 @@ GET _ilm/status
--------------------------------------------------
// TESTRESPONSE[s/"STOPPING"/$body.operation_mode/]
The ILM service will then, asynchronously, run all policies to a point
where it is safe to stop. After ILM verifies that it is safe, it will
The {ilm-init} service will then, asynchronously, run all policies to a point
where it is safe to stop. After {ilm-init} verifies that it is safe, it will
move to the `STOPPED` mode.
////
@ -131,10 +131,10 @@ GET _ilm/status
--------------------------------------------------
// TESTRESPONSE[s/"STOPPED"/$body.operation_mode/]
[float]
=== Starting ILM
[discrete]
=== Starting {ilm-init}
To start ILM and continue executing policies, use the <<ilm-start, Start API>>.
To start {ilm-init} and continue executing policies, use the <<ilm-start, Start API>>.
[source,console]
@ -151,7 +151,7 @@ GET _ilm/status
// TEST[continued]
////
The Start API will send a request to the ILM service to immediately begin
The Start API will send a request to the {ilm-init} service to immediately begin
normal operations.
[source,console-result]

519
docs/reference/ilm/update-lifecycle-policy.asciidoc
View File

@ -1,506 +1,41 @@
[role="xpack"]
[testenv="basic"]
[[update-lifecycle-policy]]
== Update lifecycle policy
=== Lifecycle policy updates
++++
<titleabbrev>Update policy</titleabbrev>
<titleabbrev>Policy updates</titleabbrev>
++++
You can update an existing lifecycle policy to fix mistakes or change
strategies for newly created indices. It is possible to update policy definitions
and an index's `index.lifecycle.name` settings independently. To prevent the situation
that phase definitions are modified while currently being executed on an index, each index
will keep the version of the current phase definition it began execution with until it completes.
This also means that changes to `min_age` will not be propagated. If a new policy is set that
introduces a later `min_age` for the currently executing phase, that new `min_age` will not
be picked up by the update.
You can change how the lifecycle of an index or collection of rolling indices is managed
by modifying the current policy or switching to a different policy.
There are three scenarios for examining the behavior updating policies and
their effects on policy execution on indices.
To ensure that policy updates don't put an index into a state where it can't exit the current phase,
the phase definition is cached in the index metadata when it enters the phase.
This cached definition is used to complete the phase.
=== Updates to policies not managing indices
When the index advances to the next phase, it uses the phase definition from the updated policy.
Indices not referencing an existing policy that is updated will not be affected.
If an index is assigned to the policy, it will be assigned the latest version of that policy
[discrete]
[[ilm-apply-changes]]
=== How changes are applied
To show this, let's create a policy `my_policy`.
When a policy is initially applied to an index, the index gets the latest version of the policy.
If you update the policy, the policy version is bumped and {ilm-init} can detect that the index
is using an earlier version that needs to be updated.
[source,console]
------------------------
PUT _ilm/policy/my_policy
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": {
"max_size": "25GB"
}
}
},
"delete": {
"min_age": "30d",
"actions": {
"delete": {}
}
}
}
}
}
------------------------
Changes to `min_age` are not propagated to the cached definition.
Changing a phase's `min_age` does not affect indices that are currently executing that phase.
This newly defined policy will be created and assigned to have a version equal
to 1. Since we haven't assigned any indices to this policy, any updates that
occur will be reflected completely on indices that are newly set to be managed
by this policy.
For example, if you create a policy that has a hot phase that does not specify a `min_age`,
indices immediately enter the hot phase when the policy is applied.
If you then update the policy to specify a `min_age` of 1 day for the hot phase,
that has no effect on indices that are already in the hot phase.
Indices created _after_ the policy update won't enter the hot phase until they are a day old.
Updating the Delete phase's minimum age can be done in an update request.
[discrete]
[[ilm-apply-new-policy]]
=== How new policies are applied
[source,console]
------------------------
PUT _ilm/policy/my_policy
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": {
"max_size": "25GB"
}
}
},
"delete": {
"min_age": "10d", <1>
"actions": {
"delete": {}
}
}
}
}
}
------------------------
// TEST[continued]
<1> update `min_age` to 10 days
//////////
[source,console]
--------------------------------------------------
GET _ilm/policy/my_policy
--------------------------------------------------
// TEST[continued]
//////////
When we get the policy, we will see it reflect our latest changes, but
with its version bumped to 2.
[source,console-result]
--------------------------------------------------
{
"my_policy": {
"version": 2, <1>
"modified_date": 82392349, <2>
"policy": {
"phases": {
"hot": {
"min_age": "0ms",
"actions": {
"rollover": {
"max_size": "25gb"
}
}
},
"delete": {
"min_age": "10d",
"actions": {
"delete": {
"delete_searchable_snapshot": true
}
}
}
}
}
}
}
--------------------------------------------------
// TESTRESPONSE[s/"modified_date": 82392349/"modified_date": $body.my_policy.modified_date/]
<1> The updated version value
<2> The timestamp when this policy was updated last.
Afterwords, any indices set to `my_policy` will execute against version 2 of
the policy.
=== Updates to executing policies
Indices preserve the phase definition from the latest policy version that existed
at the time that it entered that phase. Changes to the currently-executing phase within policy updates will
not be reflected during execution. This means that updates to the `hot` phase, for example, will not affect
indices that are currently executing the corresponding `hot` phase.
Let's say we have an index `my_index` managed by the below `my_executing_policy` definition.
[source,console]
------------------------
PUT _ilm/policy/my_executing_policy
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": {
"max_docs": 1
}
}
},
"delete": {
"min_age": "10d",
"actions": {
"delete": {}
}
}
}
}
}
------------------------
////
[source,console]
------------------------
PUT my_index
{
"settings": {
"index.lifecycle.name": "my_executing_policy"
}
}
------------------------
// TEST[continued]
////
The <<ilm-explain-lifecycle,Explain API>> is useful to introspect managed indices to see which phase definition they are currently executing.
Using this API, we can find out that `my_index` is currently checking if it is ready to be rolled over.
[source,console]
--------------------------------------------------
GET my_index/_ilm/explain
--------------------------------------------------
// TEST[continued]
[source,console-result]
--------------------------------------------------
{
"indices": {
"my_index": {
"index": "my_index",
"managed": true,
"policy": "my_executing_policy",
"lifecycle_date_millis": 1538475653281,
"age": "30s",
"phase": "hot",
"phase_time_millis": 1538475653317,
"action": "rollover",
"action_time_millis": 1538475653317,
"step": "check-rollover-ready",
"step_time_millis": 1538475653317,
"phase_execution": {
"policy": "my_executing_policy",
"modified_date_in_millis": 1538475653317,
"version": 1,
"phase_definition": {
"min_age": "0ms",
"actions": {
"rollover": {
"max_docs": 1
}
}
}
}
}
}
}
--------------------------------------------------
// TESTRESPONSE[skip:no way to know if we will get this response immediately]
We can update `my_executing_policy` to enter the hot phase after one day.
[source,console]
------------------------
PUT _ilm/policy/my_executing_policy
{
"policy": {
"phases": {
"hot": {
"min_age": "1d", <1>
"actions": {
"rollover": {
"max_docs": 1
}
}
},
"delete": {
"min_age": "10d",
"actions": {
"delete": {}
}
}
}
}
}
------------------------
// TEST[continued]
<1> updated `min_age` from "0ms" to "1d"
The index `my_index` has already entered the hot phase, so it will still
use version 1 of the policy until it completes the hot phase.
////
[source,console]
--------------------------------------------------
GET my_index/_ilm/explain
--------------------------------------------------
// TEST[continued]
////
[source,console-result]
--------------------------------------------------
{
"indices": {
"my_index": {
"index": "my_index",
"managed": true,
"policy": "my_executing_policy",
"lifecycle_date_millis": 1538475653281,
"age": "30s",
"phase": "hot",
"phase_time_millis": 1538475653317,
"action": "rollover",
"action_time_millis": 1538475653317,
"step": "check-rollover-ready",
"step_time_millis": 1538475653317,
"phase_execution": {
"policy": "my_executing_policy",
"modified_date_in_millis": 1538475653317,
"version": 1, <1>
"phase_definition": {
"min_age": "0ms",
"actions": {
"rollover": {
"max_docs": 1
}
}
}
}
}
}
}
--------------------------------------------------
// TESTRESPONSE[skip:no way to know if we will get this response immediately]
<1> the version of the policy used for executing the hot phase
We can also update `my_executing_policy` to have no rollover action and,
instead, go directly into a newly introduced `warm` phase.
[source,console]
------------------------
PUT _ilm/policy/my_executing_policy
{
"policy": {
"phases": {
"warm": {
"min_age": "1d",
"actions": {
"forcemerge": {
"max_num_segments": 1
}
}
},
"delete": {
"min_age": "10d",
"actions": {
"delete": {}
}
}
}
}
}
------------------------
// TEST[continued]
Now, version 3 of this policy has no `hot` phase, but if we run the
Explain API again, we will see that nothing has changed. The index
`my_index` is still executing version 1 of the policy.
////
[source,console]
--------------------------------------------------
GET my_index/_ilm/explain
--------------------------------------------------
// TEST[continued]
////
[source,console-result]
--------------------------------------------------
{
"indices": {
"my_index": {
"index": "my_index",
"managed": true,
"policy": "my_executing_policy",
"lifecycle_date_millis": 1538475653281,
"age": "30s",
"phase": "hot",
"phase_time_millis": 1538475653317,
"action": "rollover",
"action_time_millis": 1538475653317,
"step": "check-rollover-ready",
"step_time_millis": 1538475653317,
"phase_execution": {
"policy": "my_executing_policy",
"modified_date_in_millis": 1538475653317,
"version": 1, <1>
"phase_definition": {
"min_age": "0ms",
"actions": {
"rollover": {
"max_docs": 1
}
}
}
}
}
}
}
--------------------------------------------------
// TESTRESPONSE[skip:no way to know if we will get this response immediately]
<1> the version of the policy used for executing the hot phase
After indexing one document into `my_index` so that rollover succeeds and
moves onto the next phase, we will notice something new. The index will
move into the next phase in the updated version 3 of its policy.
////
[source,console]
--------------------------------------------------
PUT my_index/_doc/1
{
"foo": "bar"
}
GET my_index/_ilm/explain
--------------------------------------------------
// TEST[continued]
////
[source,console-result]
--------------------------------------------------
{
"indices": {
"my_index": {
"index": "my_index",
"managed": true,
"policy": "my_executing_policy",
"lifecycle_date_millis": 1538475653281,
"age": "30s",
"phase": "warm",
"phase_time_millis": 1538475653317,
"action": "forcemerge",
"action_time_millis": 1538475653317,
"step": "forcemerge",
"step_time_millis": 1538475653317,
"phase_execution": {
"policy": "my_executing_policy",
"modified_date_in_millis": 1538475653317,
"version": 3, <1>
"phase_definition": {
"min_age": "1d",
"actions": {
"forcemerge": {
"max_num_segments": 1
}
}
}
}
}
}
}
--------------------------------------------------
// TESTRESPONSE[skip:There is no way to force the index to move to the next step in a timely manner]
<1> The index has moved to using version 3 of the policy
`my_index` will move to the next phase in the latest policy definition, which is the newly added `warm` phase.
=== Switching policies for an index
Setting `index.lifecycle.name` to a different policy behaves much like a policy update, but instead of just
switching to a different version, it switches to a different policy.
After setting a policy for an index, we can switch out `my_policy` with
`my_other_policy` by just updating the index's `index.lifecycle.name`
setting to the new policy. After completing its currently executed phase,
it will move on to the next phase in `my_other_policy`. So if it was on the
`hot` phase before, it will move to the `delete` phase after the `hot` phase concluded.
////
[source,console]
------------------------
PUT _ilm/policy/my_policy
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": {
"max_size": "25GB"
}
}
},
"delete": {
"min_age": "10d",
"actions": {
"delete": {}
}
}
}
}
}
PUT _ilm/policy/my_other_policy
{
"policy": {
"phases": {
"delete": {
"min_age": "1d",
"actions": {
"delete": {}
}
}
}
}
}
PUT my_index
{
"settings": {
"index.lifecycle.name": "my_policy"
}
}
------------------------
////
[source,console]
--------------------------------------------------
PUT my_index/_settings
{
"lifecycle.name": "my_other_policy"
}
--------------------------------------------------
// TEST[continued]
The change to the new policy will not happen immediately. The currently executing phase
of the existing policy for `my_index` will continue to execute until it completes. Once
completed, `my_index` will move to being managed by the `my_other_policy`.
When you apply a different policy to a managed index,
the index completes the current phase using the cached definition from the previous policy.
The index starts using the new policy when it moves to the next phase.

18
docs/reference/ilm/using-policies-rollover.asciidoc
View File

@ -1,18 +1,8 @@
[role="xpack"]
[testenv="basic"]
[[ilm-configure-rollover]]
== Configure rollover
[[using-policies-rollover]]
== Roll over automatically
The rollover action enables you to automatically roll over to a new index based
on the index size, document count, or age. When a rollover is triggered, a new
index is created, the write alias is updated to point to the new index, and all
subsequent updates are written to the new index.
Rolling over to a new index based on size, document count, or age is preferable
to time-based rollovers. Rolling over at an arbitrary time often results in
many small indices, which can have a negative impact on performance and
resource usage.
You control when the rollover action is triggered by specifying one or more
rollover parameters. The rollover is performed once any of the criteria are
met. Because the criteria are checked periodically, the index might grow
@ -22,7 +12,7 @@ checked, specify the `indices.lifecycle.poll_interval` cluster setting.
IMPORTANT: New indices created via rollover will not automatically inherit the
policy used by the old index, and will not use any policy by default. Therefore,
it is highly recommended to apply the policy via
<<applying-policy-to-template,index template>>, including a Rollover alias
<<apply-policy-template,index template>>, including a Rollover alias
setting, for your indices which specifies the policy you wish to use for each
new index.
@ -146,7 +136,7 @@ an exception to your normal Lifecycle Policy and switching the alias to a
different index by hand, but do not want to remove the index from {ilm}
completely.
This setting is set to `true` automatically by ILM upon the successful
This setting is set to `true` automatically by {ilm-init} upon the successful
completion of a Rollover Action. However, it will be removed if
<<ilm-remove-policy,the policy is removed>> from the index.

10
docs/reference/redirects.asciidoc
View File

@ -697,3 +697,13 @@ This section was deleted.
=== Assertions
This page was deleted.
[role="exclude",id="_actions"]
=== {ilm-init} actions
See <<ilm-actions>>.
[role="exclude",id="ilm-policy-definition"]
=== {ilm-init} policy definition
See <<ilm-index-lifecycle>>.