736 lines
21 KiB
Plaintext
736 lines
21 KiB
Plaintext
[role="xpack"]
|
|
[testenv="basic"]
|
|
[[ilm-policy-definition]]
|
|
== Policy phases and actions
|
|
|
|
There are four stages in the index lifecycle, in the order
|
|
they are executed.
|
|
|
|
[options="header"]
|
|
|======
|
|
| Name | Description
|
|
| `hot` | The index is actively being written to
|
|
| `warm` | The index is generally not being written to, but is still queried
|
|
| `cold` | The index is no longer being updated and is seldom queried. The
|
|
information still needs to be searchable, but it's okay if those queries are
|
|
slower.
|
|
| `delete` | The index is no longer needed and can safely be deleted
|
|
|======
|
|
|
|
Each of these stages is called a "phase". A policy does not need to configure
|
|
each phase for an index. For example, one policy may define only the hot
|
|
phase and the delete phase, while another may define all four phases.
|
|
|
|
=== Timing
|
|
|
|
Indices enter phases based on a phase's `min_age` parameter.
|
|
The index will not enter the phase until the index's age is older than that
|
|
of the `min_age`. The parameter is configured using a time
|
|
duration format (see <<time-units, Time Units>>).
|
|
|
|
`min_age` defaults to zero seconds `0s` for each phase if not specified.
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
PUT _ilm/policy/my_policy
|
|
{
|
|
"policy": {
|
|
"phases": {
|
|
"warm": {
|
|
"min_age": "1d",
|
|
"actions": {
|
|
"allocate": {
|
|
"number_of_replicas": 1
|
|
}
|
|
}
|
|
},
|
|
"delete": {
|
|
"min_age": "30d",
|
|
"actions": {
|
|
"delete": {}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
|
|
The Above example configures a policy that moves the index into the warm
|
|
phase after one day. Until then, the index is in a waiting state. After
|
|
moving into the warm phase, it will wait until 30 days have elapsed before
|
|
moving to the delete phase and deleting the index.
|
|
|
|
`min_age` is usually the time elapsed from the time the index is created. If the
|
|
index is rolled over, then `min_age` is the time elapsed from the time the index
|
|
is rolled over. The intention here is to execute following phases and actions
|
|
relative to when data was written last to a rolled over index.
|
|
|
|
The previous phase's actions must complete before {ilm} will check `min_age` and
|
|
transition into the next phase. By default, {ilm} checks for indices that meet
|
|
policy criteria, like `min_age`, every 10 minutes. You can use the
|
|
`indices.lifecycle.poll_interval` cluster setting to control how often this
|
|
check occurs.
|
|
|
|
=== Phase Execution
|
|
|
|
The current phase definition, of an index's policy being executed, is stored
|
|
in the index's metadata. The phase and its actions are compiled into a series
|
|
of discrete steps that are executed sequentially. Since some {ilm-init} actions
|
|
are more complex and involve multiple operations against an index, each of these
|
|
operations are done in isolation in a unit called a "step". The
|
|
<<ilm-explain-lifecycle,Explain Lifecycle API>> exposes this information to us
|
|
to see which step our index is either to execute next, or is currently
|
|
executing.
|
|
|
|
=== Actions
|
|
|
|
The below list shows the actions which are available in each phase.
|
|
|
|
NOTE: The order that configured actions are performed in within each phase is
|
|
determined by automatically by {ilm-init}, and cannot be changed by changing the
|
|
policy definition.
|
|
|
|
* Hot
|
|
- <<ilm-set-priority-action,Set Priority>>
|
|
- <<ilm-unfollow-action,Unfollow>>
|
|
- <<ilm-rollover-action,Rollover>>
|
|
* Warm
|
|
- <<ilm-set-priority-action,Set Priority>>
|
|
- <<ilm-unfollow-action,Unfollow>>
|
|
- <<ilm-readonly-action,Read-Only>>
|
|
- <<ilm-allocate-action,Allocate>>
|
|
- <<ilm-shrink-action,Shrink>>
|
|
- <<ilm-forcemerge-action,Force Merge>>
|
|
* Cold
|
|
- <<ilm-set-priority-action,Set Priority>>
|
|
- <<ilm-unfollow-action,Unfollow>>
|
|
- <<ilm-allocate-action,Allocate>>
|
|
- <<ilm-freeze-action,Freeze>>
|
|
* Delete
|
|
- <<ilm-delete-action,Delete>>
|
|
|
|
[[ilm-allocate-action]]
|
|
==== Allocate
|
|
|
|
Phases allowed: warm, cold.
|
|
|
|
The Allocate action allows you to specify which nodes are allowed to host the
|
|
shards of the index and set the number of replicas.
|
|
Behind the scenes, it is modifying the index settings
|
|
for shard filtering and/or replica counts. When updating the number of replicas,
|
|
configuring allocation rules is optional. When configuring allocation rules,
|
|
setting number of replicas is optional. Although this action can be treated as
|
|
two separate index settings updates, both can be configured at once.
|
|
|
|
For more information about how {es} uses replicas for scaling, see
|
|
<<scalability>>. See <<shard-allocation-filtering>> for more information about
|
|
controlling where Elasticsearch allocates shards of a particular index.
|
|
|
|
[[ilm-allocate-options]]
|
|
.Allocate Options
|
|
[options="header"]
|
|
|======
|
|
| Name | Required | Default | Description
|
|
| `number_of_replicas` | no | - | The number of replicas to
|
|
assign to the index
|
|
| `include` | no | - | assigns an index to nodes
|
|
having at least _one_ of the attributes
|
|
| `exclude` | no | - | assigns an index to nodes having
|
|
_none_ of the attributes
|
|
| `require` | no | - | assigns an index to nodes having
|
|
_all_ of the attributes
|
|
|======
|
|
|
|
If `number_of_replicas` is not configured, then at least one of `include`,
|
|
`exclude`, and `require` is required. An empty Allocate Action with no configuration
|
|
is invalid.
|
|
|
|
===== Example: Change number of replicas
|
|
|
|
In this example, the index's number of replicas is changed to `2`, while allocation
|
|
rules are unchanged.
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
PUT _ilm/policy/my_policy
|
|
{
|
|
"policy": {
|
|
"phases": {
|
|
"warm": {
|
|
"actions": {
|
|
"allocate" : {
|
|
"number_of_replicas" : 2
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
|
|
===== Example: Assign index to node with specific "box_type" attribute
|
|
|
|
This example assigns the index to nodes with `box_type` attribute of "hot" or "warm".
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
PUT _ilm/policy/my_policy
|
|
{
|
|
"policy": {
|
|
"phases": {
|
|
"warm": {
|
|
"actions": {
|
|
"allocate" : {
|
|
"include" : {
|
|
"box_type": "hot,warm"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
|
|
===== Example: Assign index to a specific node and update replica settings
|
|
|
|
This example updates the index to have one replica per shard and be allocated
|
|
to nodes with a `box_type` attribute of "cold".
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
PUT _ilm/policy/my_policy
|
|
{
|
|
"policy": {
|
|
"phases": {
|
|
"warm": {
|
|
"actions": {
|
|
"allocate" : {
|
|
"number_of_replicas": 1,
|
|
"require" : {
|
|
"box_type": "cold"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
|
|
[[ilm-delete-action]]
|
|
==== Delete
|
|
|
|
Phases allowed: delete.
|
|
|
|
The Delete Action does just that, it deletes the index.
|
|
|
|
This action does not have any options associated with it.
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
PUT _ilm/policy/my_policy
|
|
{
|
|
"policy": {
|
|
"phases": {
|
|
"delete": {
|
|
"actions": {
|
|
"delete" : { }
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
|
|
[[ilm-forcemerge-action]]
|
|
==== Force Merge
|
|
|
|
Phases allowed: warm.
|
|
|
|
NOTE: Index will be be made read-only when this action is run
|
|
(see: <<dynamic-index-settings,index.blocks.write>>)
|
|
|
|
The Force Merge Action <<indices-forcemerge,force merges>> the index into at
|
|
most a specific number of <<indices-segments,segments>>.
|
|
|
|
[[ilm-forcemerge-options]]
|
|
.Force Merge Options
|
|
[options="header"]
|
|
|======
|
|
| Name | Required | Default | Description
|
|
| `max_num_segments` | yes | - | The number of
|
|
segments to merge to.
|
|
To fully merge the
|
|
index, set it to `1`
|
|
|======
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
PUT _ilm/policy/my_policy
|
|
{
|
|
"policy": {
|
|
"phases": {
|
|
"warm": {
|
|
"actions": {
|
|
"forcemerge" : {
|
|
"max_num_segments": 1
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
|
|
[[ilm-freeze-action]]
|
|
==== Freeze
|
|
|
|
Phases allowed: cold.
|
|
|
|
This action will <<frozen-indices, freeze>> the index
|
|
by calling the <<freeze-index-api, Freeze Index API>>.
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
PUT _ilm/policy/my_policy
|
|
{
|
|
"policy": {
|
|
"phases": {
|
|
"cold": {
|
|
"actions": {
|
|
"freeze" : { }
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
|
|
[IMPORTANT]
|
|
================================
|
|
Freezing an index will close the index and reopen it within the same API call.
|
|
This causes primaries to not be allocated for a short amount of time and
|
|
causes the cluster to go red until the primaries are allocated again.
|
|
This limitation might be removed in the future.
|
|
================================
|
|
|
|
[[ilm-readonly-action]]
|
|
==== Read-Only
|
|
|
|
Phases allowed: warm.
|
|
|
|
This action will set the index to be read-only
|
|
(see: <<dynamic-index-settings,index.blocks.write>>)
|
|
|
|
This action does not have any options associated with it.
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
PUT _ilm/policy/my_policy
|
|
{
|
|
"policy": {
|
|
"phases": {
|
|
"warm": {
|
|
"actions": {
|
|
"readonly" : { }
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
|
|
[[ilm-rollover-action]]
|
|
==== Rollover
|
|
|
|
Phases allowed: hot.
|
|
|
|
[WARNING]
|
|
index format must match pattern '^.*-\\d+$', for example (`logs-000001`).
|
|
[WARNING]
|
|
The managed index must set `index.lifecycle.rollover_alias` as the
|
|
alias to rollover. The index must also be the write index for the alias.
|
|
|
|
[IMPORTANT]
|
|
If a policy using the Rollover action is used on a <<ccr-put-follow,follower
|
|
index>>, policy execution will wait until the leader index rolls over (or has
|
|
<<skipping-rollover, otherwise been marked as complete>>), then convert the
|
|
follower index into a regular index as if <<ilm-unfollow-action,the Unfollow
|
|
action>> had been used instead of rolling over.
|
|
|
|
For example, if an index to be managed has an alias `my_data`. The managed
|
|
index "my_index" must be the write index for the alias. For more information, read
|
|
<<indices-rollover-is-write-index,Write Index Alias Behavior>>.
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
PUT my_index
|
|
{
|
|
"settings": {
|
|
"index.lifecycle.name": "my_policy",
|
|
"index.lifecycle.rollover_alias": "my_data"
|
|
},
|
|
"aliases": {
|
|
"my_data": {
|
|
"is_write_index": true
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
|
|
The Rollover Action rolls an alias over to a new index when the
|
|
existing index meets one of the rollover conditions.
|
|
|
|
|
|
[[ilm-rollover-options]]
|
|
.Rollover Options
|
|
[options="header"]
|
|
|======
|
|
| Name | Required | Default | Description
|
|
| `max_size` | no | - | max primary shard index storage size.
|
|
See <<byte-units, Byte Units>>
|
|
for formatting
|
|
| `max_docs` | no | - | max number of documents an
|
|
index is to contain before
|
|
rolling over.
|
|
| `max_age` | no | - | max time elapsed from index
|
|
creation. See
|
|
<<time-units, Time Units>>
|
|
for formatting
|
|
|======
|
|
|
|
At least one of `max_size`, `max_docs`, `max_age` or any combinations of the
|
|
three are required to be specified.
|
|
|
|
===== Example: Rollover when index is too large
|
|
|
|
This example rolls the index over when it is at least 100 gigabytes.
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
PUT _ilm/policy/my_policy
|
|
{
|
|
"policy": {
|
|
"phases": {
|
|
"hot": {
|
|
"actions": {
|
|
"rollover" : {
|
|
"max_size": "100GB"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
|
|
===== Example: Rollover when index has too many documents
|
|
|
|
This example rolls the index over when it contains at least
|
|
100000000 documents.
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
PUT _ilm/policy/my_policy
|
|
{
|
|
"policy": {
|
|
"phases": {
|
|
"hot": {
|
|
"actions": {
|
|
"rollover" : {
|
|
"max_docs": 100000000
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
|
|
===== Example: Rollover when index is too old
|
|
|
|
This example rolls the index over when it has been created at least
|
|
7 days ago.
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
PUT _ilm/policy/my_policy
|
|
{
|
|
"policy": {
|
|
"phases": {
|
|
"hot": {
|
|
"actions": {
|
|
"rollover" : {
|
|
"max_age": "7d"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
|
|
===== Example: Rollover when index is too old or too large
|
|
|
|
This example rolls the index over when it has been created at least
|
|
7 days ago or it is at least 100 gigabytes. In this case, the index will be
|
|
rolled over when any of the conditions is met.
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
PUT _ilm/policy/my_policy
|
|
{
|
|
"policy": {
|
|
"phases": {
|
|
"hot": {
|
|
"actions": {
|
|
"rollover" : {
|
|
"max_age": "7d",
|
|
"max_size": "100GB"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
|
|
===== Example: Rollover condition stalls phase transition
|
|
|
|
The Rollover action will only complete once one of its conditions is
|
|
met. This means that any proceeding phases will be blocked until Rollover
|
|
succeeds.
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
PUT /_ilm/policy/rollover_policy
|
|
{
|
|
"policy": {
|
|
"phases": {
|
|
"hot": {
|
|
"actions": {
|
|
"rollover": {
|
|
"max_size": "50G"
|
|
}
|
|
}
|
|
},
|
|
"delete": {
|
|
"min_age": "1d",
|
|
"actions": {
|
|
"delete": {}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
|
|
The above example illustrates a policy which attempts to delete an
|
|
index one day after the index has been rolled over. It does not
|
|
delete the index one day after it has been created.
|
|
|
|
[[ilm-set-priority-action]]
|
|
==== Set Priority
|
|
|
|
Phases allowed: hot, warm, cold.
|
|
|
|
This action sets the <<recovery-prioritization, index priority>> on the index as
|
|
soon as the policy enters the hot, warm, or cold phase. Indices with a higher
|
|
priority will be recovered before indices with lower priorities following a node
|
|
restart. Generally, indexes in the hot phase should have the highest value and
|
|
indexes in the cold phase should have the lowest values. For example:
|
|
100 for the hot phase, 50 for the warm phase, and 0 for the cold phase.
|
|
Indicies that don't set this value have an implicit default priority of 1.
|
|
|
|
[[ilm-set-priority-options]]
|
|
.Set Priority Options
|
|
[options="header"]
|
|
|======
|
|
| Name | Required | Default | Description
|
|
| `priority` | yes | - | The priority for the index. Must be 0 or greater.
|
|
The value may also be set to null to remove the priority.
|
|
|
|
|======
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
PUT _ilm/policy/my_policy
|
|
{
|
|
"policy": {
|
|
"phases": {
|
|
"warm": {
|
|
"actions": {
|
|
"set_priority" : {
|
|
"priority": 50
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
|
|
[[ilm-shrink-action]]
|
|
==== Shrink
|
|
|
|
NOTE: Index will be be made read-only when this action is run
|
|
(see: <<dynamic-index-settings,index.blocks.write>>)
|
|
[IMPORTANT]
|
|
If a policy using the Shrink action is used on a <<ccr-put-follow,follower
|
|
index>>, policy execution will wait until the leader index rolls over (or has
|
|
<<skipping-rollover, otherwise been marked as complete>>), then convert the
|
|
follower index into a regular index as if <<ilm-unfollow-action,the Unfollow
|
|
action>> had been used before shrink is applied, as shrink cannot be safely
|
|
applied to follower indices.
|
|
|
|
This action shrinks an existing index into a new index with fewer primary
|
|
shards. It calls the <<indices-shrink-index,Shrink API>> to shrink the index.
|
|
Since allocating all the primary shards of the index to one node is a
|
|
prerequisite, this action will first allocate the primary shards to a valid
|
|
node. After shrinking, it will swap aliases pointing to the original index
|
|
into the new shrunken index. The new index will also have a new name:
|
|
"shrink-<origin-index-name>". So if the original index was called "logs",
|
|
then the new index will be named "shrink-logs".
|
|
|
|
[[ilm-shrink-options]]
|
|
.Shrink Options
|
|
[options="header"]
|
|
|======
|
|
| Name | Required | Default | Description
|
|
| `number_of_shards` | yes | - | The number of shards
|
|
to shrink to. must be
|
|
a factor of the number
|
|
of shards in the
|
|
source index.
|
|
|======
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
PUT _ilm/policy/my_policy
|
|
{
|
|
"policy": {
|
|
"phases": {
|
|
"warm": {
|
|
"actions": {
|
|
"shrink" : {
|
|
"number_of_shards": 1
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
|
|
[[ilm-unfollow-action]]
|
|
==== Unfollow
|
|
|
|
[IMPORTANT]
|
|
This action may be used explicitly, as shown below, but this action is also run
|
|
before <<ilm-rollover-action,the Rollover action>> and <<ilm-shrink-action,the
|
|
Shrink action>> as described in the documentation for those actions.
|
|
|
|
This action turns a {ref}/ccr-apis.html[ccr] follower index
|
|
into a regular index. This can be desired when moving follower
|
|
indices into the next phase. Also certain actions like shrink
|
|
and rollover can then be performed safely on follower indices.
|
|
|
|
This action will wait until is it safe to convert a follower index into a
|
|
regular index. In particular, the following conditions must be met:
|
|
|
|
* The leader index must have `index.lifecycle.indexing_complete` set to `true`.
|
|
This happens automatically if the leader index is rolled over using
|
|
<<ilm-rollover-action,the Rollover action>>, or may be set manually using
|
|
the <<indices-update-settings,Index Settings API>>.
|
|
* All operations performed on the leader index must have been replicated to the
|
|
follower index. This ensures that no operations will be lost when the index is
|
|
converted into a regular index.
|
|
|
|
If the unfollow action encounters a follower index then
|
|
the following operations will be performed on it:
|
|
|
|
* Pauses indexing following for the follower index.
|
|
* Closes the follower index.
|
|
* Unfollows the follower index.
|
|
* Opens the follower index (which is at this point is a regular index).
|
|
|
|
The unfollow action does not have any options and
|
|
if it encounters a non follower index, then the
|
|
unfollow action leaves that index untouched and
|
|
lets the next action operate on this index.
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
PUT _ilm/policy/my_policy
|
|
{
|
|
"policy": {
|
|
"phases": {
|
|
"hot": {
|
|
"actions": {
|
|
"unfollow" : {}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
|
|
=== Full Policy
|
|
|
|
With all of these actions, we can support complex management strategies for our
|
|
indices. This policy will define an index that will start in the hot phase,
|
|
rolling over every 50 GB or 7 days. After 30 days it enters the warm phase
|
|
and increases the replicas to 2, force merges and shrinks. After 60 days
|
|
it enters the cold phase and allocates to "cold" nodes, and after 90 days the
|
|
index is deleted.
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
PUT _ilm/policy/full_policy
|
|
{
|
|
"policy": {
|
|
"phases": {
|
|
"hot": {
|
|
"actions": {
|
|
"rollover": {
|
|
"max_age": "7d",
|
|
"max_size": "50G"
|
|
}
|
|
}
|
|
},
|
|
"warm": {
|
|
"min_age": "30d",
|
|
"actions": {
|
|
"forcemerge": {
|
|
"max_num_segments": 1
|
|
},
|
|
"shrink": {
|
|
"number_of_shards": 1
|
|
},
|
|
"allocate": {
|
|
"number_of_replicas": 2
|
|
}
|
|
}
|
|
},
|
|
"cold": {
|
|
"min_age": "60d",
|
|
"actions": {
|
|
"allocate": {
|
|
"require": {
|
|
"type": "cold"
|
|
}
|
|
}
|
|
}
|
|
},
|
|
"delete": {
|
|
"min_age": "90d",
|
|
"actions": {
|
|
"delete": {}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|