mirror of
https://github.com/honeymoose/OpenSearch.git
synced 2025-02-06 13:08:29 +00:00
49bd8715ff
We inject an Unfollow action before Shrink because the Shrink action cannot be safely used on a following index, as it may not be fully caught up with the leader index before the "original" following index is deleted and replaced with a non-following Shrunken index. The Unfollow action will verify that 1) the index is marked as "complete", and 2) all operations up to this point have been replicated from the leader to the follower before explicitly disconnecting the follower from the leader. Injecting an Unfollow action before the Rollover action is done mainly as a convenience: This allow users to use the same lifecycle policy on both the leader and follower cluster without having to explictly modify the policy to unfollow the index, while doing what we expect users to want in most cases.
748 lines
21 KiB
Plaintext
748 lines
21 KiB
Plaintext
[role="xpack"]
|
|
[testenv="basic"]
|
|
[[ilm-policy-definition]]
|
|
== Policy phases and actions
|
|
|
|
There are four stages in the index lifecycle, in the order
|
|
they are executed.
|
|
|
|
[options="header"]
|
|
|======
|
|
| Name | Description
|
|
| `hot` | The index is actively being written to
|
|
| `warm` | The index is generally not being written to, but is still queried
|
|
| `cold` | The index is no longer being updated and is seldom queried. The
|
|
information still needs to be searchable, but it's okay if those queries are
|
|
slower.
|
|
| `delete` | The index is no longer needed and can safely be deleted
|
|
|======
|
|
|
|
Each of these stages is called a "phase". A policy does not need to configure
|
|
each phase for an index. For example, one policy may define only the hot
|
|
phase and the delete phase, while another may define all four phases.
|
|
|
|
=== Timing
|
|
|
|
Indices enter phases based on a phase's `min_age` parameter.
|
|
The index will not enter the phase until the index's age is older than that
|
|
of the `min_age`. The parameter is configured using a time
|
|
duration format (see <<time-units, Time Units>>).
|
|
|
|
`min_age` defaults to zero seconds `0s` for each phase if not specified.
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
PUT _ilm/policy/my_policy
|
|
{
|
|
"policy": {
|
|
"phases": {
|
|
"warm": {
|
|
"min_age": "1d",
|
|
"actions": {
|
|
"allocate": {
|
|
"number_of_replicas": 1
|
|
}
|
|
}
|
|
},
|
|
"delete": {
|
|
"min_age": "30d",
|
|
"actions": {
|
|
"delete": {}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
|
|
The Above example configures a policy that moves the index into the warm
|
|
phase after one day. Until then, the index is in a waiting state. After
|
|
moving into the warm phase, it will wait until 30 days have elapsed before
|
|
moving to the delete phase and deleting the index.
|
|
|
|
`min_age` is usually the time elapsed from the time the index is created. If the
|
|
index is rolled over, then `min_age` is the time elapsed from the time the index
|
|
is rolled over. The intention here is to execute following phases and actions
|
|
relative to when data was written last to a rolled over index.
|
|
|
|
The previous phase's actions must complete before {ilm} will check `min_age`
|
|
and transition into the next phase.
|
|
|
|
=== Phase Execution
|
|
|
|
The current phase definition, of an index's policy being executed, is stored
|
|
in the index's metadata. The phase and its actions are compiled into a series
|
|
of discrete steps that are executed sequentially. Since some {ilm-init} actions
|
|
are more complex and involve multiple operations against an index, each of these
|
|
operations are done in isolation in a unit called a "step". The
|
|
<<ilm-explain-lifecycle,Explain Lifecycle API>> exposes this information to us
|
|
to see which step our index is either to execute next, or is currently
|
|
executing.
|
|
|
|
=== Actions
|
|
|
|
The below list shows the actions which are available in each phase.
|
|
|
|
* Hot
|
|
- <<ilm-set-priority-action,Set Priority>>
|
|
- <<ilm-rollover-action,Rollover>>
|
|
- <<ilm-unfollow-action,Unfollow>>
|
|
* Warm
|
|
- <<ilm-set-priority-action,Set Priority>>
|
|
- <<ilm-allocate-action,Allocate>>
|
|
- <<ilm-readonly-action,Read-Only>>
|
|
- <<ilm-forcemerge-action,Force Merge>>
|
|
- <<ilm-shrink-action,Shrink>>
|
|
- <<ilm-unfollow-action,Unfollow>>
|
|
* Cold
|
|
- <<ilm-set-priority-action,Set Priority>>
|
|
- <<ilm-allocate-action,Allocate>>
|
|
- <<ilm-freeze-action,Freeze>>
|
|
- <<ilm-unfollow-action,Unfollow>>
|
|
* Delete
|
|
- <<ilm-delete-action,Delete>>
|
|
|
|
[[ilm-allocate-action]]
|
|
==== Allocate
|
|
|
|
Phases allowed: warm, cold.
|
|
|
|
The Allocate action allows you to specify which nodes are allowed to host the
|
|
shards of the index and set the number of replicas.
|
|
Behind the scenes, it is modifying the index settings
|
|
for shard filtering and/or replica counts. When updating the number of replicas,
|
|
configuring allocation rules is optional. When configuring allocation rules,
|
|
setting number of replicas is optional. Although this action can be treated as
|
|
two separate index settings updates, both can be configured at once.
|
|
|
|
Read more about index replicas <<getting-started-shards-and-replicas,here>>.
|
|
Read more about shard allocation filtering in
|
|
the <<shard-allocation-filtering,Shard allocation filtering documentation>>.
|
|
|
|
[[ilm-allocate-options]]
|
|
.Allocate Options
|
|
[options="header"]
|
|
|======
|
|
| Name | Required | Default | Description
|
|
| `number_of_replicas` | no | - | The number of replicas to
|
|
assign to the index
|
|
| `include` | no | - | assigns an index to nodes
|
|
having at least _one_ of the attributes
|
|
| `exclude` | no | - | assigns an index to nodes having
|
|
_none_ of the attributes
|
|
| `require` | no | - | assigns an index to nodes having
|
|
_all_ of the attributes
|
|
|======
|
|
|
|
If `number_of_replicas` is not configured, then at least one of `include`,
|
|
`exclude`, and `require` is required. An empty Allocate Action with no configuration
|
|
is invalid.
|
|
|
|
===== Example: Change number of replicas
|
|
|
|
In this example, the index's number of replicas is changed to `2`, while allocation
|
|
rules are unchanged.
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
PUT _ilm/policy/my_policy
|
|
{
|
|
"policy": {
|
|
"phases": {
|
|
"warm": {
|
|
"actions": {
|
|
"allocate" : {
|
|
"number_of_replicas" : 2
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
|
|
===== Example: Assign index to node with specific "box_type" attribute
|
|
|
|
This example assigns the index to nodes with `box_type` attribute of "hot" or "warm".
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
PUT _ilm/policy/my_policy
|
|
{
|
|
"policy": {
|
|
"phases": {
|
|
"warm": {
|
|
"actions": {
|
|
"allocate" : {
|
|
"include" : {
|
|
"box_type": "hot,warm"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
|
|
===== Example: Assign index to a specific node and update replica settings
|
|
|
|
This example updates the index to have one replica per shard and be allocated
|
|
to nodes with a `box_type` attribute of "cold".
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
PUT _ilm/policy/my_policy
|
|
{
|
|
"policy": {
|
|
"phases": {
|
|
"warm": {
|
|
"actions": {
|
|
"allocate" : {
|
|
"number_of_replicas": 1,
|
|
"require" : {
|
|
"box_type": "cold"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
|
|
[[ilm-delete-action]]
|
|
==== Delete
|
|
|
|
Phases allowed: delete.
|
|
|
|
The Delete Action does just that, it deletes the index.
|
|
|
|
This action does not have any options associated with it.
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
PUT _ilm/policy/my_policy
|
|
{
|
|
"policy": {
|
|
"phases": {
|
|
"delete": {
|
|
"actions": {
|
|
"delete" : { }
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
|
|
[[ilm-forcemerge-action]]
|
|
==== Force Merge
|
|
|
|
Phases allowed: warm.
|
|
|
|
NOTE: Index will be be made read-only when this action is run
|
|
(see: <<dynamic-index-settings,index.blocks.write>>)
|
|
|
|
The Force Merge Action <<indices-forcemerge,force merges>> the index into at
|
|
most a specific number of <<indices-segments,segments>>.
|
|
|
|
[[ilm-forcemerge-options]]
|
|
.Force Merge Options
|
|
[options="header"]
|
|
|======
|
|
| Name | Required | Default | Description
|
|
| `max_num_segments` | yes | - | The number of
|
|
segments to merge to.
|
|
To fully merge the
|
|
index, set it to `1`
|
|
|======
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
PUT _ilm/policy/my_policy
|
|
{
|
|
"policy": {
|
|
"phases": {
|
|
"warm": {
|
|
"actions": {
|
|
"forcemerge" : {
|
|
"max_num_segments": 1
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
|
|
[[ilm-freeze-action]]
|
|
==== Freeze
|
|
|
|
Phases allowed: cold.
|
|
|
|
This action will <<frozen-indices, freeze>> the index
|
|
by calling the <<freeze-index-api, Freeze Index API>>.
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
PUT _ilm/policy/my_policy
|
|
{
|
|
"policy": {
|
|
"phases": {
|
|
"cold": {
|
|
"actions": {
|
|
"freeze" : { }
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
|
|
[IMPORTANT]
|
|
================================
|
|
Freezing an index will close the index and reopen it within the same API call.
|
|
This causes primaries to not be allocated for a short amount of time and
|
|
causes the cluster to go red until the primaries are allocated again.
|
|
This limitation might be removed in the future.
|
|
================================
|
|
|
|
[[ilm-readonly-action]]
|
|
==== Read-Only
|
|
|
|
Phases allowed: warm.
|
|
|
|
This action will set the index to be read-only
|
|
(see: <<dynamic-index-settings,index.blocks.write>>)
|
|
|
|
This action does not have any options associated with it.
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
PUT _ilm/policy/my_policy
|
|
{
|
|
"policy": {
|
|
"phases": {
|
|
"warm": {
|
|
"actions": {
|
|
"readonly" : { }
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
|
|
[[ilm-rollover-action]]
|
|
==== Rollover
|
|
|
|
Phases allowed: hot.
|
|
|
|
[WARNING]
|
|
index format must match pattern '^.*-\\d+$', for example (`logs-000001`).
|
|
[WARNING]
|
|
The managed index must set `index.lifecycle.rollover_alias` as the
|
|
alias to rollover. The index must also be the write index for the alias.
|
|
|
|
[IMPORTANT]
|
|
If a policy using the Rollover action is used on a <<ccr-put-follow,follower
|
|
index>>, policy execution will wait until the leader index rolls over (or has
|
|
<<skipping-rollover, otherwise been marked as complete>>), then convert the
|
|
follower index into a regular index as if <<ilm-unfollow-action,the Unfollow
|
|
action>> had been used instead of rolling over.
|
|
|
|
For example, if an index to be managed has an alias `my_data`. The managed
|
|
index "my_index" must be the write index for the alias. For more information, read
|
|
<<indices-rollover-is-write-index,Write Index Alias Behavior>>.
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
PUT my_index
|
|
{
|
|
"settings": {
|
|
"index.lifecycle.name": "my_policy",
|
|
"index.lifecycle.rollover_alias": "my_data"
|
|
},
|
|
"aliases": {
|
|
"my_data": {
|
|
"is_write_index": true
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
|
|
The Rollover Action rolls an alias over to a new index when the
|
|
existing index meets one of the rollover conditions.
|
|
|
|
|
|
[[ilm-rollover-options]]
|
|
.Rollover Options
|
|
[options="header"]
|
|
|======
|
|
| Name | Required | Default | Description
|
|
| `max_size` | no | - | max index storage size.
|
|
See <<byte-units, Byte Units>>
|
|
for formatting
|
|
| `max_docs` | no | - | max number of documents an
|
|
index is to contain before
|
|
rolling over.
|
|
| `max_age` | no | - | max time elapsed from index
|
|
creation. See
|
|
<<time-units, Time Units>>
|
|
for formatting
|
|
|======
|
|
|
|
At least one of `max_size`, `max_docs`, `max_age` or any combinations of the
|
|
three are required to be specified.
|
|
|
|
===== Example: Rollover when index is too large
|
|
|
|
This example rolls the index over when it is at least 100 gigabytes.
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
PUT _ilm/policy/my_policy
|
|
{
|
|
"policy": {
|
|
"phases": {
|
|
"hot": {
|
|
"actions": {
|
|
"rollover" : {
|
|
"max_size": "100GB"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
|
|
===== Example: Rollover when index has too many documents
|
|
|
|
This example rolls the index over when it contains at least
|
|
100000000 documents.
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
PUT _ilm/policy/my_policy
|
|
{
|
|
"policy": {
|
|
"phases": {
|
|
"hot": {
|
|
"actions": {
|
|
"rollover" : {
|
|
"max_docs": 100000000
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
|
|
===== Example: Rollover when index is too old
|
|
|
|
This example rolls the index over when it has been created at least
|
|
7 days ago.
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
PUT _ilm/policy/my_policy
|
|
{
|
|
"policy": {
|
|
"phases": {
|
|
"hot": {
|
|
"actions": {
|
|
"rollover" : {
|
|
"max_age": "7d"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
|
|
===== Example: Rollover when index is too old or too large
|
|
|
|
This example rolls the index over when it has been created at least
|
|
7 days ago or it is at least 100 gigabytes. In this case, the index will be
|
|
rolled over when any of the conditions is met.
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
PUT _ilm/policy/my_policy
|
|
{
|
|
"policy": {
|
|
"phases": {
|
|
"hot": {
|
|
"actions": {
|
|
"rollover" : {
|
|
"max_age": "7d",
|
|
"max_size": "100GB"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
|
|
|
|
===== Example: Rollover condition stalls phase transition
|
|
|
|
The Rollover action will only complete once one of its conditions is
|
|
met. This means that any proceeding phases will be blocked until Rollover
|
|
succeeds.
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
PUT /_ilm/policy/rollover_policy
|
|
{
|
|
"policy": {
|
|
"phases": {
|
|
"hot": {
|
|
"actions": {
|
|
"rollover": {
|
|
"max_size": "50G"
|
|
}
|
|
}
|
|
},
|
|
"delete": {
|
|
"min_age": "1d",
|
|
"actions": {
|
|
"delete": {}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
|
|
The above example illustrates a policy which attempts to delete an
|
|
index one day after the index has been rolled over. It does not
|
|
delete the index one day after it has been created.
|
|
|
|
[[ilm-set-priority-action]]
|
|
==== Set Priority
|
|
|
|
Phases allowed: hot, warm, cold.
|
|
|
|
This action sets the <<recovery-prioritization, index priority>> on the index as
|
|
soon as the policy enters the hot, warm, or cold phase. Indices with a higher
|
|
priority will be recovered before indices with lower priorities following a node
|
|
restart. Generally, indexes in the hot phase should have the highest value and
|
|
indexes in the cold phase should have the lowest values. For example:
|
|
100 for the hot phase, 50 for the warm phase, and 0 for the cold phase.
|
|
Indicies that don't set this value have an implicit default priority of 1.
|
|
|
|
[[ilm-set-priority-options]]
|
|
.Set Priority Options
|
|
[options="header"]
|
|
|======
|
|
| Name | Required | Default | Description
|
|
| `priority` | yes | - | The priority for the index. Must be 0 or greater.
|
|
The value may also be set to null to remove the priority.
|
|
|
|
|======
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
PUT _ilm/policy/my_policy
|
|
{
|
|
"policy": {
|
|
"phases": {
|
|
"warm": {
|
|
"actions": {
|
|
"set_priority" : {
|
|
"priority": 50
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
|
|
[[ilm-shrink-action]]
|
|
==== Shrink
|
|
|
|
NOTE: Index will be be made read-only when this action is run
|
|
(see: <<dynamic-index-settings,index.blocks.write>>)
|
|
[IMPORTANT]
|
|
If a policy using the Shrink action is used on a <<ccr-put-follow,follower
|
|
index>>, policy execution will wait until the leader index rolls over (or has
|
|
<<skipping-rollover, otherwise been marked as complete>>), then convert the
|
|
follower index into a regular index as if <<ilm-unfollow-action,the Unfollow
|
|
action>> had been used before shrink is applied, as shrink cannot be safely
|
|
applied to follower indices.
|
|
|
|
This action shrinks an existing index into a new index with fewer primary
|
|
shards. It calls the <<indices-shrink-index,Shrink API>> to shrink the index.
|
|
Since allocating all the primary shards of the index to one node is a
|
|
prerequisite, this action will first allocate the primary shards to a valid
|
|
node. After shrinking, it will swap aliases pointing to the original index
|
|
into the new shrunken index. The new index will also have a new name:
|
|
"shrink-<origin-index-name>". So if the original index was called "logs",
|
|
then the new index will be named "shrink-logs".
|
|
|
|
[[ilm-shrink-options]]
|
|
.Shrink Options
|
|
[options="header"]
|
|
|======
|
|
| Name | Required | Default | Description
|
|
| `number_of_shards` | yes | - | The number of shards
|
|
to shrink to. must be
|
|
a factor of the number
|
|
of shards in the
|
|
source index.
|
|
|======
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
PUT _ilm/policy/my_policy
|
|
{
|
|
"policy": {
|
|
"phases": {
|
|
"warm": {
|
|
"actions": {
|
|
"shrink" : {
|
|
"number_of_shards": 1
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
|
|
[[ilm-unfollow-action]]
|
|
==== Unfollow
|
|
|
|
[IMPORTANT]
|
|
This action may be used explicitly, as shown below, but this action is also run
|
|
before <<ilm-rollover-action,the Rollover action>> and <<ilm-shrink-action,the
|
|
Shrink action>> as described in the documentation for those actions.
|
|
|
|
This action turns a {ref}/ccr-apis.html[ccr] follower index
|
|
into a regular index. This can be desired when moving follower
|
|
indices into the next phase. Also certain actions like shrink
|
|
and rollover can then be performed safely on follower indices.
|
|
|
|
This action will wait until is it safe to convert a follower index into a
|
|
regular index. In particular, the following conditions must be met:
|
|
|
|
* The leader index must have `index.lifecycle.indexing_complete` set to `true`.
|
|
This happens automatically if the leader index is rolled over using
|
|
<<ilm-rollover-action,the Rollover action>>, or may be set manually using
|
|
the <<indices-update-settings,Index Settings API>>.
|
|
* All operations performed on the leader index must have been replicated to the
|
|
follower index. This ensures that no operations will be lost when the index is
|
|
converted into a regular index.
|
|
|
|
If the unfollow action encounters a follower index then
|
|
the following operations will be performed on it:
|
|
|
|
* Pauses indexing following for the follower index.
|
|
* Closes the follower index.
|
|
* Unfollows the follower index.
|
|
* Opens the follower index (which is at this point is a regular index).
|
|
|
|
The unfollow action does not have any options and
|
|
if it encounters a non follower index, then the
|
|
unfollow action leaves that index untouched and
|
|
lets the next action operate on this index.
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
PUT _ilm/policy/my_policy
|
|
{
|
|
"policy": {
|
|
"phases": {
|
|
"hot": {
|
|
"actions": {
|
|
"unfollow" : {}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
|
|
=== Full Policy
|
|
|
|
With all of these actions, we can support complex management strategies for our
|
|
indices. This policy will define an index that will start in the hot phase,
|
|
rolling over every 50 GB or 7 days. After 30 days it enters the warm phase
|
|
and increases the replicas to 2, force merges and shrinks. After 60 days
|
|
it enters the cold phase and allocates to "cold" nodes, and after 90 days the
|
|
index is deleted.
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
PUT _ilm/policy/full_policy
|
|
{
|
|
"policy": {
|
|
"phases": {
|
|
"hot": {
|
|
"actions": {
|
|
"rollover": {
|
|
"max_age": "7d",
|
|
"max_size": "50G"
|
|
}
|
|
}
|
|
},
|
|
"warm": {
|
|
"min_age": "30d",
|
|
"actions": {
|
|
"forcemerge": {
|
|
"max_num_segments": 1
|
|
},
|
|
"shrink": {
|
|
"number_of_shards": 1
|
|
},
|
|
"allocate": {
|
|
"number_of_replicas": 2
|
|
}
|
|
}
|
|
},
|
|
"cold": {
|
|
"min_age": "60d",
|
|
"actions": {
|
|
"allocate": {
|
|
"require": {
|
|
"type": "cold"
|
|
}
|
|
}
|
|
}
|
|
},
|
|
"delete": {
|
|
"min_age": "90d",
|
|
"actions": {
|
|
"delete": {}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|