[ILM] add docs explaining high-level concepts (#35322)
add docs explaining high-level concepts
This commit is contained in:
Tal Levy
GitHub
parent
803eccec11
commit
30a713cc3f
|
|
@ -52,10 +52,12 @@ hardware.
|
||||||
|
|
||||||
include::getting-started-ilm.asciidoc[]
|
include::getting-started-ilm.asciidoc[]
|
||||||
|
|
||||||
include::using-policies-rollover.asciidoc[]
|
include::policy-definitions.asciidoc[]
|
||||||
|
|
||||||
include::set-up-lifecycle-policy.asciidoc[]
|
include::set-up-lifecycle-policy.asciidoc[]
|
||||||
|
|
||||||
|
include::using-policies-rollover.asciidoc[]
|
||||||
|
|
||||||
include::update-lifecycle-policy.asciidoc[]
|
include::update-lifecycle-policy.asciidoc[]
|
||||||
|
|
||||||
include::get-index-lifecycle-information.asciidoc[]
|
include::get-index-lifecycle-information.asciidoc[]
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,549 @@
|
||||||
|
[role="xpack"]
|
||||||
|
[testenv="basic"]
|
||||||
|
[[ilm-policy-definition]]
|
||||||
|
== Policy Phases and Actions
|
||||||
|
|
||||||
|
There are four stages in the index lifecycle, in the order
|
||||||
|
they are executed.
|
||||||
|
|
||||||
|
[options="header"]
|
||||||
|
|======
|
||||||
|
| Name | Description
|
||||||
|
| `hot` | The index is actively being written to
|
||||||
|
| `warm` | The index is generally not being written to, but is still queried
|
||||||
|
| `cold` | The index is no longer being updated and is seldom queried. The
|
||||||
|
information still needs to be searchable, but it's okay if those queries are
|
||||||
|
slower.
|
||||||
|
| `delete` | The index is no longer needed and can safely be deleted
|
||||||
|
|======
|
||||||
|
|
||||||
|
Each of these stages is called a "phase". A policy does not need to configure
|
||||||
|
each phase for an index. For example, one policy may define only the hot
|
||||||
|
phase and the delete phase, while another may define all four phases.
|
||||||
|
|
||||||
|
=== Timing
|
||||||
|
|
||||||
|
Indices enter phases based on a phase's `min_age` parameter.
|
||||||
|
The index will not enter the phase until the index's age is older than that
|
||||||
|
of the `min_age`. The parameter is configured using a time
|
||||||
|
duration format (see <<time-units, Time Units>>).
|
||||||
|
|
||||||
|
`min_age` defaults to zero seconds `0s` for each phase if not specified.
|
||||||
|
|
||||||
|
[source,js]
|
||||||
|
--------------------------------------------------
|
||||||
|
PUT _ilm/policy/my_policy
|
||||||
|
{
|
||||||
|
"policy": {
|
||||||
|
"phases": {
|
||||||
|
"warm": {
|
||||||
|
"min_age": "1d",
|
||||||
|
"actions": {
|
||||||
|
"allocate": {
|
||||||
|
"number_of_replicas": 1
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"delete": {
|
||||||
|
"min_age": "30d",
|
||||||
|
"actions": {
|
||||||
|
"delete": {}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
--------------------------------------------------
|
||||||
|
// CONSOLE
|
||||||
|
|
||||||
|
The Above example configures a policy that moves the index into the warm
|
||||||
|
phase after one day. Until then, the index is in a waiting state. After
|
||||||
|
moving into the warm phase, it will wait until 30 days have elapsed before
|
||||||
|
moving to the delete phase and deleting the index.
|
||||||
|
|
||||||
|
`min_age` is usually the time elapsed from the time the index is created. If the
|
||||||
|
index is rolled over, then `min_age` is the time elapsed from the time the index
|
||||||
|
is rolled over. The intention here is to execute following phases and actions
|
||||||
|
relative to when data was written last to a rolled over index.
|
||||||
|
|
||||||
|
=== Actions
|
||||||
|
|
||||||
|
The below list shows the actions which are available in each phase.
|
||||||
|
|
||||||
|
* Hot
|
||||||
|
- <<ilm-rollover-action,Rollover>>
|
||||||
|
* Warm
|
||||||
|
- <<ilm-allocate-action,Allocate>>
|
||||||
|
- <<ilm-readonly-action,Read-Only>>
|
||||||
|
- <<ilm-forcemerge-action,Force Merge>>
|
||||||
|
- <<ilm-shrink-action,Shrink>>
|
||||||
|
* Cold
|
||||||
|
- <<ilm-allocate-action,Allocate>>
|
||||||
|
* Delete
|
||||||
|
- <<ilm-delete-action,Delete>>
|
||||||
|
|
||||||
|
[[ilm-allocate-action]]
|
||||||
|
==== Allocate
|
||||||
|
|
||||||
|
Phases allowed: warm, cold.
|
||||||
|
|
||||||
|
The Allocate action allows you to specify which nodes are allowed to host the
|
||||||
|
shards of the index and set the number of replicas.
|
||||||
|
Behind the scenes, it is modifying the index settings
|
||||||
|
for shard filtering and/or replica counts. When updating the number of replicas,
|
||||||
|
configuring allocation rules is optional. When configuring allocation rules,
|
||||||
|
setting number of replicas is optional. Although this action can be treated as
|
||||||
|
two separate index settings updates, both can be configured at once.
|
||||||
|
|
||||||
|
Read more about index replicas <<getting-started-shards-and-replicas,here>>.
|
||||||
|
Read more about shard allocation filtering in
|
||||||
|
the <<shard-allocation-filtering,Shard allocation filtering documentation>>.
|
||||||
|
|
||||||
|
[[ilm-allocate-options]]
|
||||||
|
.Allocate Options
|
||||||
|
[options="header"]
|
||||||
|
|======
|
||||||
|
| Name | Required | Default | Description
|
||||||
|
| `number_of_replicas` | no | - | The number of replicas to
|
||||||
|
assign to the index
|
||||||
|
| `include` | no | - | assigns an index to nodes
|
||||||
|
having at least _one_ of the attributes
|
||||||
|
| `exclude` | no | - | assigns an index to nodes having
|
||||||
|
_none_ of the attributes
|
||||||
|
| `require` | no | - | assigns an index to nodes having
|
||||||
|
_all_ of the attributes
|
||||||
|
|======
|
||||||
|
|
||||||
|
If `number_of_replicas` is not configured, then at least one of `include`,
|
||||||
|
`exclude`, and `require` is required. An empty Allocate Action with no configuration
|
||||||
|
is invalid.
|
||||||
|
|
||||||
|
===== Example: Change number of replicas
|
||||||
|
|
||||||
|
In this example, the index's number of replicas is changed to `2`, while allocation
|
||||||
|
rules are unchanged.
|
||||||
|
|
||||||
|
[source,js]
|
||||||
|
--------------------------------------------------
|
||||||
|
PUT _ilm/policy/my_policy
|
||||||
|
{
|
||||||
|
"policy": {
|
||||||
|
"phases": {
|
||||||
|
"warm": {
|
||||||
|
"actions": {
|
||||||
|
"allocate" : {
|
||||||
|
"number_of_replicas" : 2
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
--------------------------------------------------
|
||||||
|
// CONSOLE
|
||||||
|
|
||||||
|
===== Example: Assign index to node with specific "box_type" attribute
|
||||||
|
|
||||||
|
This example assigns the index to nodes with `box_type` attribute of "hot" or "warm".
|
||||||
|
|
||||||
|
[source,js]
|
||||||
|
--------------------------------------------------
|
||||||
|
PUT _ilm/policy/my_policy
|
||||||
|
{
|
||||||
|
"policy": {
|
||||||
|
"phases": {
|
||||||
|
"warm": {
|
||||||
|
"actions": {
|
||||||
|
"allocate" : {
|
||||||
|
"include" : {
|
||||||
|
"box_type": "hot,warm"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
--------------------------------------------------
|
||||||
|
// CONSOLE
|
||||||
|
|
||||||
|
===== Example: Assign index to a specific node and update replica settings
|
||||||
|
|
||||||
|
This example updates the index to have one replica per shard and be allocated
|
||||||
|
to nodes with a `box_type` attribute of "cold".
|
||||||
|
|
||||||
|
[source,js]
|
||||||
|
--------------------------------------------------
|
||||||
|
PUT _ilm/policy/my_policy
|
||||||
|
{
|
||||||
|
"policy": {
|
||||||
|
"phases": {
|
||||||
|
"warm": {
|
||||||
|
"actions": {
|
||||||
|
"allocate" : {
|
||||||
|
"number_of_replicas": 1,
|
||||||
|
"require" : {
|
||||||
|
"box_type": "cold"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
--------------------------------------------------
|
||||||
|
// CONSOLE
|
||||||
|
|
||||||
|
[[ilm-delete-action]]
|
||||||
|
==== Delete
|
||||||
|
|
||||||
|
Phases allowed: delete.
|
||||||
|
|
||||||
|
The Delete Action does just that, it deletes the index.
|
||||||
|
|
||||||
|
This action does not have any options associated with it.
|
||||||
|
|
||||||
|
[source,js]
|
||||||
|
--------------------------------------------------
|
||||||
|
PUT _ilm/policy/my_policy
|
||||||
|
{
|
||||||
|
"policy": {
|
||||||
|
"phases": {
|
||||||
|
"delete": {
|
||||||
|
"actions": {
|
||||||
|
"delete" : { }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
--------------------------------------------------
|
||||||
|
// CONSOLE
|
||||||
|
|
||||||
|
[[ilm-forcemerge-action]]
|
||||||
|
==== Force Merge
|
||||||
|
|
||||||
|
Phases allowed: warm.
|
||||||
|
|
||||||
|
NOTE: Index will be be made read-only when this action is run
|
||||||
|
(see: <<dynamic-index-settings,index.blocks.write>>)
|
||||||
|
|
||||||
|
The Force Merge Action <<indices-forcemerge,force merges>> the index into at
|
||||||
|
most a specific number of <<indices-segments,segments>>.
|
||||||
|
|
||||||
|
[[ilm-forcemerge-options]]
|
||||||
|
.Force Merge Options
|
||||||
|
[options="header"]
|
||||||
|
|======
|
||||||
|
| Name | Required | Default | Description
|
||||||
|
| `max_num_segments` | yes | - | The number of
|
||||||
|
segments to merge to.
|
||||||
|
To fully merge the
|
||||||
|
index, set it to `1`
|
||||||
|
|======
|
||||||
|
|
||||||
|
[source,js]
|
||||||
|
--------------------------------------------------
|
||||||
|
PUT _ilm/policy/my_policy
|
||||||
|
{
|
||||||
|
"policy": {
|
||||||
|
"phases": {
|
||||||
|
"warm": {
|
||||||
|
"actions": {
|
||||||
|
"forcemerge" : {
|
||||||
|
"max_num_segments": 1
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
--------------------------------------------------
|
||||||
|
// CONSOLE
|
||||||
|
|
||||||
|
[[ilm-readonly-action]]
|
||||||
|
==== Read-Only
|
||||||
|
|
||||||
|
Phases allowed: warm.
|
||||||
|
|
||||||
|
This action will set the index to be read-only
|
||||||
|
(see: <<dynamic-index-settings,index.blocks.write>>)
|
||||||
|
|
||||||
|
This action does not have any options associated with it.
|
||||||
|
|
||||||
|
[source,js]
|
||||||
|
--------------------------------------------------
|
||||||
|
PUT _ilm/policy/my_policy
|
||||||
|
{
|
||||||
|
"policy": {
|
||||||
|
"phases": {
|
||||||
|
"warm": {
|
||||||
|
"actions": {
|
||||||
|
"readonly" : { }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
--------------------------------------------------
|
||||||
|
// CONSOLE
|
||||||
|
|
||||||
|
[[ilm-rollover-action]]
|
||||||
|
==== Rollover
|
||||||
|
|
||||||
|
Phases allowed: hot.
|
||||||
|
|
||||||
|
[WARNING]
|
||||||
|
index format must match pattern '^.*-\\d+$', for example (`logs-000001`).
|
||||||
|
[WARNING]
|
||||||
|
The managed index must set `index.lifecycle.rollover_alias` as the
|
||||||
|
alias to rollover. The index must also be the write index for the alias.
|
||||||
|
|
||||||
|
For example, if an index to be managed has an alias `my_data`. The managed
|
||||||
|
index "my_index" must be the write index for the alias. For more information, read
|
||||||
|
<<indices-rollover-is-write-index,Write Index Alias Behavior>>.
|
||||||
|
|
||||||
|
[source,js]
|
||||||
|
--------------------------------------------------
|
||||||
|
PUT my_index
|
||||||
|
{
|
||||||
|
"settings": {
|
||||||
|
"index.lifecycle.name": "my_policy",
|
||||||
|
"index.lifecycle.rollover_alias": "my_data"
|
||||||
|
},
|
||||||
|
"aliases": {
|
||||||
|
"my_data": {
|
||||||
|
"is_write_index": true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
--------------------------------------------------
|
||||||
|
// CONSOLE
|
||||||
|
|
||||||
|
The Rollover Action rolls an alias over to a new index when the
|
||||||
|
existing index meets one of the rollover conditions.
|
||||||
|
|
||||||
|
|
||||||
|
[[ilm-rollover-options]]
|
||||||
|
.Rollover Options
|
||||||
|
[options="header"]
|
||||||
|
|======
|
||||||
|
| Name | Required | Default | Description
|
||||||
|
| `max_size` | no | - | max index storage size.
|
||||||
|
See <<byte-units, Byte Units>>
|
||||||
|
for formatting
|
||||||
|
| `max_docs` | no | - | max number of documents an
|
||||||
|
index is to contain before
|
||||||
|
rolling over.
|
||||||
|
| `max_age` | no | - | max time elapsed from index
|
||||||
|
creation. See
|
||||||
|
<<time-units, Time Units>>
|
||||||
|
for formatting
|
||||||
|
|======
|
||||||
|
|
||||||
|
At least one of `max_size`, `max_docs`, `max_age` or any combinations of the
|
||||||
|
three are required to be specified.
|
||||||
|
|
||||||
|
===== Example: Rollover when index is too large
|
||||||
|
|
||||||
|
This example rolls the index over when it is at least 100 gigabytes.
|
||||||
|
|
||||||
|
[source,js]
|
||||||
|
--------------------------------------------------
|
||||||
|
PUT _ilm/policy/my_policy
|
||||||
|
{
|
||||||
|
"policy": {
|
||||||
|
"phases": {
|
||||||
|
"hot": {
|
||||||
|
"actions": {
|
||||||
|
"rollover" : {
|
||||||
|
"max_size": "100GB"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
--------------------------------------------------
|
||||||
|
// CONSOLE
|
||||||
|
|
||||||
|
===== Example: Rollover when index has too many documents
|
||||||
|
|
||||||
|
This example rolls the index over when it contains at least
|
||||||
|
1000000 documents.
|
||||||
|
|
||||||
|
[source,js]
|
||||||
|
--------------------------------------------------
|
||||||
|
PUT _ilm/policy/my_policy
|
||||||
|
{
|
||||||
|
"policy": {
|
||||||
|
"phases": {
|
||||||
|
"hot": {
|
||||||
|
"actions": {
|
||||||
|
"rollover" : {
|
||||||
|
"max_docs": 1000000
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
--------------------------------------------------
|
||||||
|
// CONSOLE
|
||||||
|
|
||||||
|
===== Example: Rollover when index is too old
|
||||||
|
|
||||||
|
This example rolls the index over when it has been created at least
|
||||||
|
7 days ago.
|
||||||
|
|
||||||
|
[source,js]
|
||||||
|
--------------------------------------------------
|
||||||
|
PUT _ilm/policy/my_policy
|
||||||
|
{
|
||||||
|
"policy": {
|
||||||
|
"phases": {
|
||||||
|
"hot": {
|
||||||
|
"actions": {
|
||||||
|
"rollover" : {
|
||||||
|
"max_age": "7d"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
--------------------------------------------------
|
||||||
|
// CONSOLE
|
||||||
|
|
||||||
|
===== Example: Rollover when index is too old or too large
|
||||||
|
|
||||||
|
This example rolls the index over when it has been created at least
|
||||||
|
7 days ago or it is at least 100 gigabytes. In this case, the index will be
|
||||||
|
rolled over when any of the conditions is met.
|
||||||
|
|
||||||
|
[source,js]
|
||||||
|
--------------------------------------------------
|
||||||
|
PUT _ilm/policy/my_policy
|
||||||
|
{
|
||||||
|
"policy": {
|
||||||
|
"phases": {
|
||||||
|
"hot": {
|
||||||
|
"actions": {
|
||||||
|
"rollover" : {
|
||||||
|
"max_age": "7d",
|
||||||
|
"max_size": "100GB"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
--------------------------------------------------
|
||||||
|
// CONSOLE
|
||||||
|
|
||||||
|
|
||||||
|
[[ilm-shrink-action]]
|
||||||
|
==== Shrink
|
||||||
|
|
||||||
|
NOTE: Index will be be made read-only when this action is run
|
||||||
|
(see: <<dynamic-index-settings,index.blocks.write>>)
|
||||||
|
|
||||||
|
This action shrinks an existing index into a new index with fewer primary
|
||||||
|
shards. It calls the <<indices-shrink-index,Shrink API>> to shrink the index.
|
||||||
|
Since allocating all the primary shards of the index to one node is a
|
||||||
|
prerequisite, this action will first allocate the primary shards to a valid
|
||||||
|
node. After shrinking, it will swap aliases pointing to the original index
|
||||||
|
into the new shrunken index. The new index will also have a new name:
|
||||||
|
"shrink-<origin-index-name>". So if the original index was called "logs",
|
||||||
|
then the new index will be named "shrink-logs".
|
||||||
|
|
||||||
|
[[ilm-shrink-options]]
|
||||||
|
.Shrink Options
|
||||||
|
[options="header"]
|
||||||
|
|======
|
||||||
|
| Name | Required | Default | Description
|
||||||
|
| `number_of_shards` | yes | - | The number of shards
|
||||||
|
to shrink to. must be
|
||||||
|
a factor of the number
|
||||||
|
of shards in the
|
||||||
|
source index.
|
||||||
|
|======
|
||||||
|
|
||||||
|
[source,js]
|
||||||
|
--------------------------------------------------
|
||||||
|
PUT _ilm/policy/my_policy
|
||||||
|
{
|
||||||
|
"policy": {
|
||||||
|
"phases": {
|
||||||
|
"warm": {
|
||||||
|
"actions": {
|
||||||
|
"shrink" : {
|
||||||
|
"number_of_shards": 1
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
--------------------------------------------------
|
||||||
|
// CONSOLE
|
||||||
|
|
||||||
|
|
||||||
|
=== Full Policy
|
||||||
|
|
||||||
|
With all of these actions, we can support complex management strategies for our
|
||||||
|
indices. This policy will define an index that will start in the hot phase,
|
||||||
|
rolling over every 20g or 7 days. After 30 days it enters the warm phase
|
||||||
|
and increases the replicas to 2, force merges and shrinks. After 60 days
|
||||||
|
it enters the cold phase and allocates to "cold" nodes, and after 90 days the
|
||||||
|
index is deleted.
|
||||||
|
|
||||||
|
[source,js]
|
||||||
|
--------------------------------------------------
|
||||||
|
PUT _ilm/policy/full_policy
|
||||||
|
{
|
||||||
|
"policy": {
|
||||||
|
"phases": {
|
||||||
|
"hot": {
|
||||||
|
"actions": {
|
||||||
|
"rollover": {
|
||||||
|
"max_age": "7d",
|
||||||
|
"max_size": "20G"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"warm": {
|
||||||
|
"min_age": "30d",
|
||||||
|
"actions": {
|
||||||
|
"forcemerge": {
|
||||||
|
"max_num_segments": 1
|
||||||
|
},
|
||||||
|
"shrink": {
|
||||||
|
"number_of_shards": 1
|
||||||
|
},
|
||||||
|
"allocate": {
|
||||||
|
"number_of_replicas": 2
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"cold": {
|
||||||
|
"min_age": "60d",
|
||||||
|
"actions": {
|
||||||
|
"allocate": {
|
||||||
|
"require": {
|
||||||
|
"type": "cold"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"delete": {
|
||||||
|
"min_age": "90d",
|
||||||
|
"actions": {
|
||||||
|
"delete": {}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
--------------------------------------------------
|
||||||
|
// CONSOLE
|
||||||
Loading…
Reference in New Issue