[ILM] add docs explaining high-level concepts (#35322)
add docs explaining high-level concepts
This commit is contained in:
Tal Levy
GitHub
parent
803eccec11
commit
30a713cc3f
|
|
@ -52,10 +52,12 @@ hardware.
|
|||
|
||||
include::getting-started-ilm.asciidoc[]
|
||||
|
||||
include::using-policies-rollover.asciidoc[]
|
||||
include::policy-definitions.asciidoc[]
|
||||
|
||||
include::set-up-lifecycle-policy.asciidoc[]
|
||||
|
||||
include::using-policies-rollover.asciidoc[]
|
||||
|
||||
include::update-lifecycle-policy.asciidoc[]
|
||||
|
||||
include::get-index-lifecycle-information.asciidoc[]
|
||||
|
|
|
|||
|
|
@ -0,0 +1,549 @@
|
|||
[role="xpack"]
|
||||
[testenv="basic"]
|
||||
[[ilm-policy-definition]]
|
||||
== Policy Phases and Actions
|
||||
|
||||
There are four stages in the index lifecycle, in the order
|
||||
they are executed.
|
||||
|
||||
[options="header"]
|
||||
|======
|
||||
| Name | Description
|
||||
| `hot` | The index is actively being written to
|
||||
| `warm` | The index is generally not being written to, but is still queried
|
||||
| `cold` | The index is no longer being updated and is seldom queried. The
|
||||
information still needs to be searchable, but it's okay if those queries are
|
||||
slower.
|
||||
| `delete` | The index is no longer needed and can safely be deleted
|
||||
|======
|
||||
|
||||
Each of these stages is called a "phase". A policy does not need to configure
|
||||
each phase for an index. For example, one policy may define only the hot
|
||||
phase and the delete phase, while another may define all four phases.
|
||||
|
||||
=== Timing
|
||||
|
||||
Indices enter phases based on a phase's `min_age` parameter.
|
||||
The index will not enter the phase until the index's age is older than that
|
||||
of the `min_age`. The parameter is configured using a time
|
||||
duration format (see <<time-units, Time Units>>).
|
||||
|
||||
`min_age` defaults to zero seconds `0s` for each phase if not specified.
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
PUT _ilm/policy/my_policy
|
||||
{
|
||||
"policy": {
|
||||
"phases": {
|
||||
"warm": {
|
||||
"min_age": "1d",
|
||||
"actions": {
|
||||
"allocate": {
|
||||
"number_of_replicas": 1
|
||||
}
|
||||
}
|
||||
},
|
||||
"delete": {
|
||||
"min_age": "30d",
|
||||
"actions": {
|
||||
"delete": {}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
--------------------------------------------------
|
||||
// CONSOLE
|
||||
|
||||
The Above example configures a policy that moves the index into the warm
|
||||
phase after one day. Until then, the index is in a waiting state. After
|
||||
moving into the warm phase, it will wait until 30 days have elapsed before
|
||||
moving to the delete phase and deleting the index.
|
||||
|
||||
`min_age` is usually the time elapsed from the time the index is created. If the
|
||||
index is rolled over, then `min_age` is the time elapsed from the time the index
|
||||
is rolled over. The intention here is to execute following phases and actions
|
||||
relative to when data was written last to a rolled over index.
|
||||
|
||||
=== Actions
|
||||
|
||||
The below list shows the actions which are available in each phase.
|
||||
|
||||
* Hot
|
||||
- <<ilm-rollover-action,Rollover>>
|
||||
* Warm
|
||||
- <<ilm-allocate-action,Allocate>>
|
||||
- <<ilm-readonly-action,Read-Only>>
|
||||
- <<ilm-forcemerge-action,Force Merge>>
|
||||
- <<ilm-shrink-action,Shrink>>
|
||||
* Cold
|
||||
- <<ilm-allocate-action,Allocate>>
|
||||
* Delete
|
||||
- <<ilm-delete-action,Delete>>
|
||||
|
||||
[[ilm-allocate-action]]
|
||||
==== Allocate
|
||||
|
||||
Phases allowed: warm, cold.
|
||||
|
||||
The Allocate action allows you to specify which nodes are allowed to host the
|
||||
shards of the index and set the number of replicas.
|
||||
Behind the scenes, it is modifying the index settings
|
||||
for shard filtering and/or replica counts. When updating the number of replicas,
|
||||
configuring allocation rules is optional. When configuring allocation rules,
|
||||
setting number of replicas is optional. Although this action can be treated as
|
||||
two separate index settings updates, both can be configured at once.
|
||||
|
||||
Read more about index replicas <<getting-started-shards-and-replicas,here>>.
|
||||
Read more about shard allocation filtering in
|
||||
the <<shard-allocation-filtering,Shard allocation filtering documentation>>.
|
||||
|
||||
[[ilm-allocate-options]]
|
||||
.Allocate Options
|
||||
[options="header"]
|
||||
|======
|
||||
| Name | Required | Default | Description
|
||||
| `number_of_replicas` | no | - | The number of replicas to
|
||||
assign to the index
|
||||
| `include` | no | - | assigns an index to nodes
|
||||
having at least _one_ of the attributes
|
||||
| `exclude` | no | - | assigns an index to nodes having
|
||||
_none_ of the attributes
|
||||
| `require` | no | - | assigns an index to nodes having
|
||||
_all_ of the attributes
|
||||
|======
|
||||
|
||||
If `number_of_replicas` is not configured, then at least one of `include`,
|
||||
`exclude`, and `require` is required. An empty Allocate Action with no configuration
|
||||
is invalid.
|
||||
|
||||
===== Example: Change number of replicas
|
||||
|
||||
In this example, the index's number of replicas is changed to `2`, while allocation
|
||||
rules are unchanged.
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
PUT _ilm/policy/my_policy
|
||||
{
|
||||
"policy": {
|
||||
"phases": {
|
||||
"warm": {
|
||||
"actions": {
|
||||
"allocate" : {
|
||||
"number_of_replicas" : 2
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
--------------------------------------------------
|
||||
// CONSOLE
|
||||
|
||||
===== Example: Assign index to node with specific "box_type" attribute
|
||||
|
||||
This example assigns the index to nodes with `box_type` attribute of "hot" or "warm".
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
PUT _ilm/policy/my_policy
|
||||
{
|
||||
"policy": {
|
||||
"phases": {
|
||||
"warm": {
|
||||
"actions": {
|
||||
"allocate" : {
|
||||
"include" : {
|
||||
"box_type": "hot,warm"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
--------------------------------------------------
|
||||
// CONSOLE
|
||||
|
||||
===== Example: Assign index to a specific node and update replica settings
|
||||
|
||||
This example updates the index to have one replica per shard and be allocated
|
||||
to nodes with a `box_type` attribute of "cold".
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
PUT _ilm/policy/my_policy
|
||||
{
|
||||
"policy": {
|
||||
"phases": {
|
||||
"warm": {
|
||||
"actions": {
|
||||
"allocate" : {
|
||||
"number_of_replicas": 1,
|
||||
"require" : {
|
||||
"box_type": "cold"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
--------------------------------------------------
|
||||
// CONSOLE
|
||||
|
||||
[[ilm-delete-action]]
|
||||
==== Delete
|
||||
|
||||
Phases allowed: delete.
|
||||
|
||||
The Delete Action does just that, it deletes the index.
|
||||
|
||||
This action does not have any options associated with it.
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
PUT _ilm/policy/my_policy
|
||||
{
|
||||
"policy": {
|
||||
"phases": {
|
||||
"delete": {
|
||||
"actions": {
|
||||
"delete" : { }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
--------------------------------------------------
|
||||
// CONSOLE
|
||||
|
||||
[[ilm-forcemerge-action]]
|
||||
==== Force Merge
|
||||
|
||||
Phases allowed: warm.
|
||||
|
||||
NOTE: Index will be be made read-only when this action is run
|
||||
(see: <<dynamic-index-settings,index.blocks.write>>)
|
||||
|
||||
The Force Merge Action <<indices-forcemerge,force merges>> the index into at
|
||||
most a specific number of <<indices-segments,segments>>.
|
||||
|
||||
[[ilm-forcemerge-options]]
|
||||
.Force Merge Options
|
||||
[options="header"]
|
||||
|======
|
||||
| Name | Required | Default | Description
|
||||
| `max_num_segments` | yes | - | The number of
|
||||
segments to merge to.
|
||||
To fully merge the
|
||||
index, set it to `1`
|
||||
|======
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
PUT _ilm/policy/my_policy
|
||||
{
|
||||
"policy": {
|
||||
"phases": {
|
||||
"warm": {
|
||||
"actions": {
|
||||
"forcemerge" : {
|
||||
"max_num_segments": 1
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
--------------------------------------------------
|
||||
// CONSOLE
|
||||
|
||||
[[ilm-readonly-action]]
|
||||
==== Read-Only
|
||||
|
||||
Phases allowed: warm.
|
||||
|
||||
This action will set the index to be read-only
|
||||
(see: <<dynamic-index-settings,index.blocks.write>>)
|
||||
|
||||
This action does not have any options associated with it.
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
PUT _ilm/policy/my_policy
|
||||
{
|
||||
"policy": {
|
||||
"phases": {
|
||||
"warm": {
|
||||
"actions": {
|
||||
"readonly" : { }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
--------------------------------------------------
|
||||
// CONSOLE
|
||||
|
||||
[[ilm-rollover-action]]
|
||||
==== Rollover
|
||||
|
||||
Phases allowed: hot.
|
||||
|
||||
[WARNING]
|
||||
index format must match pattern '^.*-\\d+$', for example (`logs-000001`).
|
||||
[WARNING]
|
||||
The managed index must set `index.lifecycle.rollover_alias` as the
|
||||
alias to rollover. The index must also be the write index for the alias.
|
||||
|
||||
For example, if an index to be managed has an alias `my_data`. The managed
|
||||
index "my_index" must be the write index for the alias. For more information, read
|
||||
<<indices-rollover-is-write-index,Write Index Alias Behavior>>.
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
PUT my_index
|
||||
{
|
||||
"settings": {
|
||||
"index.lifecycle.name": "my_policy",
|
||||
"index.lifecycle.rollover_alias": "my_data"
|
||||
},
|
||||
"aliases": {
|
||||
"my_data": {
|
||||
"is_write_index": true
|
||||
}
|
||||
}
|
||||
}
|
||||
--------------------------------------------------
|
||||
// CONSOLE
|
||||
|
||||
The Rollover Action rolls an alias over to a new index when the
|
||||
existing index meets one of the rollover conditions.
|
||||
|
||||
|
||||
[[ilm-rollover-options]]
|
||||
.Rollover Options
|
||||
[options="header"]
|
||||
|======
|
||||
| Name | Required | Default | Description
|
||||
| `max_size` | no | - | max index storage size.
|
||||
See <<byte-units, Byte Units>>
|
||||
for formatting
|
||||
| `max_docs` | no | - | max number of documents an
|
||||
index is to contain before
|
||||
rolling over.
|
||||
| `max_age` | no | - | max time elapsed from index
|
||||
creation. See
|
||||
<<time-units, Time Units>>
|
||||
for formatting
|
||||
|======
|
||||
|
||||
At least one of `max_size`, `max_docs`, `max_age` or any combinations of the
|
||||
three are required to be specified.
|
||||
|
||||
===== Example: Rollover when index is too large
|
||||
|
||||
This example rolls the index over when it is at least 100 gigabytes.
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
PUT _ilm/policy/my_policy
|
||||
{
|
||||
"policy": {
|
||||
"phases": {
|
||||
"hot": {
|
||||
"actions": {
|
||||
"rollover" : {
|
||||
"max_size": "100GB"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
--------------------------------------------------
|
||||
// CONSOLE
|
||||
|
||||
===== Example: Rollover when index has too many documents
|
||||
|
||||
This example rolls the index over when it contains at least
|
||||
1000000 documents.
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
PUT _ilm/policy/my_policy
|
||||
{
|
||||
"policy": {
|
||||
"phases": {
|
||||
"hot": {
|
||||
"actions": {
|
||||
"rollover" : {
|
||||
"max_docs": 1000000
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
--------------------------------------------------
|
||||
// CONSOLE
|
||||
|
||||
===== Example: Rollover when index is too old
|
||||
|
||||
This example rolls the index over when it has been created at least
|
||||
7 days ago.
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
PUT _ilm/policy/my_policy
|
||||
{
|
||||
"policy": {
|
||||
"phases": {
|
||||
"hot": {
|
||||
"actions": {
|
||||
"rollover" : {
|
||||
"max_age": "7d"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
--------------------------------------------------
|
||||
// CONSOLE
|
||||
|
||||
===== Example: Rollover when index is too old or too large
|
||||
|
||||
This example rolls the index over when it has been created at least
|
||||
7 days ago or it is at least 100 gigabytes. In this case, the index will be
|
||||
rolled over when any of the conditions is met.
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
PUT _ilm/policy/my_policy
|
||||
{
|
||||
"policy": {
|
||||
"phases": {
|
||||
"hot": {
|
||||
"actions": {
|
||||
"rollover" : {
|
||||
"max_age": "7d",
|
||||
"max_size": "100GB"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
--------------------------------------------------
|
||||
// CONSOLE
|
||||
|
||||
|
||||
[[ilm-shrink-action]]
|
||||
==== Shrink
|
||||
|
||||
NOTE: Index will be be made read-only when this action is run
|
||||
(see: <<dynamic-index-settings,index.blocks.write>>)
|
||||
|
||||
This action shrinks an existing index into a new index with fewer primary
|
||||
shards. It calls the <<indices-shrink-index,Shrink API>> to shrink the index.
|
||||
Since allocating all the primary shards of the index to one node is a
|
||||
prerequisite, this action will first allocate the primary shards to a valid
|
||||
node. After shrinking, it will swap aliases pointing to the original index
|
||||
into the new shrunken index. The new index will also have a new name:
|
||||
"shrink-<origin-index-name>". So if the original index was called "logs",
|
||||
then the new index will be named "shrink-logs".
|
||||
|
||||
[[ilm-shrink-options]]
|
||||
.Shrink Options
|
||||
[options="header"]
|
||||
|======
|
||||
| Name | Required | Default | Description
|
||||
| `number_of_shards` | yes | - | The number of shards
|
||||
to shrink to. must be
|
||||
a factor of the number
|
||||
of shards in the
|
||||
source index.
|
||||
|======
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
PUT _ilm/policy/my_policy
|
||||
{
|
||||
"policy": {
|
||||
"phases": {
|
||||
"warm": {
|
||||
"actions": {
|
||||
"shrink" : {
|
||||
"number_of_shards": 1
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
--------------------------------------------------
|
||||
// CONSOLE
|
||||
|
||||
|
||||
=== Full Policy
|
||||
|
||||
With all of these actions, we can support complex management strategies for our
|
||||
indices. This policy will define an index that will start in the hot phase,
|
||||
rolling over every 20g or 7 days. After 30 days it enters the warm phase
|
||||
and increases the replicas to 2, force merges and shrinks. After 60 days
|
||||
it enters the cold phase and allocates to "cold" nodes, and after 90 days the
|
||||
index is deleted.
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
PUT _ilm/policy/full_policy
|
||||
{
|
||||
"policy": {
|
||||
"phases": {
|
||||
"hot": {
|
||||
"actions": {
|
||||
"rollover": {
|
||||
"max_age": "7d",
|
||||
"max_size": "20G"
|
||||
}
|
||||
}
|
||||
},
|
||||
"warm": {
|
||||
"min_age": "30d",
|
||||
"actions": {
|
||||
"forcemerge": {
|
||||
"max_num_segments": 1
|
||||
},
|
||||
"shrink": {
|
||||
"number_of_shards": 1
|
||||
},
|
||||
"allocate": {
|
||||
"number_of_replicas": 2
|
||||
}
|
||||
}
|
||||
},
|
||||
"cold": {
|
||||
"min_age": "60d",
|
||||
"actions": {
|
||||
"allocate": {
|
||||
"require": {
|
||||
"type": "cold"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"delete": {
|
||||
"min_age": "90d",
|
||||
"actions": {
|
||||
"delete": {}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
--------------------------------------------------
|
||||
// CONSOLE
|
||||
Loading…
Reference in New Issue