mirror of
https://github.com/honeymoose/OpenSearch.git
synced 2025-02-06 13:08:29 +00:00
49bd8715ff
We inject an Unfollow action before Shrink because the Shrink action cannot be safely used on a following index, as it may not be fully caught up with the leader index before the "original" following index is deleted and replaced with a non-following Shrunken index. The Unfollow action will verify that 1) the index is marked as "complete", and 2) all operations up to this point have been replicated from the leader to the follower before explicitly disconnecting the follower from the leader. Injecting an Unfollow action before the Rollover action is done mainly as a convenience: This allow users to use the same lifecycle policy on both the leader and follower cluster without having to explictly modify the policy to unfollow the index, while doing what we expect users to want in most cases.
157 lines
6.1 KiB
Plaintext
157 lines
6.1 KiB
Plaintext
[role="xpack"]
|
|
[testenv="basic"]
|
|
[[using-policies-rollover]]
|
|
== Using policies to manage index rollover
|
|
|
|
The rollover action enables you to automatically roll over to a new index based
|
|
on the index size, document count, or age. When a rollover is triggered, a new
|
|
index is created, the write alias is updated to point to the new index, and all
|
|
subsequent updates are written to the new index.
|
|
|
|
Rolling over to a new index based on size, document count, or age is preferable
|
|
to time-based rollovers. Rolling over at an arbitrary time often results in
|
|
many small indices, which can have a negative impact on performance and
|
|
resource usage.
|
|
|
|
You control when the rollover action is triggered by specifying one or more
|
|
rollover parameters. The rollover is performed once any of the criteria are
|
|
met. Because the criteria are checked periodically, the index might grow
|
|
slightly beyond the specified threshold. To control how often the criteria are
|
|
checked, specify the `indices.lifecycle.poll_interval` cluster setting.
|
|
|
|
IMPORTANT: New indices created via rollover will not automatically inherit the
|
|
policy used by the old index, and will not use any policy by default. Therefore,
|
|
it is highly recommended to apply the policy via
|
|
<<applying-policy-to-template,index template>>, including a Rollover alias
|
|
setting, for your indices which specifies the policy you wish to use for each
|
|
new index.
|
|
|
|
The rollover action takes the following parameters:
|
|
|
|
.`rollover` Action Parameters
|
|
|===
|
|
|Name |Description
|
|
|max_size |The maximum estimated size the index is allowed to grow
|
|
to. Defaults to `null`. Optional.
|
|
|max_docs |The maximum number of document the index should
|
|
contain. Defaults to `null`. Optional.
|
|
|max_age |The maximum age of the index. Defaults to `null`. Optional.
|
|
|===
|
|
|
|
These parameters are used to determine when the index is considered "full" and
|
|
a rollover should be performed. Where multiple criteria are defined the
|
|
rollover operation will be performed once any of the criteria are met.
|
|
|
|
The following request defines a policy with a rollover action that triggers
|
|
when the index size reaches 25GB. The old index is subsequently deleted after
|
|
30 days.
|
|
|
|
NOTE: Once an index rolls over, {ilm} uses the timestamp of the rollover
|
|
operation rather than the index creation time to evaluate when to move the
|
|
index to the next phase. For indices that have rolled over, the `min_age`
|
|
criteria specified for a phase is relative to the rollover time for indices. In
|
|
this example, that means the index will be deleted 30 days after rollover, not
|
|
30 days from when the index was created.
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
PUT /_ilm/policy/my_policy
|
|
{
|
|
"policy": {
|
|
"phases": {
|
|
"hot": {
|
|
"actions": {
|
|
"rollover": {
|
|
"max_size": "25GB"
|
|
}
|
|
}
|
|
},
|
|
"delete": {
|
|
"min_age": "30d",
|
|
"actions": {
|
|
"delete": {}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
|
|
To use an {ilm} policy, you need to specify it in the index template used to
|
|
create the indices. For example, the following template associates `my_policy`
|
|
with indices created from the template `my_template`.
|
|
|
|
[source,js]
|
|
-----------------------
|
|
PUT _template/my_template
|
|
{
|
|
"index_patterns": ["test-*"], <1>
|
|
"settings": {
|
|
"number_of_shards": 1,
|
|
"number_of_replicas": 1,
|
|
"index.lifecycle.name": "my_policy", <2>
|
|
"index.lifecycle.rollover_alias": "test-alias" <3>
|
|
}
|
|
}
|
|
-----------------------
|
|
// CONSOLE
|
|
<1> Template applies to all indices with the prefix test-
|
|
<2> Associates my_policy with all indices created with this template
|
|
<3> Rolls over the write alias test when the rollover action is triggered
|
|
|
|
To be able to start using the policy for these `test-*` indexes we need to
|
|
bootstrap the process by creating the first index.
|
|
|
|
[source,js]
|
|
-----------------------
|
|
PUT test-000001 <1>
|
|
{
|
|
"aliases": {
|
|
"test-alias":{
|
|
"is_write_index": true <2>
|
|
}
|
|
}
|
|
}
|
|
-----------------------
|
|
// CONSOLE
|
|
<1> Creates the index called test-000001. The rollover action increments the
|
|
suffix number for each subsequent index.
|
|
<2> Designates this index as the write index for this alias.
|
|
|
|
When the rollover is performed, the newly-created index is set as the write
|
|
index for the rolled over alias. Documents sent to the alias are indexed into
|
|
the new index, enabling indexing to continue uninterrupted.
|
|
|
|
[[skipping-rollover]]
|
|
=== Skipping Rollover
|
|
|
|
The `index.lifecycle.indexing_complete` setting indicates to {ilm} whether this
|
|
index has already been rolled over. If it is set to `true`, that indicates that
|
|
this index has already been rolled over and does not need to be rolled over
|
|
again. Therefore, {ilm} will skip any Rollover Action configured in the
|
|
associated lifecycle policy for this index. This is useful if you need to make
|
|
an exception to your normal Lifecycle Policy and switching the alias to a
|
|
different index by hand, but do not want to remove the index from {ilm}
|
|
completely.
|
|
|
|
This setting is set to `true` automatically by ILM upon the successful
|
|
completion of a Rollover Action. However, it will be removed if
|
|
<<ilm-remove-policy,the policy is removed>> from the index.
|
|
|
|
IMPORTANT: If `index.lifecycle.indexing_complete` is set to `true` on an index,
|
|
it will not be rolled over by {ilm}, but {ilm} will verify that this index is no
|
|
longer the write index for the alias specified by
|
|
`index.lifecycle.rollover_alias`. If that setting is missing, or if the index is
|
|
still the write index for that alias, this index will be moved to the
|
|
<<index-lifecycle-error-handling,error step>>.
|
|
|
|
For example, if you wish to change the name of new indices while retaining
|
|
previous data in accordance with your configured policy, you can create the
|
|
template for the new index name pattern and the first index with the new name
|
|
manually, change the write index of the alias using the <<indices-aliases, Index
|
|
Aliases API>>, and set `index.lifecycle.indexing_complete` to `true` on the old
|
|
index to indicate that it does not need to be rolled over. This way, {ilm} will
|
|
continue to manage the old index in accordance with its existing policy, as well
|
|
as the new one, with no interruption.
|