diff --git a/docs/reference/glossary.asciidoc b/docs/reference/glossary.asciidoc index d5ace4e654c..7ec861ccc72 100644 --- a/docs/reference/glossary.asciidoc +++ b/docs/reference/glossary.asciidoc @@ -268,7 +268,6 @@ shard will never be started on the same node as its primary shard. -- // tag::rollover-def[] // tag::rollover-def-short[] - Creates a new index for a rollover target when the existing index reaches a certain size, number of docs, or age. A rollover target can be either an <> or a <>. // end::rollover-def-short[] diff --git a/docs/reference/ilm/example-index-lifecycle-policy.asciidoc b/docs/reference/ilm/example-index-lifecycle-policy.asciidoc new file mode 100644 index 00000000000..b6ec8367028 --- /dev/null +++ b/docs/reference/ilm/example-index-lifecycle-policy.asciidoc @@ -0,0 +1,165 @@ +[role="xpack"] + +[[example-using-index-lifecycle-policy]] +=== Tutorial: Manage {filebeat} time-based indices +++++ +Manage {filebeat} time-based indices +++++ + +With {ilm} ({ilm-init}), you can create policies that perform actions automatically +on indices as they age and grow. {ilm-init} policies help you to manage +performance, resilience, and retention of your data during its lifecycle. This tutorial shows +you how to use {kib}’s *Index Lifecycle Policies* to modify and create {ilm-init} +policies. You can learn more about all of the actions, benefits, and lifecycle +phases in the <>. + + +[discrete] +[[example-using-index-lifecycle-policy-scenario]] +==== Scenario + +You’re tasked with sending syslog files to an {es} cluster. This +log data has the following data retention guidelines: + +* Keep logs on hot data nodes for 30 days +* Roll over to a new index if the size reaches 50GB +* After 30 days: +** Move the logs to warm data nodes +** Set <> to 1 +** <> multiple index segments to free up the space used by deleted documents +* Delete logs after 90 days + + +[discrete] +[[example-using-index-lifecycle-policy-prerequisites]] +==== Prerequisites + +To complete this tutorial, you'll need: + +* An {es} cluster with hot and warm nodes configured for shard allocation +awareness. + +** {ess}: +Choose the {cloud}/ec-getting-started-templates-hot-warm.html[hot-warm architecture] deployment template. + +** Self-managed cluster: +Add node attributes as described for {ref}/shard-allocation-filtering.html[shard allocation filtering]. ++ +For example, you can set this in your `elasticsearch.yml` for each data node: ++ +[source,yaml] +-------------------------------------------------------------------------------- +node.attr.data: "warm" +-------------------------------------------------------------------------------- + +* A server with {filebeat} installed and configured to send logs to the `elasticsearch` +output as described in {filebeat-ref}/filebeat-getting-started.html[Getting Started with {filebeat}]. + +[discrete] +[[example-using-index-lifecycle-policy-view-fb-ilm-policy]] +==== View the {filebeat} {ilm-init} policy + +{filebeat} includes a default {ilm-init} policy that enables rollover. {ilm-init} +is enabled automatically if you’re using the default `filebeat.yml` and index template. + +To view the default policy in {kib}: + +. Go to Management and select *Index Lifecycle Policies*. +. Search for _filebeat_ +. Select the _filebeat-version_ policy. + +This policy initiates the rollover action when the index size reaches 50GB or +becomes 30 days old. + +[role="screenshot"] +image::images/ilm/tutorial-ilm-hotphaserollover-default.png["Default policy"] + + +[discrete] +==== Modify the policy + +The default policy is enough to prevent the creation of many tiny daily indices. +You can modify the policy to meet more complex requirements. + +. Activate the warm phase. ++ +-- +[role="screenshot"] +image::images/ilm/tutorial-ilm-modify-default-warm-phase-rollover.png["Modify to add warm phase"] + +.. Set one of the following options to control when the index moves to the warm phase: + +*** Provide a value for *Timing for warm phase*. Setting this to *15* keeps the +indices on hot nodes for a range of 15-45 days, depending on when the initial +rollover occurred. + +*** Enable *Move to warm phase on rollover*. The index might move to the warm phase +more quickly than intended if it reaches the *Maximum index size* before the +the *Maximum age*. + +.. In the *Select a node attribute to control shard allocation* dropdown, select +*data:warm(2)* to migrate shards to warm data nodes. + +.. Change *Number of replicas* to *1*. + +.. Enable *Force merge data* and set *Number of segments* to *1*. + +NOTE: When rollover is enabled in the hot phase, action timing in the other phases +is based on the rollover date. +-- + +. Activate the delete phase and set *Timing for delete phase* to *90* days. ++ +[role="screenshot"] +image::images/ilm/tutorial-ilm-delete-rollover.png["Add a delete phase"] + +[discrete] +==== Create a custom policy + +If meeting a specific retention time period is most important, you can create a +custom policy. For this option, you use {filebeat} daily indices without +rollover. + +To create a custom policy: + +. Go to Management and select *Index Lifecycle Policies*. +. Click *Create policy*. +. Activate the warm phase and configure it as follows: ++ +-- +**Timing for warm phase**: 30 days from index creation + +**Node attribute**: `data:warm` + +**Number of replicas**: 1 + +**Force merge data**: enable + +**Number of segments**: 1 + +[role="screenshot"] +image::images/ilm/tutorial-ilm-custom-policy.png["Modify the custom policy to add a warm phase"] +-- + +. Activate the delete phase and set the timing to 90 days. ++ +[role="screenshot"] +image::images/ilm/tutorial-ilm-delete-phase-creation.png["Delete phase"] + +To configure the index to use the new policy: + +. Go to Management and select *Index Lifecycle Policies*. +. Find your {ilm-init} policy and click its *Actions* link. +. Choose *Add policy to index template*. +. Select your {filebeat} index template name from the *Index template* list. For example, `filebeat-7.5.x`. +. Click *Add Policy* to save the changes. ++ +NOTE: If you initially used the default {filebeat} {ilm-init} policy, you will +see a notice that the template already has a policy associated with it. Confirm +that you want to overwrite that configuration. + +When you change the policy associated with the index template, the active +index will continue to use the policy it was associated with at index creation +unless you manually update it. The next new index will use the updated policy. +For more reasons that your {ilm-init} policy changes might be delayed, see +<>. diff --git a/docs/reference/ilm/ilm-overview.asciidoc b/docs/reference/ilm/ilm-overview.asciidoc index 94c03fb02ad..037b95e61dc 100644 --- a/docs/reference/ilm/ilm-overview.asciidoc +++ b/docs/reference/ilm/ilm-overview.asciidoc @@ -9,6 +9,7 @@ You can create and apply {ilm-cap} ({ilm-init}) policies to automatically manage your indices according to your performance, resiliency, and retention requirements. + Index lifecycle policies can trigger actions such as: * **Rollover**: @@ -39,9 +40,9 @@ For example, if you are indexing metrics data from a fleet of ATMs into Elasticsearch, you might define a policy that says: . When the index reaches 50GB, roll over to a new index. -. Move the old index into the warm stage, mark it read only, and shrink it down +. Move the old index into the warm phase, mark it read only, and shrink it down to a single shard. -. After 7 days, move the index into the cold stage and move it to less expensive +. After 7 days, move the index into the cold phase and move it to less expensive hardware. . Delete the index once the required 30 day retention period is reached. diff --git a/docs/reference/ilm/ilm-tutorial.asciidoc b/docs/reference/ilm/ilm-tutorial.asciidoc index e70f25bc72a..b02337e3f84 100644 --- a/docs/reference/ilm/ilm-tutorial.asciidoc +++ b/docs/reference/ilm/ilm-tutorial.asciidoc @@ -7,9 +7,6 @@ Automate rollover ++++ -This tutorial demonstrates how to use {ilm} -({ilm-init}) to manage indices that contain time-series data. - When you continuously index timestamped documents into {es}, you typically use a <> so you can periodically roll over to a new index. @@ -51,9 +48,6 @@ A lifecycle policy specifies the phases in the index lifecycle and the actions to perform in each phase. A lifecycle can have up to four phases: `hot`, `warm`, `cold`, and `delete`. -You can define and manage policies through {kib} Management or with the -<> API. - For example, you might define a `timeseries_policy` that has two phases: * A `hot` phase that defines a rollover action to specify that an index rolls over when it @@ -61,8 +55,16 @@ reaches either a `max_size` of 50 gigabytes or a `max_age` of 30 days. * A `delete` phase that sets `min_age` to remove the index 90 days after rollover. Note that this value is relative to the rollover time, not the index creation time. -The underlying put policy request looks like this: +You can create the policy through {kib} Management or with the +<> API. +To create the policy from {kib}, go to Management and click **Index Lifecycle Policies**. +[role="screenshot"] +image:images/ilm/create-policy.png[] + +.API example +[%collapsible] +==== [source,console] ------------------------ PUT _ilm/policy/timeseries_policy @@ -91,10 +93,7 @@ PUT _ilm/policy/timeseries_policy <2> Trigger the `rollover` action when either of the conditions are met. <3> Move the index into the `delete` phase 90 days after rollover. <4> Trigger the `delete` action when the index enters the delete phase. - -You can also invoke this API directly to add lifecycle policies. - -For the complete list of actions that {ilm} can perform, see <>. +==== [discrete] [[ilm-gs-apply-policy]] @@ -114,8 +113,9 @@ You can use the {kib} Create template wizard to add the template. This wizard invokes the put _index_template API to create the <> with the options you specify. -The underlying request looks like this: - +.API example +[%collapsible] +==== [source,console] ----------------------- PUT _index_template/timeseries_template @@ -148,8 +148,7 @@ in all documents indexed into the `timeseries` data stream. <3> The name of the {ilm-init} policy used to manage the data stream. <4> A <> or <> field mapping for the timestamp field specified in the `timestamp_field` property - -You can also invoke this API directly to add templates. +==== [discrete] [[ilm-gs-create-the-data-stream]] @@ -317,10 +316,14 @@ that match the index pattern. * `index.lifecycle.rollover_alias` specifies the index alias to be rolled over when the rollover action is triggered for an index. -You can use the {kib} Create template wizard to add the template. -This wizard invokes the put template API to create the template with the options you specify. +You can use the {kib} Create template wizard to add the template. +To access the wizard, go to Management, click **Index Management**, +and select the **Index Templates** view. -The underlying request looks like this: +[role="screenshot"] +image:images/ilm/create-template-wizard.png[] + +The create template request for the example template looks like this: [source,console] ----------------------- @@ -342,9 +345,6 @@ PUT _template/timeseries_template <3> The name of the alias used to reference these indices. Required for policies that use the rollover action. -You can also invoke this API directly to add templates. - - ////////////////////////// [source,console] diff --git a/docs/reference/ilm/ilm-with-existing-indices.asciidoc b/docs/reference/ilm/ilm-with-existing-indices.asciidoc index 98c2ea4d732..35824a9137e 100644 --- a/docs/reference/ilm/ilm-with-existing-indices.asciidoc +++ b/docs/reference/ilm/ilm-with-existing-indices.asciidoc @@ -68,7 +68,7 @@ but a combined index needs to be retained until you are ready to delete the new . Reduce the {ilm-init} poll interval to ensure that the index doesn't grow too large while waiting for the rollover check. -By default, {ilm-init} checks rollover conditions every 10 minutes. +By default, {ilm-init} checks to see what actions need to be taken every 10 minutes. + -- [source,console] diff --git a/docs/reference/ilm/index.asciidoc b/docs/reference/ilm/index.asciidoc index ddde53a1700..550608916d8 100644 --- a/docs/reference/ilm/index.asciidoc +++ b/docs/reference/ilm/index.asciidoc @@ -5,17 +5,20 @@ [partintro] -- -You can configure {ilm} ({ilm-init}) policies to automatically manage indices according to -your performance, resiliency, and retention requirements. +You can configure {ilm} ({ilm-init}) policies to automatically manage indices +according to your performance, resiliency, and retention requirements. For example, you could use {ilm-init} to: * Spin up a new index when an index reaches a certain size or number of documents * Create a new index each day, week, or month and archive previous ones * Delete stale indices to enforce data retention standards - + +You can create and manage index lifecycle policies through {kib} Management or the {ilm-init} APIs. When you enable {ilm} for {beats} or the {ls} {es} output plugin, -{ilm-init} is configured automatically. -You can modify the default policies through {kib} Management or the {ilm-init} APIs. +default policies are configured automatically. + +[role="screenshot"] +image:images/ilm/index-lifecycle-policies.png[] [TIP] To automatically back up your indices and manage snapshots, @@ -24,13 +27,15 @@ use < * <> * <> * <> -* <> +* <> * <> * <> * <> * <> * <> * <> +* <> +* <> -- include::ilm-overview.asciidoc[] @@ -39,6 +44,8 @@ include::ilm-concepts.asciidoc[] include::ilm-tutorial.asciidoc[] +include::example-index-lifecycle-policy.asciidoc[leveloffset=-1] + include::ilm-actions.asciidoc[] include::set-up-lifecycle-policy.asciidoc[] diff --git a/docs/reference/ilm/set-up-lifecycle-policy.asciidoc b/docs/reference/ilm/set-up-lifecycle-policy.asciidoc index 6f70dcdaedc..7a2f38bdbad 100644 --- a/docs/reference/ilm/set-up-lifecycle-policy.asciidoc +++ b/docs/reference/ilm/set-up-lifecycle-policy.asciidoc @@ -10,11 +10,13 @@ To configure a lifecycle policy for <>, you create the policy and add it to the <>. To use a policy to manage an index that doesn't roll over, -you can specify a lifecycle policy when you create it. +you can specify a lifecycle policy when you create the index, +or apply a policy directly to an existing index. {ilm-init} policies are stored in the global cluster state and can be included in snapshots by setting `include_global_state` to `true` when you <>. -When the snapshot is restored, all of the policies in the global state are restored and any local policies with the same names are overwritten. +When the snapshot is restored, all of the policies in the global state are restored and +any local policies with the same names are overwritten. IMPORTANT: When you enable {ilm} for {beats} or the {ls} {es} output plugin, the necessary policies and configuration changes are applied automatically. @@ -25,12 +27,19 @@ bootstrap an initial index. [[ilm-create-policy]] === Create lifecycle policy -You use the <> to define a new lifecycle policy. -For example, the following request creates `my_policy`, a -policy that defines a hot and and delete phase. -When the index reaches 25GB, it rolls over directly to the delete phase. -The index is deleted 30 days after rollover. +To create lifecycle policies through {kib} Management +go to Management and click **Index Lifecycle Policies**. +[role="screenshot"] +image:images/ilm/create-policy.png[] + +You specify the lifecycle phases for the policy and the actions to perform in each phase. + +The <> API is invoked to add the policy to the {es} cluster. + +.API example +[%collapsible] +==== [source,console] ------------------------ PUT _ilm/policy/my_policy @@ -57,6 +66,7 @@ PUT _ilm/policy/my_policy <1> Roll over the index when it reaches 25GB in size <2> Delete the index 30 days after rollover +==== [discrete] [[apply-policy-template]] @@ -64,11 +74,19 @@ PUT _ilm/policy/my_policy To use a policy that triggers the rollover action, you need to configure the policy in the index template used to create each new index. - -In addition to specifying the name of the policy in the `index.lifecycle.name` setting, -you specify a `index.lifecycle.rollover_alias` for referencing -the indices managed by this policy. +You specify the name of the policy and the alias used to reference the rolling indices. +To use the Create template wizard to create a template from {kib} Management, +go to Management, click **Index Management** and select the **Index Templates** view. + +[role="screenshot"] +image:images/ilm/create-template-wizard.png[] + +The wizard invokes the <> to add templates to a cluster. + +.API example +[%collapsible] +==== [source,console] ----------------------- PUT _template/my_template @@ -86,7 +104,7 @@ PUT _template/my_template <1> Use this template for all new indices whose names begin with `test-` <2> Apply `my_policy` to new indices created with this template <3> Define an index alias for referencing indices managed by `my_policy` - +==== ////////////////////////// [source,console] @@ -101,8 +119,14 @@ DELETE /_template/my_template [[create-initial-index]] ==== Create an initial managed index -You need to manually create the first index managed by a policy that uses the rollover action -and designate it as the write index. +When you set up policies for your own rolling indices, you need to manually create the first index +managed by a policy and designate it as the write index. + +IMPORTANT: When you enable {ilm} for {beats} or the {ls} {es} output plugin, +the necessary policies and configuration changes are applied automatically. +You can modify the default policies, but you do not need to explicitly configure a policy or +bootstrap an initial index. + The name of the index must match the pattern defined in the index template and end with a number. This number is incremented to generate the name of indices created by the rollover action. @@ -133,10 +157,20 @@ index exceeds 25GB. [[apply-policy-manually]] === Apply lifecycle policy manually -When you create an index, you can apply a lifecycle policy -by specifying the `index.lifecycle.name` setting. -This causes {ilm-init} to immediately start managing the index. +You can specify a policy when you create an index or +apply a policy to an existing index through {kib} Management or +the <>. +When you apply a policy, {ilm-init} immediately starts managing the index. +IMPORTANT: Do not manually apply a policy that uses the rollover action. +Policies that use rollover must be applied by the <>. +Otherwise, the policy is not carried forward when the rollover action creates a new index. + +The `index.lifecycle.name` setting specifies an index's policy. + +.API example +[%collapsible] +==== [source,console] ----------------------- PUT test-index @@ -144,14 +178,12 @@ PUT test-index "settings": { "number_of_shards": 1, "number_of_replicas": 1, - "index.lifecycle.name": "my_policy" + "index.lifecycle.name": "my_policy" <1> } } ----------------------- - -IMPORTANT: Do not manually apply a policy that uses the rollover action. -Policies that use rollover must be applied by the <>. -Otherwise, the policy is not carried forward when the rollover action creates a new index. +<1> Sets the lifecycle policy for the index. +==== [discrete] [[apply-policy-multiple]] diff --git a/docs/reference/images/ilm/create-policy.png b/docs/reference/images/ilm/create-policy.png new file mode 100644 index 00000000000..f6d86fa9b4e Binary files /dev/null and b/docs/reference/images/ilm/create-policy.png differ diff --git a/docs/reference/images/ilm/create-template-wizard.png b/docs/reference/images/ilm/create-template-wizard.png new file mode 100644 index 00000000000..b18c36366be Binary files /dev/null and b/docs/reference/images/ilm/create-template-wizard.png differ diff --git a/docs/reference/images/ilm/index-lifecycle-policies.png b/docs/reference/images/ilm/index-lifecycle-policies.png new file mode 100644 index 00000000000..184188be181 Binary files /dev/null and b/docs/reference/images/ilm/index-lifecycle-policies.png differ diff --git a/docs/reference/images/ilm/tutorial-ilm-custom-policy.png b/docs/reference/images/ilm/tutorial-ilm-custom-policy.png new file mode 100644 index 00000000000..03b67829f60 Binary files /dev/null and b/docs/reference/images/ilm/tutorial-ilm-custom-policy.png differ diff --git a/docs/reference/images/ilm/tutorial-ilm-delete-phase-creation.png b/docs/reference/images/ilm/tutorial-ilm-delete-phase-creation.png new file mode 100644 index 00000000000..91a55733c28 Binary files /dev/null and b/docs/reference/images/ilm/tutorial-ilm-delete-phase-creation.png differ diff --git a/docs/reference/images/ilm/tutorial-ilm-delete-rollover.png b/docs/reference/images/ilm/tutorial-ilm-delete-rollover.png new file mode 100644 index 00000000000..ba021ecc2ac Binary files /dev/null and b/docs/reference/images/ilm/tutorial-ilm-delete-rollover.png differ diff --git a/docs/reference/images/ilm/tutorial-ilm-hotphaserollover-default.png b/docs/reference/images/ilm/tutorial-ilm-hotphaserollover-default.png new file mode 100644 index 00000000000..a9088c63d88 Binary files /dev/null and b/docs/reference/images/ilm/tutorial-ilm-hotphaserollover-default.png differ diff --git a/docs/reference/images/ilm/tutorial-ilm-modify-default-warm-phase-rollover.png b/docs/reference/images/ilm/tutorial-ilm-modify-default-warm-phase-rollover.png new file mode 100644 index 00000000000..c6f1e9b40e9 Binary files /dev/null and b/docs/reference/images/ilm/tutorial-ilm-modify-default-warm-phase-rollover.png differ diff --git a/docs/reference/index.asciidoc b/docs/reference/index.asciidoc index eba1f3c2c37..c71c5ecc4b3 100644 --- a/docs/reference/index.asciidoc +++ b/docs/reference/index.asciidoc @@ -7,7 +7,6 @@ :xes-repo-dir: {elasticsearch-root}/x-pack/docs/{lang} :es-repo-dir: {elasticsearch-root}/docs/reference - include::../Versions.asciidoc[] include::intro.asciidoc[]