diff --git a/docs/reference/ilm/getting-started-ilm.asciidoc b/docs/reference/ilm/getting-started-ilm.asciidoc index ad3596e92a1..3973aa0ad93 100644 --- a/docs/reference/ilm/getting-started-ilm.asciidoc +++ b/docs/reference/ilm/getting-started-ilm.asciidoc @@ -1,15 +1,228 @@ [role="xpack"] +[testenv="basic"] [[getting-started-index-lifecycle-management]] == Getting started with {ilm} -Create a policy that rolls over after 1 day deletes an index after 30 days +Let's jump into {ILM} by working through a hands-on scenario. +This section will leverage many new concepts unique to {ILM} that +you may not be familiar with. The following sections will explore +these in more details. -Show create policy API req/res +The goal of this example is to set up a set of indices that will encapsulate +the data from a time series data source. We can imagine there is a system +like {filebeat-ref}[Filebeat] that continuously indexes documents into +our writing index. We wish to roll over the index after it reaches a size +of 50 gigabytes, or has been created 30 days ago, and then delete the index +after 90 days. -Show assign policy to index API req/res +=== Setting up a new policy -Show both the API and how it is done with `index.lifecyce.name` using the -create-index API +There are many new features introduced by {ILM}, but we will only focus on +a few that are needed for our example. For starters, we will use the +<> API to define our first policy. Lifecycle +policies are defined in JSON and include specific +<>. -Show explain API to show current state, but ignore the “step” related info, -only focus on managed/phase/action +[source,js] +------------------------ +PUT _ilm/policy/datastream_policy <1> +{ + "policy": { <2> + "phases": { + "hot": { <3> + "actions": { + "rollover": { <4> + "max_size": "50GB", + "max_age": "30d" + } + } + }, + "delete": { + "min_age": "90d", <5> + "actions": { + "delete": {} <6> + } + } + } + } +} +------------------------ +// CONSOLE +// TEST +<1> call to the <> endpoint to create + a new policy named "datastream_policy" +<2> policy definition sub-object +<3> the hot phase defined in the "phases" section. Optional `min_age` field + not defined -- defaults to `0ms` +<4> rollover action definition +<5> delete phase begins after 90 days +<6> delete action definition + + +Here we created the policy called `datastream_policy` which rolls over +the index being written to after it reaches 50 gigabytes, or it is 30 +days old. The rollover will occur when either of these conditions is true. +The index will be deleted 90 days after it is rolled over. + +=== Applying a policy to our index + +There are <> to associate a +policy to an index. Since we wish specific settings to be applied to +the new index created from Rollover, we will set the policy via +index templates. + + +[source,js] +----------------------- +PUT _template/datastream_template +{ + "index_patterns": ["datastream-*"], <1> + "settings": { + "number_of_shards": 1, + "number_of_replicas": 1, + "index.lifecycle.name": "datastream_policy", <2> + "index.lifecycle.rollover_alias": "datastream" <3> + } +} +----------------------- +// CONSOLE +// TEST[continued] +<1> match all indices starting with "datastream-". These will include all + newly created indices from actions like rollover +<2> the name of the lifecycle policy managing the index +<3> alias to use for the rollover action, required since a rollover action is + defined in the policy. + +The above index template introduces a few new settings specific to {ILM}. +The first being `index.lifecycle.name`. This setting will configure +the "datastream_policy" to the index applying this template. This means +that all newly created indices prefixed "datastream-" will be managed by +our policy. The other setting used here is `index.lifecycle.rollover_alias`. +This setting is required when using a policy containing the rollover +action and specifies which alias to rollover on behalf of this index. +The intention here is that the rollover alias is also defined on the index. + +To begin, we will want to bootstrap our first index to write to. + + +[source,js] +----------------------- +PUT datastream-000001 +{ + "aliases": { + "datastream": { + "is_write_index": true + } + } +} +----------------------- +// CONSOLE +// TEST[continued] + +When creating our index, we have to consider a few important configurations +that tie our index and our policy together correctly. We need to make sure +that our index name matches our index template pattern of "datastream-*", +which it does. We are using the <> in our policy, which +requires that our index name ends with a number. In our case, we used +`000001`. This is important so that Rollover can increment this number when +naming the new index created from rolling over. + +Our index creation request leverages its template to apply our settings, +but we must also configure our rollover alias: "datastream". To do this, +we take advantage of <>. This is a way +to define an alias to be used for both reading and writing, with only one +index being the index that is being written to at a time. Rollover swaps +the write index to be the new index created from rollover, and sets the +alias to be read-only for the source index. + +=== Checking progress + +Now that we have an index managed by our policy, how do we tell what is going +on? Which phase are we in? Is something broken? This section will go over a +few APIs and their responses to help us inspect our indices with respect +to {ILM}. + +With the help of the <>, we can know +things like which phase we're in and when we entered that phase. The API +will also provide further info if errors occurred, or if we are blocked on +certain checks within actions. + +[source,js] +-------------------------------------------------- +GET datastream-*/_ilm/explain +-------------------------------------------------- +// CONSOLE +// TEST[continued] + +The above request will retrieve {ILM} execution information for all our +managed indices. + + +[source,js] +-------------------------------------------------- +{ + "indices": { + "datastream-000001": { + "index": "datastream-000001", + "managed": true, <1> + "policy": "datastream_policy", <2> + "lifecycle_date_millis": 1538475653281, + "phase": "hot", <3> + "phase_time_millis": 1538475653317, + "action": "rollover", <4> + "action_time_millis": 1538475653317, + "step": "attempt_rollover", <5> + "step_time_millis": 1538475653317, + "phase_execution": { + "policy": "datastream_policy", + "phase_definition": { <6> + "min_age": "0ms", + "actions": { + "rollover": { + "max_size": "50gb", + "max_age": "30d" + } + } + }, + "version": 1, <7> + "modified_date_in_millis": 1539609701576 + } + } + } +} +-------------------------------------------------- +// CONSOLE +// TESTRESPONSE[s/"lifecycle_date_millis": 1538475653281/"lifecycle_date_millis": $body.indices.datastream-000001.lifecycle_date_millis/] +// TESTRESPONSE[s/"phase_time_millis": 1538475653317/"phase_time_millis": $body.indices.datastream-000001.phase_time_millis/] +// TESTRESPONSE[s/"action_time_millis": 1538475653317/"action_time_millis": $body.indices.datastream-000001.action_time_millis/] +// TESTRESPONSE[s/"step_time_millis": 1538475653317/"step_time_millis": $body.indices.datastream-000001.step_time_millis/] +// TESTRESPONSE[s/"modified_date_in_millis": 1539609701576/"modified_date_in_millis": $body.indices.datastream-000001.phase_execution.modified_date_in_millis/] +<1> this index is managed by ILM +<2> the policy in question, in this case, "datastream_policy" +<3> what phase the index is currently in +<4> what action the index is currently on +<5> what step the index is currently on +<6> the definition of the phase + (in this case, the "hot" phase) that the index is currently on +<7> the version of the policy being used to execute the current phase + +You can read about the full details of this response in the +<>. For now, let's focus on how +the response details which phase, action, and step we're in. We are in the +"hot" phase, and "rollover" action. Rollover will continue to be called +by {ILM} until its conditions are met and it rolls over the index. +Afterwards, the original index will stay in the hot phase until 90 more +days pass and it is deleted in the delete phase. +As time goes on, new indices will be created and deleted. +With `datastream-000002` being created when the index mets the rollover +conditions and `datastream-000003` created after that. We will be able +to search across all of our managed indices using the "datastream" alias, +and we will be able to write to our to-be-rolled-over write indices using +that same alias. + + + +That's it! We have our first use-case managed by {ILM}. + +To learn more about all our APIs, +check out <>.