[ILM] getting started documentation (#35379)
This commit is contained in:
Tal Levy
GitHub
parent
8ea999e489
commit
f70d4470f7
|
|
@ -1,15 +1,228 @@
|
|||
[role="xpack"]
|
||||
[testenv="basic"]
|
||||
[[getting-started-index-lifecycle-management]]
|
||||
== Getting started with {ilm}
|
||||
|
||||
Create a policy that rolls over after 1 day deletes an index after 30 days
|
||||
Let's jump into {ILM} by working through a hands-on scenario.
|
||||
This section will leverage many new concepts unique to {ILM} that
|
||||
you may not be familiar with. The following sections will explore
|
||||
these in more details.
|
||||
|
||||
Show create policy API req/res
|
||||
The goal of this example is to set up a set of indices that will encapsulate
|
||||
the data from a time series data source. We can imagine there is a system
|
||||
like {filebeat-ref}[Filebeat] that continuously indexes documents into
|
||||
our writing index. We wish to roll over the index after it reaches a size
|
||||
of 50 gigabytes, or has been created 30 days ago, and then delete the index
|
||||
after 90 days.
|
||||
|
||||
Show assign policy to index API req/res
|
||||
=== Setting up a new policy
|
||||
|
||||
Show both the API and how it is done with `index.lifecyce.name` using the
|
||||
create-index API
|
||||
There are many new features introduced by {ILM}, but we will only focus on
|
||||
a few that are needed for our example. For starters, we will use the
|
||||
<<ilm-put-lifecycle,Put Policy>> API to define our first policy. Lifecycle
|
||||
policies are defined in JSON and include specific
|
||||
<<ilm-policy-definition,phases and actions>>.
|
||||
|
||||
Show explain API to show current state, but ignore the “step” related info,
|
||||
only focus on managed/phase/action
|
||||
[source,js]
|
||||
------------------------
|
||||
PUT _ilm/policy/datastream_policy <1>
|
||||
{
|
||||
"policy": { <2>
|
||||
"phases": {
|
||||
"hot": { <3>
|
||||
"actions": {
|
||||
"rollover": { <4>
|
||||
"max_size": "50GB",
|
||||
"max_age": "30d"
|
||||
}
|
||||
}
|
||||
},
|
||||
"delete": {
|
||||
"min_age": "90d", <5>
|
||||
"actions": {
|
||||
"delete": {} <6>
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
------------------------
|
||||
// CONSOLE
|
||||
// TEST
|
||||
<1> call to the <<ilm-put-lifecycle,put lifecycle API>> endpoint to create
|
||||
a new policy named "datastream_policy"
|
||||
<2> policy definition sub-object
|
||||
<3> the hot phase defined in the "phases" section. Optional `min_age` field
|
||||
not defined -- defaults to `0ms`
|
||||
<4> rollover action definition
|
||||
<5> delete phase begins after 90 days
|
||||
<6> delete action definition
|
||||
|
||||
|
||||
Here we created the policy called `datastream_policy` which rolls over
|
||||
the index being written to after it reaches 50 gigabytes, or it is 30
|
||||
days old. The rollover will occur when either of these conditions is true.
|
||||
The index will be deleted 90 days after it is rolled over.
|
||||
|
||||
=== Applying a policy to our index
|
||||
|
||||
There are <<set-up-lifecycle-policy,a few ways>> to associate a
|
||||
policy to an index. Since we wish specific settings to be applied to
|
||||
the new index created from Rollover, we will set the policy via
|
||||
index templates.
|
||||
|
||||
|
||||
[source,js]
|
||||
-----------------------
|
||||
PUT _template/datastream_template
|
||||
{
|
||||
"index_patterns": ["datastream-*"], <1>
|
||||
"settings": {
|
||||
"number_of_shards": 1,
|
||||
"number_of_replicas": 1,
|
||||
"index.lifecycle.name": "datastream_policy", <2>
|
||||
"index.lifecycle.rollover_alias": "datastream" <3>
|
||||
}
|
||||
}
|
||||
-----------------------
|
||||
// CONSOLE
|
||||
// TEST[continued]
|
||||
<1> match all indices starting with "datastream-". These will include all
|
||||
newly created indices from actions like rollover
|
||||
<2> the name of the lifecycle policy managing the index
|
||||
<3> alias to use for the rollover action, required since a rollover action is
|
||||
defined in the policy.
|
||||
|
||||
The above index template introduces a few new settings specific to {ILM}.
|
||||
The first being `index.lifecycle.name`. This setting will configure
|
||||
the "datastream_policy" to the index applying this template. This means
|
||||
that all newly created indices prefixed "datastream-" will be managed by
|
||||
our policy. The other setting used here is `index.lifecycle.rollover_alias`.
|
||||
This setting is required when using a policy containing the rollover
|
||||
action and specifies which alias to rollover on behalf of this index.
|
||||
The intention here is that the rollover alias is also defined on the index.
|
||||
|
||||
To begin, we will want to bootstrap our first index to write to.
|
||||
|
||||
|
||||
[source,js]
|
||||
-----------------------
|
||||
PUT datastream-000001
|
||||
{
|
||||
"aliases": {
|
||||
"datastream": {
|
||||
"is_write_index": true
|
||||
}
|
||||
}
|
||||
}
|
||||
-----------------------
|
||||
// CONSOLE
|
||||
// TEST[continued]
|
||||
|
||||
When creating our index, we have to consider a few important configurations
|
||||
that tie our index and our policy together correctly. We need to make sure
|
||||
that our index name matches our index template pattern of "datastream-*",
|
||||
which it does. We are using the <<ilm-rollover-action, Rollover Action>> in our policy, which
|
||||
requires that our index name ends with a number. In our case, we used
|
||||
`000001`. This is important so that Rollover can increment this number when
|
||||
naming the new index created from rolling over.
|
||||
|
||||
Our index creation request leverages its template to apply our settings,
|
||||
but we must also configure our rollover alias: "datastream". To do this,
|
||||
we take advantage of <<aliases-write-index,write indices>>. This is a way
|
||||
to define an alias to be used for both reading and writing, with only one
|
||||
index being the index that is being written to at a time. Rollover swaps
|
||||
the write index to be the new index created from rollover, and sets the
|
||||
alias to be read-only for the source index.
|
||||
|
||||
=== Checking progress
|
||||
|
||||
Now that we have an index managed by our policy, how do we tell what is going
|
||||
on? Which phase are we in? Is something broken? This section will go over a
|
||||
few APIs and their responses to help us inspect our indices with respect
|
||||
to {ILM}.
|
||||
|
||||
With the help of the <<ilm-explain-lifecycle,Explain API>>, we can know
|
||||
things like which phase we're in and when we entered that phase. The API
|
||||
will also provide further info if errors occurred, or if we are blocked on
|
||||
certain checks within actions.
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
GET datastream-*/_ilm/explain
|
||||
--------------------------------------------------
|
||||
// CONSOLE
|
||||
// TEST[continued]
|
||||
|
||||
The above request will retrieve {ILM} execution information for all our
|
||||
managed indices.
|
||||
|
||||
|
||||
[source,js]
|
||||
--------------------------------------------------
|
||||
{
|
||||
"indices": {
|
||||
"datastream-000001": {
|
||||
"index": "datastream-000001",
|
||||
"managed": true, <1>
|
||||
"policy": "datastream_policy", <2>
|
||||
"lifecycle_date_millis": 1538475653281,
|
||||
"phase": "hot", <3>
|
||||
"phase_time_millis": 1538475653317,
|
||||
"action": "rollover", <4>
|
||||
"action_time_millis": 1538475653317,
|
||||
"step": "attempt_rollover", <5>
|
||||
"step_time_millis": 1538475653317,
|
||||
"phase_execution": {
|
||||
"policy": "datastream_policy",
|
||||
"phase_definition": { <6>
|
||||
"min_age": "0ms",
|
||||
"actions": {
|
||||
"rollover": {
|
||||
"max_size": "50gb",
|
||||
"max_age": "30d"
|
||||
}
|
||||
}
|
||||
},
|
||||
"version": 1, <7>
|
||||
"modified_date_in_millis": 1539609701576
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
--------------------------------------------------
|
||||
// CONSOLE
|
||||
// TESTRESPONSE[s/"lifecycle_date_millis": 1538475653281/"lifecycle_date_millis": $body.indices.datastream-000001.lifecycle_date_millis/]
|
||||
// TESTRESPONSE[s/"phase_time_millis": 1538475653317/"phase_time_millis": $body.indices.datastream-000001.phase_time_millis/]
|
||||
// TESTRESPONSE[s/"action_time_millis": 1538475653317/"action_time_millis": $body.indices.datastream-000001.action_time_millis/]
|
||||
// TESTRESPONSE[s/"step_time_millis": 1538475653317/"step_time_millis": $body.indices.datastream-000001.step_time_millis/]
|
||||
// TESTRESPONSE[s/"modified_date_in_millis": 1539609701576/"modified_date_in_millis": $body.indices.datastream-000001.phase_execution.modified_date_in_millis/]
|
||||
<1> this index is managed by ILM
|
||||
<2> the policy in question, in this case, "datastream_policy"
|
||||
<3> what phase the index is currently in
|
||||
<4> what action the index is currently on
|
||||
<5> what step the index is currently on
|
||||
<6> the definition of the phase
|
||||
(in this case, the "hot" phase) that the index is currently on
|
||||
<7> the version of the policy being used to execute the current phase
|
||||
|
||||
You can read about the full details of this response in the
|
||||
<<ilm-explain-lifecycle, explain API docs>>. For now, let's focus on how
|
||||
the response details which phase, action, and step we're in. We are in the
|
||||
"hot" phase, and "rollover" action. Rollover will continue to be called
|
||||
by {ILM} until its conditions are met and it rolls over the index.
|
||||
Afterwards, the original index will stay in the hot phase until 90 more
|
||||
days pass and it is deleted in the delete phase.
|
||||
As time goes on, new indices will be created and deleted.
|
||||
With `datastream-000002` being created when the index mets the rollover
|
||||
conditions and `datastream-000003` created after that. We will be able
|
||||
to search across all of our managed indices using the "datastream" alias,
|
||||
and we will be able to write to our to-be-rolled-over write indices using
|
||||
that same alias.
|
||||
|
||||
|
||||
|
||||
That's it! We have our first use-case managed by {ILM}.
|
||||
|
||||
To learn more about all our APIs,
|
||||
check out <<index-lifecycle-management-api,ILM APIs>>.
|
||||
|
|
|
|||
Loading…
Reference in New Issue