233 lines
8.1 KiB
Plaintext
233 lines
8.1 KiB
Plaintext
[role="xpack"]
|
|
[testenv="basic"]
|
|
[[getting-started-index-lifecycle-management]]
|
|
== Getting started with {ilm}
|
|
|
|
beta[]
|
|
|
|
Let's jump into {ILM} by working through a hands-on scenario.
|
|
This section will leverage many new concepts unique to {ILM} that
|
|
you may not be familiar with. The following sections will explore
|
|
these in more details.
|
|
|
|
The goal of this example is to set up a set of indices that will encapsulate
|
|
the data from a time series data source. We can imagine there is a system
|
|
like {filebeat-ref}[Filebeat] that continuously indexes documents into
|
|
our writing index. We wish to roll over the index after it reaches a size
|
|
of 50 gigabytes, or has been created 30 days ago, and then delete the index
|
|
after 90 days.
|
|
|
|
=== Setting up a new policy
|
|
|
|
beta[]
|
|
|
|
There are many new features introduced by {ILM}, but we will only focus on
|
|
a few that are needed for our example. For starters, we will use the
|
|
<<ilm-put-lifecycle,Put Policy>> API to define our first policy. Lifecycle
|
|
policies are defined in JSON and include specific
|
|
<<ilm-policy-definition,phases and actions>>.
|
|
|
|
[source,js]
|
|
------------------------
|
|
PUT _ilm/policy/datastream_policy <1>
|
|
{
|
|
"policy": { <2>
|
|
"phases": {
|
|
"hot": { <3>
|
|
"actions": {
|
|
"rollover": { <4>
|
|
"max_size": "50GB",
|
|
"max_age": "30d"
|
|
}
|
|
}
|
|
},
|
|
"delete": {
|
|
"min_age": "90d", <5>
|
|
"actions": {
|
|
"delete": {} <6>
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
------------------------
|
|
// CONSOLE
|
|
// TEST
|
|
<1> call to the <<ilm-put-lifecycle,put lifecycle API>> endpoint to create
|
|
a new policy named "datastream_policy"
|
|
<2> policy definition sub-object
|
|
<3> the hot phase defined in the "phases" section. Optional `min_age` field
|
|
not defined -- defaults to `0ms`
|
|
<4> rollover action definition
|
|
<5> delete phase begins after 90 days
|
|
<6> delete action definition
|
|
|
|
|
|
Here we created the policy called `datastream_policy` which rolls over
|
|
the index being written to after it reaches 50 gigabytes, or it is 30
|
|
days old. The rollover will occur when either of these conditions is true.
|
|
The index will be deleted 90 days after it is rolled over.
|
|
|
|
=== Applying a policy to our index
|
|
|
|
beta[]
|
|
|
|
There are <<set-up-lifecycle-policy,a few ways>> to associate a
|
|
policy to an index. Since we wish specific settings to be applied to
|
|
the new index created from Rollover, we will set the policy via
|
|
index templates.
|
|
|
|
|
|
[source,js]
|
|
-----------------------
|
|
PUT _template/datastream_template
|
|
{
|
|
"index_patterns": ["datastream-*"], <1>
|
|
"settings": {
|
|
"number_of_shards": 1,
|
|
"number_of_replicas": 1,
|
|
"index.lifecycle.name": "datastream_policy", <2>
|
|
"index.lifecycle.rollover_alias": "datastream" <3>
|
|
}
|
|
}
|
|
-----------------------
|
|
// CONSOLE
|
|
// TEST[continued]
|
|
<1> match all indices starting with "datastream-". These will include all
|
|
newly created indices from actions like rollover
|
|
<2> the name of the lifecycle policy managing the index
|
|
<3> alias to use for the rollover action, required since a rollover action is
|
|
defined in the policy.
|
|
|
|
The above index template introduces a few new settings specific to {ILM}.
|
|
The first being `index.lifecycle.name`. This setting will configure
|
|
the "datastream_policy" to the index applying this template. This means
|
|
that all newly created indices prefixed "datastream-" will be managed by
|
|
our policy. The other setting used here is `index.lifecycle.rollover_alias`.
|
|
This setting is required when using a policy containing the rollover
|
|
action and specifies which alias to rollover on behalf of this index.
|
|
The intention here is that the rollover alias is also defined on the index.
|
|
|
|
To begin, we will want to bootstrap our first index to write to.
|
|
|
|
|
|
[source,js]
|
|
-----------------------
|
|
PUT datastream-000001
|
|
{
|
|
"aliases": {
|
|
"datastream": {
|
|
"is_write_index": true
|
|
}
|
|
}
|
|
}
|
|
-----------------------
|
|
// CONSOLE
|
|
// TEST[continued]
|
|
|
|
When creating our index, we have to consider a few important configurations
|
|
that tie our index and our policy together correctly. We need to make sure
|
|
that our index name matches our index template pattern of "datastream-*",
|
|
which it does. We are using the <<ilm-rollover-action, Rollover Action>> in our policy, which
|
|
requires that our index name ends with a number. In our case, we used
|
|
`000001`. This is important so that Rollover can increment this number when
|
|
naming the new index created from rolling over.
|
|
|
|
Our index creation request leverages its template to apply our settings,
|
|
but we must also configure our rollover alias: "datastream". To do this,
|
|
we take advantage of <<aliases-write-index,write indices>>. This is a way
|
|
to define an alias to be used for both reading and writing, with only one
|
|
index being the index that is being written to at a time. Rollover swaps
|
|
the write index to be the new index created from rollover, and sets the
|
|
alias to be read-only for the source index.
|
|
|
|
=== Checking progress
|
|
|
|
beta[]
|
|
|
|
Now that we have an index managed by our policy, how do we tell what is going
|
|
on? Which phase are we in? Is something broken? This section will go over a
|
|
few APIs and their responses to help us inspect our indices with respect
|
|
to {ILM}.
|
|
|
|
With the help of the <<ilm-explain-lifecycle,Explain API>>, we can know
|
|
things like which phase we're in and when we entered that phase. The API
|
|
will also provide further info if errors occurred, or if we are blocked on
|
|
certain checks within actions.
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
GET datastream-*/_ilm/explain
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
// TEST[continued]
|
|
|
|
The above request will retrieve {ILM} execution information for all our
|
|
managed indices.
|
|
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
{
|
|
"indices": {
|
|
"datastream-000001": {
|
|
"index": "datastream-000001",
|
|
"managed": true, <1>
|
|
"policy": "datastream_policy", <2>
|
|
"lifecycle_date_millis": 1538475653281,
|
|
"phase": "hot", <3>
|
|
"phase_time_millis": 1538475653317,
|
|
"action": "rollover", <4>
|
|
"action_time_millis": 1538475653317,
|
|
"step": "attempt_rollover", <5>
|
|
"step_time_millis": 1538475653317,
|
|
"phase_execution": {
|
|
"policy": "datastream_policy",
|
|
"phase_definition": { <6>
|
|
"min_age": "0ms",
|
|
"actions": {
|
|
"rollover": {
|
|
"max_size": "50gb",
|
|
"max_age": "30d"
|
|
}
|
|
}
|
|
},
|
|
"version": 1, <7>
|
|
"modified_date_in_millis": 1539609701576
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
// TESTRESPONSE[skip:no way to know if we will get this response immediately]
|
|
<1> this index is managed by ILM
|
|
<2> the policy in question, in this case, "datastream_policy"
|
|
<3> what phase the index is currently in
|
|
<4> what action the index is currently on
|
|
<5> what step the index is currently on
|
|
<6> the definition of the phase
|
|
(in this case, the "hot" phase) that the index is currently on
|
|
<7> the version of the policy being used to execute the current phase
|
|
|
|
You can read about the full details of this response in the
|
|
<<ilm-explain-lifecycle, explain API docs>>. For now, let's focus on how
|
|
the response details which phase, action, and step we're in. We are in the
|
|
"hot" phase, and "rollover" action. Rollover will continue to be called
|
|
by {ILM} until its conditions are met and it rolls over the index.
|
|
Afterwards, the original index will stay in the hot phase until 90 more
|
|
days pass and it is deleted in the delete phase.
|
|
As time goes on, new indices will be created and deleted.
|
|
With `datastream-000002` being created when the index mets the rollover
|
|
conditions and `datastream-000003` created after that. We will be able
|
|
to search across all of our managed indices using the "datastream" alias,
|
|
and we will be able to write to our to-be-rolled-over write indices using
|
|
that same alias.
|
|
|
|
|
|
|
|
That's it! We have our first use-case managed by {ILM}.
|
|
|
|
To learn more about all our APIs,
|
|
check out <<index-lifecycle-management-api,ILM APIs>>.
|