338 lines
9.5 KiB
Plaintext
338 lines
9.5 KiB
Plaintext
[[indices-rollover-index]]
|
|
== Rollover Index
|
|
|
|
The rollover index API rolls an alias over to a new index when the existing
|
|
index is considered to be too large or too old.
|
|
|
|
The API accepts a single alias name and a list of `conditions`. The alias must point to a write index for
|
|
a Rollover request to be valid. There are two ways this can be achieved, and depending on the configuration, the
|
|
alias metadata will be updated differently. The two scenarios are as follows:
|
|
|
|
- The alias only points to a single index with `is_write_index` not configured (defaults to `null`).
|
|
|
|
In this scenario, the original index will have their rollover alias will be added to the newly created index, and removed
|
|
from the original (rolled-over) index.
|
|
|
|
- The alias points to one or more indices with `is_write_index` set to `true` on the index to be rolled over (the write index).
|
|
|
|
In this scenario, the write index will have its rollover alias' `is_write_index` set to `false`, while the newly created index
|
|
will now have the rollover alias pointing to it as the write index with `is_write_index` as `true`.
|
|
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
PUT /logs-000001 <1>
|
|
{
|
|
"aliases": {
|
|
"logs_write": {}
|
|
}
|
|
}
|
|
|
|
# Add > 1000 documents to logs-000001
|
|
|
|
POST /logs_write/_rollover <2>
|
|
{
|
|
"conditions": {
|
|
"max_age": "7d",
|
|
"max_docs": 1000,
|
|
"max_size": "5gb"
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
// TEST[setup:huge_twitter]
|
|
// TEST[s/# Add > 1000 documents to logs-000001/POST _reindex?refresh\n{"source":{"index":"twitter"},"dest":{"index":"logs-000001"}}/]
|
|
<1> Creates an index called `logs-0000001` with the alias `logs_write`.
|
|
<2> If the index pointed to by `logs_write` was created 7 or more days ago, or
|
|
contains 1,000 or more documents, or has an index size at least around 5GB, then the `logs-000002` index is created
|
|
and the `logs_write` alias is updated to point to `logs-000002`.
|
|
|
|
The above request might return the following response:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
{
|
|
"acknowledged": true,
|
|
"shards_acknowledged": true,
|
|
"old_index": "logs-000001",
|
|
"new_index": "logs-000002",
|
|
"rolled_over": true, <1>
|
|
"dry_run": false, <2>
|
|
"conditions": { <3>
|
|
"[max_age: 7d]": false,
|
|
"[max_docs: 1000]": true,
|
|
"[max_size: 5gb]": false,
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// TESTRESPONSE
|
|
<1> Whether the index was rolled over.
|
|
<2> Whether the rollover was dry run.
|
|
<3> The result of each condition.
|
|
|
|
[float]
|
|
=== Naming the new index
|
|
|
|
If the name of the existing index ends with `-` and a number -- e.g.
|
|
`logs-000001` -- then the name of the new index will follow the same pattern,
|
|
incrementing the number (`logs-000002`). The number is zero-padded with a length
|
|
of 6, regardless of the old index name.
|
|
|
|
If the old name doesn't match this pattern then you must specify the name for
|
|
the new index as follows:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
POST /my_alias/_rollover/my_new_index_name
|
|
{
|
|
"conditions": {
|
|
"max_age": "7d",
|
|
"max_docs": 1000,
|
|
"max_size": "5gb"
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
// TEST[s/^/PUT my_old_index_name\nPUT my_old_index_name\/_alias\/my_alias\n/]
|
|
|
|
[float]
|
|
=== Using date math with the rollover API
|
|
|
|
It can be useful to use <<date-math-index-names,date math>> to name the
|
|
rollover index according to the date that the index rolled over, e.g.
|
|
`logstash-2016.02.03`. The rollover API supports date math, but requires the
|
|
index name to end with a dash followed by a number, e.g.
|
|
`logstash-2016.02.03-1` which is incremented every time the index is rolled
|
|
over. For instance:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
# PUT /<logs-{now/d}-1> with URI encoding:
|
|
PUT /%3Clogs-%7Bnow%2Fd%7D-1%3E <1>
|
|
{
|
|
"aliases": {
|
|
"logs_write": {}
|
|
}
|
|
}
|
|
|
|
PUT logs_write/_doc/1
|
|
{
|
|
"message": "a dummy log"
|
|
}
|
|
|
|
POST logs_write/_refresh
|
|
|
|
# Wait for a day to pass
|
|
|
|
POST /logs_write/_rollover <2>
|
|
{
|
|
"conditions": {
|
|
"max_docs": "1"
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
// TEST[s/now/2016.10.31||/]
|
|
<1> Creates an index named with today's date (e.g.) `logs-2016.10.31-1`
|
|
<2> Rolls over to a new index with today's date, e.g. `logs-2016.10.31-000002` if run immediately, or `logs-2016.11.01-000002` if run after 24 hours
|
|
|
|
//////////////////////////
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
GET _alias
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
// TEST[continued]
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
{
|
|
"logs-2016.10.31-000002": {
|
|
"aliases": {
|
|
"logs_write": {}
|
|
}
|
|
},
|
|
"logs-2016.10.31-1": {
|
|
"aliases": {}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// TESTRESPONSE
|
|
|
|
//////////////////////////
|
|
|
|
These indices can then be referenced as described in the
|
|
<<date-math-index-names,date math documentation>>. For example, to search
|
|
over indices created in the last three days, you could do the following:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
# GET /<logs-{now/d}-*>,<logs-{now/d-1d}-*>,<logs-{now/d-2d}-*>/_search
|
|
GET /%3Clogs-%7Bnow%2Fd%7D-*%3E%2C%3Clogs-%7Bnow%2Fd-1d%7D-*%3E%2C%3Clogs-%7Bnow%2Fd-2d%7D-*%3E/_search
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
// TEST[continued]
|
|
// TEST[s/now/2016.10.31||/]
|
|
|
|
[float]
|
|
=== Defining the new index
|
|
|
|
The settings, mappings, and aliases for the new index are taken from any
|
|
matching <<indices-templates,index templates>>. Additionally, you can specify
|
|
`settings`, `mappings`, and `aliases` in the body of the request, just like the
|
|
<<indices-create-index,create index>> API. Values specified in the request
|
|
override any values set in matching index templates. For example, the following
|
|
`rollover` request overrides the `index.number_of_shards` setting:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
PUT /logs-000001
|
|
{
|
|
"aliases": {
|
|
"logs_write": {}
|
|
}
|
|
}
|
|
|
|
POST /logs_write/_rollover
|
|
{
|
|
"conditions" : {
|
|
"max_age": "7d",
|
|
"max_docs": 1000,
|
|
"max_size": "5gb"
|
|
},
|
|
"settings": {
|
|
"index.number_of_shards": 2
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
|
|
[float]
|
|
=== Dry run
|
|
|
|
The rollover API supports `dry_run` mode, where request conditions can be
|
|
checked without performing the actual rollover:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
PUT /logs-000001
|
|
{
|
|
"aliases": {
|
|
"logs_write": {}
|
|
}
|
|
}
|
|
|
|
POST /logs_write/_rollover?dry_run
|
|
{
|
|
"conditions" : {
|
|
"max_age": "7d",
|
|
"max_docs": 1000,
|
|
"max_size": "5gb"
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
|
|
[float]
|
|
=== Wait For Active Shards
|
|
|
|
Because the rollover operation creates a new index to rollover to, the
|
|
<<create-index-wait-for-active-shards,`wait_for_active_shards`>> setting on
|
|
index creation applies to the rollover action as well.
|
|
|
|
[[indices-rollover-is-write-index]]
|
|
[float]
|
|
=== Write Index Alias Behavior
|
|
|
|
The rollover alias when rolling over a write index that has `is_write_index` explicitly set to `true` is not
|
|
swapped during rollover actions. Since having an alias point to multiple indices is ambiguous in distinguishing
|
|
which is the correct write index to roll over, it is not valid to rollover an alias that points to multiple indices.
|
|
For this reason, the default behavior is to swap which index is being pointed to by the write-oriented alias. This
|
|
was `logs_write` in some of the above examples. Since setting `is_write_index` enables an alias to point to multiple indices
|
|
while also being explicit as to which is the write index that rollover should target, removing the alias from the rolled over
|
|
index is not necessary. This simplifies things by allowing for one alias to behave both as the write and read aliases for
|
|
indices that are being managed with Rollover.
|
|
|
|
Look at the behavior of the aliases in the following example where `is_write_index` is set on the rolled over index.
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
PUT my_logs_index-000001
|
|
{
|
|
"aliases": {
|
|
"logs": { "is_write_index": true } <1>
|
|
}
|
|
}
|
|
|
|
PUT logs/_doc/1
|
|
{
|
|
"message": "a dummy log"
|
|
}
|
|
|
|
POST logs/_refresh
|
|
|
|
POST /logs/_rollover
|
|
{
|
|
"conditions": {
|
|
"max_docs": "1"
|
|
}
|
|
}
|
|
|
|
PUT logs/_doc/2 <2>
|
|
{
|
|
"message": "a newer log"
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
<1> configures `my_logs_index` as the write index for the `logs` alias
|
|
<2> newly indexed documents against the `logs` alias will write to the new index
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
{
|
|
"_index" : "my_logs_index-000002",
|
|
"_type" : "_doc",
|
|
"_id" : "2",
|
|
"_version" : 1,
|
|
"result" : "created",
|
|
"_shards" : {
|
|
"total" : 2,
|
|
"successful" : 1,
|
|
"failed" : 0
|
|
},
|
|
"_seq_no" : 0,
|
|
"_primary_term" : 1
|
|
}
|
|
--------------------------------------------------
|
|
// TESTRESPONSE
|
|
|
|
//////////////////////////
|
|
[source,js]
|
|
--------------------------------------------------
|
|
GET _alias
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
// TEST[continued]
|
|
//////////////////////////
|
|
|
|
After the rollover, the alias metadata for the two indices will have the `is_write_index` setting
|
|
reflect each index's role, with the newly created index as the write index.
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
{
|
|
"my_logs_index-000002": {
|
|
"aliases": {
|
|
"logs": { "is_write_index": true }
|
|
}
|
|
},
|
|
"my_logs_index-000001": {
|
|
"aliases": {
|
|
"logs": { "is_write_index" : false }
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// TESTRESPONSE
|