mirror of
https://github.com/honeymoose/OpenSearch.git
synced 2025-02-06 21:18:31 +00:00
2713549533
Today we try to pull stats from index writer but we do not get a consistent view of stats. Under heavy indexing, this inconsistency can be very skewed indeed. In particular, it can lead to the number of deleted docs being reported as negative and this leads to serialization issues. Instead, we should provide a consistent view of the stats by using an index reader. Relates #22317
229 lines
6.0 KiB
Plaintext
229 lines
6.0 KiB
Plaintext
[[indices-rollover-index]]
|
|
== Rollover Index
|
|
|
|
The rollover index API rolls an alias over to a new index when the existing
|
|
index is considered to be too large or too old.
|
|
|
|
The API accepts a single alias name and a list of `conditions`. The alias
|
|
must point to a single index only. If the index satisfies the specified
|
|
conditions then a new index is created and the alias is switched to point to
|
|
the new index.
|
|
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
PUT /logs-000001 <1>
|
|
{
|
|
"aliases": {
|
|
"logs_write": {}
|
|
}
|
|
}
|
|
|
|
# Add > 1000 documents to logs-000001
|
|
|
|
POST /logs_write/_rollover <2>
|
|
{
|
|
"conditions": {
|
|
"max_age": "7d",
|
|
"max_docs": 1000
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
// TEST[setup:huge_twitter]
|
|
// TEST[s/# Add > 1000 documents to logs-000001/POST _reindex?refresh\n{"source":{"index":"twitter"},"dest":{"index":"logs-000001"}}/]
|
|
<1> Creates an index called `logs-0000001` with the alias `logs_write`.
|
|
<2> If the index pointed to by `logs_write` was created 7 or more days ago, or
|
|
contains 1,000 or more documents, then the `logs-000002` index is created
|
|
and the `logs_write` alias is updated to point to `logs-000002`.
|
|
|
|
The above request might return the following response:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
{
|
|
"acknowledged": true,
|
|
"shards_acknowledged": true,
|
|
"old_index": "logs-000001",
|
|
"new_index": "logs-000002",
|
|
"rolled_over": true, <1>
|
|
"dry_run": false, <2>
|
|
"conditions": { <3>
|
|
"[max_age: 7d]": false,
|
|
"[max_docs: 1000]": true
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// TESTRESPONSE
|
|
<1> Whether the index was rolled over.
|
|
<2> Whether the rollover was dry run.
|
|
<3> The result of each condition.
|
|
|
|
[float]
|
|
=== Naming the new index
|
|
|
|
If the name of the existing index ends with `-` and a number -- e.g.
|
|
`logs-000001` -- then the name of the new index will follow the same pattern,
|
|
incrementing the number (`logs-000002`). The number is zero-padded with a length
|
|
of 6, regardless of the old index name.
|
|
|
|
If the old name doesn't match this pattern then you must specify the name for
|
|
the new index as follows:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
POST /my_alias/_rollover/my_new_index_name
|
|
{
|
|
"conditions": {
|
|
"max_age": "7d",
|
|
"max_docs": 1000
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
// TEST[s/^/PUT my_old_index_name\nPUT my_old_index_name\/_alias\/my_alias\n/]
|
|
|
|
[float]
|
|
=== Using date math with the rolllover API
|
|
|
|
It can be useful to use <<date-math-index-names,date math>> to name the
|
|
rollover index according to the date that the index rolled over, e.g.
|
|
`logstash-2016.02.03`. The rollover API supports date math, but requires the
|
|
index name to end with a dash followed by a number, e.g.
|
|
`logstash-2016.02.03-1` which is incremented every time the index is rolled
|
|
over. For instance:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
# PUT /<logs-{now/d}-1> with URI encoding:
|
|
PUT /%3Clogs-%7Bnow%2Fd%7D-1%3E <1>
|
|
{
|
|
"aliases": {
|
|
"logs_write": {}
|
|
}
|
|
}
|
|
|
|
PUT logs_write/log/1
|
|
{
|
|
"message": "a dummy log"
|
|
}
|
|
|
|
POST logs_write/_refresh
|
|
|
|
# Wait for a day to pass
|
|
|
|
POST /logs_write/_rollover <2>
|
|
{
|
|
"conditions": {
|
|
"max_docs": "1"
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
// TEST[s/now/2016.10.31||/]
|
|
<1> Creates an index named with today's date (e.g.) `logs-2016.10.31-1`
|
|
<2> Rolls over to a new index with today's date, e.g. `logs-2016.10.31-000002` if run immediately, or `logs-2016.11.01-000002` if run after 24 hours
|
|
|
|
//////////////////////////
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
GET _alias
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
// TEST[continued]
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
{
|
|
"logs-2016.10.31-000002": {
|
|
"aliases": {
|
|
"logs_write": {}
|
|
}
|
|
},
|
|
"logs-2016.10.31-1": {
|
|
"aliases": {}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// TESTRESPONSE
|
|
|
|
//////////////////////////
|
|
|
|
These indices can then be referenced as described in the
|
|
<<date-math-index-names,date math documentation>>. For example, to search
|
|
over indices created in the last three days, you could do the following:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
# GET /<logs-{now/d}-*>,<logs-{now/d-1d}-*>,<logs-{now/d-2d}-*>/_search
|
|
GET /%3Clogs-%7Bnow%2Fd%7D-*%3E%2C%3Clogs-%7Bnow%2Fd-1d%7D-*%3E%2C%3Clogs-%7Bnow%2Fd-2d%7D-*%3E/_search
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
// TEST[continued]
|
|
// TEST[s/now/2016.10.31||/]
|
|
|
|
[float]
|
|
=== Defining the new index
|
|
|
|
The settings, mappings, and aliases for the new index are taken from any
|
|
matching <<indices-templates,index templates>>. Additionally, you can specify
|
|
`settings`, `mappings`, and `aliases` in the body of the request, just like the
|
|
<<indices-create-index,create index>> API. Values specified in the request
|
|
override any values set in matching index templates. For example, the following
|
|
`rollover` request overrides the `index.number_of_shards` setting:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
PUT /logs-000001
|
|
{
|
|
"aliases": {
|
|
"logs_write": {}
|
|
}
|
|
}
|
|
|
|
POST /logs_write/_rollover
|
|
{
|
|
"conditions" : {
|
|
"max_age": "7d",
|
|
"max_docs": 1000
|
|
},
|
|
"settings": {
|
|
"index.number_of_shards": 2
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
|
|
[float]
|
|
=== Dry run
|
|
|
|
The rollover API supports `dry_run` mode, where request conditions can be
|
|
checked without performing the actual rollover:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
PUT /logs-000001
|
|
{
|
|
"aliases": {
|
|
"logs_write": {}
|
|
}
|
|
}
|
|
|
|
POST /logs_write/_rollover?dry_run
|
|
{
|
|
"conditions" : {
|
|
"max_age": "7d",
|
|
"max_docs": 1000
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
|
|
[float]
|
|
=== Wait For Active Shards
|
|
|
|
Because the rollover operation creates a new index to rollover to, the
|
|
<<create-index-wait-for-active-shards,`wait_for_active_shards`>> setting on
|
|
index creation applies to the rollover action as well.
|