143 lines
5.6 KiB
Plaintext
143 lines
5.6 KiB
Plaintext
[[date-index-name-processor]]
|
|
=== Date Index Name Processor
|
|
|
|
The purpose of this processor is to point documents to the right time based index based
|
|
on a date or timestamp field in a document by using the <<date-math-index-names, date math index name support>>.
|
|
|
|
The processor sets the `_index` metadata field with a date math index name expression based on the provided index name
|
|
prefix, a date or timestamp field in the documents being processed and the provided date rounding.
|
|
|
|
First, this processor fetches the date or timestamp from a field in the document being processed. Optionally,
|
|
date formatting can be configured on how the field's value should be parsed into a date. Then this date,
|
|
the provided index name prefix and the provided date rounding get formatted into a date math index name expression.
|
|
Also here optionally date formatting can be specified on how the date should be formatted into a date math index name
|
|
expression.
|
|
|
|
An example pipeline that points documents to a monthly index that starts with a `my-index-` prefix based on a
|
|
date in the `date1` field:
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
PUT _ingest/pipeline/monthlyindex
|
|
{
|
|
"description": "monthly date-time index naming",
|
|
"processors" : [
|
|
{
|
|
"date_index_name" : {
|
|
"field" : "date1",
|
|
"index_name_prefix" : "my-index-",
|
|
"date_rounding" : "M"
|
|
}
|
|
}
|
|
]
|
|
}
|
|
--------------------------------------------------
|
|
|
|
|
|
Using that pipeline for an index request:
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
PUT /my-index/_doc/1?pipeline=monthlyindex
|
|
{
|
|
"date1" : "2016-04-25T12:02:01.789Z"
|
|
}
|
|
--------------------------------------------------
|
|
// TEST[continued]
|
|
|
|
[source,console-result]
|
|
--------------------------------------------------
|
|
{
|
|
"_index" : "my-index-2016-04-01",
|
|
"_type" : "_doc",
|
|
"_id" : "1",
|
|
"_version" : 1,
|
|
"result" : "created",
|
|
"_shards" : {
|
|
"total" : 2,
|
|
"successful" : 1,
|
|
"failed" : 0
|
|
},
|
|
"_seq_no" : 55,
|
|
"_primary_term" : 1
|
|
}
|
|
--------------------------------------------------
|
|
// TESTRESPONSE[s/"_seq_no" : \d+/"_seq_no" : $body._seq_no/ s/"_primary_term" : 1/"_primary_term" : $body._primary_term/]
|
|
|
|
|
|
The above request will not index this document into the `my-index` index, but into the `my-index-2016-04-01` index because
|
|
it was rounded by month. This is because the date-index-name-processor overrides the `_index` property of the document.
|
|
|
|
To see the date-math value of the index supplied in the actual index request which resulted in the above document being
|
|
indexed into `my-index-2016-04-01` we can inspect the effects of the processor using a simulate request.
|
|
|
|
|
|
[source,console]
|
|
--------------------------------------------------
|
|
POST _ingest/pipeline/_simulate
|
|
{
|
|
"pipeline" :
|
|
{
|
|
"description": "monthly date-time index naming",
|
|
"processors" : [
|
|
{
|
|
"date_index_name" : {
|
|
"field" : "date1",
|
|
"index_name_prefix" : "my-index-",
|
|
"date_rounding" : "M"
|
|
}
|
|
}
|
|
]
|
|
},
|
|
"docs": [
|
|
{
|
|
"_source": {
|
|
"date1": "2016-04-25T12:02:01.789Z"
|
|
}
|
|
}
|
|
]
|
|
}
|
|
--------------------------------------------------
|
|
|
|
and the result:
|
|
|
|
[source,console-result]
|
|
--------------------------------------------------
|
|
{
|
|
"docs" : [
|
|
{
|
|
"doc" : {
|
|
"_id" : "_id",
|
|
"_index" : "<my-index-{2016-04-25||/M{yyyy-MM-dd|UTC}}>",
|
|
"_type" : "_doc",
|
|
"_source" : {
|
|
"date1" : "2016-04-25T12:02:01.789Z"
|
|
},
|
|
"_ingest" : {
|
|
"timestamp" : "2016-11-08T19:43:03.850+0000"
|
|
}
|
|
}
|
|
}
|
|
]
|
|
}
|
|
--------------------------------------------------
|
|
// TESTRESPONSE[s/2016-11-08T19:43:03.850\+0000/$body.docs.0.doc._ingest.timestamp/]
|
|
|
|
The above example shows that `_index` was set to `<my-index-{2016-04-25||/M{yyyy-MM-dd|UTC}}>`. Elasticsearch
|
|
understands this to mean `2016-04-01` as is explained in the <<date-math-index-names, date math index name documentation>>
|
|
|
|
[[date-index-name-options]]
|
|
.Date index name options
|
|
[options="header"]
|
|
|======
|
|
| Name | Required | Default | Description
|
|
| `field` | yes | - | The field to get the date or timestamp from.
|
|
| `index_name_prefix` | no | - | A prefix of the index name to be prepended before the printed date. Supports <<accessing-template-fields,template snippets>>.
|
|
| `date_rounding` | yes | - | How to round the date when formatting the date into the index name. Valid values are: `y` (year), `M` (month), `w` (week), `d` (day), `h` (hour), `m` (minute) and `s` (second). Supports <<accessing-template-fields,template snippets>>.
|
|
| `date_formats` | no | yyyy-MM-dd+++'T'+++HH:mm:ss.SSSXX | An array of the expected date formats for parsing dates / timestamps in the document being preprocessed. Can be a java time pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N.
|
|
| `timezone` | no | UTC | The timezone to use when parsing the date and when date math index supports resolves expressions into concrete index names.
|
|
| `locale` | no | ENGLISH | The locale to use when parsing the date from the document being preprocessed, relevant when parsing month names or week days.
|
|
| `index_name_format` | no | yyyy-MM-dd | The format to be used when printing the parsed date into the index name. A valid java time pattern is expected here. Supports <<accessing-template-fields,template snippets>>.
|
|
include::common-options.asciidoc[]
|
|
|======
|