|
|
|
|
@ -0,0 +1,410 @@
|
|
|
|
|
[[migrate-to-java-time]]
|
|
|
|
|
=== Java time migration guide
|
|
|
|
|
|
|
|
|
|
With 7.0, {es} switched from joda time to java time for date-related parsing,
|
|
|
|
|
formatting, and calculations. This guide is designed to help you determine
|
|
|
|
|
if your cluster is impacted and, if so, prepare for the upgrade.
|
|
|
|
|
|
|
|
|
|
[discrete]
|
|
|
|
|
[[java-time-convert-date-formats]]
|
|
|
|
|
=== Convert date formats
|
|
|
|
|
|
|
|
|
|
To upgrade to 7.0, you'll need to convert any joda-time date formats
|
|
|
|
|
to their java-time equivalents.
|
|
|
|
|
|
|
|
|
|
To help track this effort, you can prefix java-time date formats with an `8`
|
|
|
|
|
in {es} 6.8 and later versions.
|
|
|
|
|
|
|
|
|
|
For example, you can change the date format `YYYY-MM-dd` to `8yyyy-MM-dd` to
|
|
|
|
|
indicate the date format uses java time.
|
|
|
|
|
|
|
|
|
|
{es} treats date formats starting with the `8` prefix differently depending on
|
|
|
|
|
the version:
|
|
|
|
|
|
|
|
|
|
*6.8*: Date formats with an `8` prefix are handled as java-time formats. Date
|
|
|
|
|
formats without an `8` prefix are treated as joda-time formats. We recommend
|
|
|
|
|
converting these joda-time formats to java-time _before_ upgrading to 7.x.
|
|
|
|
|
|
|
|
|
|
*7.x and later*: For indices created in 6.x, date formats without an `8` prefix
|
|
|
|
|
are treated as joda-time formats. For indices created in 7.x and later versions,
|
|
|
|
|
all date formats are treated as java-time formats, regardless of whether it
|
|
|
|
|
starts with an `8` prefix.
|
|
|
|
|
|
|
|
|
|
[[java-time-migration-impacted-features]]
|
|
|
|
|
==== Impacted features
|
|
|
|
|
The switch to java time only impacts custom <<date,`date`>> and
|
|
|
|
|
<<date_nanos,`date_nanos`>> formats.
|
|
|
|
|
|
|
|
|
|
These formats are commonly used in:
|
|
|
|
|
|
|
|
|
|
* <<mapping,Index mappings>>
|
|
|
|
|
* <<indices-templates,Index templates>>
|
|
|
|
|
* <<pipeline,Ingest pipelines>>
|
|
|
|
|
|
|
|
|
|
If you don't use custom date formats, you can skip the rest of this guide.
|
|
|
|
|
Most custom date formats are compatible. However, several require
|
|
|
|
|
an update.
|
|
|
|
|
|
|
|
|
|
To see if your date format is impacted, use the <<migration-api-deprecation,deprecation info API>>
|
|
|
|
|
or the {kibana-ref}/upgrade-assistant.html[Kibana upgrade assistant].
|
|
|
|
|
|
|
|
|
|
[[java-time-migration-incompatible-date-formats]]
|
|
|
|
|
==== Incompatible date formats
|
|
|
|
|
Custom date formats containing the following joda-time literals should be
|
|
|
|
|
changed before upgrading.
|
|
|
|
|
|
|
|
|
|
`Y` (Year of era)::
|
|
|
|
|
+
|
|
|
|
|
--
|
|
|
|
|
Replace with `y`.
|
|
|
|
|
|
|
|
|
|
*Example:*
|
|
|
|
|
`YYYY-MM-dd` should become `yyyy-MM-dd`.
|
|
|
|
|
|
|
|
|
|
In java time, `Y` is used for
|
|
|
|
|
https://docs.oracle.com/javase/8/docs/api/java/time/temporal/WeekFields.html[week-based year].
|
|
|
|
|
Using `Y` in place of `y` could result in off-by-one errors in year calculation.
|
|
|
|
|
|
|
|
|
|
For pattern `YYYY-ww` and date `2019-01-01T00:00:00.000Z` will give `2019-01`
|
|
|
|
|
For pattern `YYYY-ww` and date `2018-12-31T00:00:00.000Z` will give `2019-01` (counter-intuitive) because there is >4 days of that week in 2019
|
|
|
|
|
--
|
|
|
|
|
|
|
|
|
|
`y` (Year)::
|
|
|
|
|
+
|
|
|
|
|
--
|
|
|
|
|
Replace with `u`.
|
|
|
|
|
|
|
|
|
|
*Example:*
|
|
|
|
|
`yyyy-MM-dd` should become `uuuu-MM-dd`.
|
|
|
|
|
|
|
|
|
|
In java time, `y` is used for year of era. `u` can contain non-positive
|
|
|
|
|
values while `y` cannot. `y` can also be associated with an era field.
|
|
|
|
|
--
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
`C` (Century of era)::
|
|
|
|
|
+
|
|
|
|
|
--
|
|
|
|
|
Century of era is not supported in java time.
|
|
|
|
|
There is no replacement. Instead, we recommend you preprocess your input.
|
|
|
|
|
--
|
|
|
|
|
|
|
|
|
|
`x` (Week year)::
|
|
|
|
|
+
|
|
|
|
|
--
|
|
|
|
|
Replace with `Y`.
|
|
|
|
|
|
|
|
|
|
In java time, `x` means https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html[zone-offset].
|
|
|
|
|
|
|
|
|
|
[WARNING]
|
|
|
|
|
====
|
|
|
|
|
Failure to properly convert `x` (Week year) to `Y` could result in data loss.
|
|
|
|
|
====
|
|
|
|
|
--
|
|
|
|
|
|
|
|
|
|
`Z` (Zone offset/id)::
|
|
|
|
|
+
|
|
|
|
|
--
|
|
|
|
|
Replace with multiple `X`'s.
|
|
|
|
|
|
|
|
|
|
`Z` has a similar meaning in java time. However, java time expects different
|
|
|
|
|
numbers of literals to parse different forms.
|
|
|
|
|
|
|
|
|
|
Consider migrating to `X`, which gives you more control over how time is parsed.
|
|
|
|
|
For example, the joda-time format `YYYY-MM-dd'T'hh:mm:ssZZ` accepts the following dates:
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
2010-01-01T01:02:03Z
|
|
|
|
|
2010-01-01T01:02:03+01
|
|
|
|
|
2010-01-01T01:02:03+01:02
|
|
|
|
|
2010-01-01T01:02:03+01:02:03
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
In java time, you cannot parse all these dates using a single format
|
|
|
|
|
Instead, you must specify 3 separate formats:
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
2010-01-01T01:02:03Z
|
|
|
|
|
2010-01-01T01:02:03+01
|
|
|
|
|
both parsed with yyyy-MM-dd'T'hh:mm:ssX
|
|
|
|
|
|
|
|
|
|
2010-01-01T01:02:03+01:02
|
|
|
|
|
yyyy-MM-dd'T'hh:mm:ssXXX
|
|
|
|
|
|
|
|
|
|
2010-01-01T01:02:03+01:02:03
|
|
|
|
|
yyyy-MM-dd'T'hh:mm:ssXXXXX
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The formats must then be delimited using `||`:
|
|
|
|
|
[source,txt]
|
|
|
|
|
--------------------------------------------------
|
|
|
|
|
yyyy-MM-dd'T'hh:mm:ssX||yyyy-MM-dd'T'hh:mm:ssXXX||yyyy-MM-dd'T'hh:mm:ssXXXXX
|
|
|
|
|
--------------------------------------------------
|
|
|
|
|
|
|
|
|
|
The same applies if you expect your pattern to occur without a colon (`:`):
|
|
|
|
|
For example, the `YYYY-MM-dd'T'hh:mm:ssZ` format accepts the following date forms:
|
|
|
|
|
```
|
|
|
|
|
2010-01-01T01:02:03Z
|
|
|
|
|
2010-01-01T01:02:03+01
|
|
|
|
|
2010-01-01T01:02:03+0102
|
|
|
|
|
2010-01-01T01:02:03+010203
|
|
|
|
|
```
|
|
|
|
|
To accept all these forms in java time, you must use the `||` delimiter:
|
|
|
|
|
[source,txt]
|
|
|
|
|
--------------------------------------------------
|
|
|
|
|
yyyy-MM-dd'T'hh:mm:ssX||yyyy-MM-dd'T'hh:mm:ssXX||yyyy-MM-dd'T'hh:mm:ssXXXX
|
|
|
|
|
--------------------------------------------------
|
|
|
|
|
--
|
|
|
|
|
|
|
|
|
|
`d` (Day)::
|
|
|
|
|
+
|
|
|
|
|
--
|
|
|
|
|
In java time, `d` is still interpreted as "day" but is less flexible.
|
|
|
|
|
|
|
|
|
|
For example, the joda-time date format `YYYY-MM-dd` accepts `2010-01-01` or
|
|
|
|
|
`2010-01-1`.
|
|
|
|
|
|
|
|
|
|
In java time, you must use the `||` delimiter to provide specify each format:
|
|
|
|
|
|
|
|
|
|
[source,txt]
|
|
|
|
|
--------------------------------------------------
|
|
|
|
|
yyyy-MM-dd||yyyy-MM-d
|
|
|
|
|
--------------------------------------------------
|
|
|
|
|
|
|
|
|
|
In java time, `d` also does not accept more than 2 digits. To accept days with more
|
|
|
|
|
than two digits, you must include a text literal in your java-time date format.
|
|
|
|
|
For example, to parse `2010-01-00001`, you must use the following java-time date format:
|
|
|
|
|
|
|
|
|
|
[source,txt]
|
|
|
|
|
--------------------------------------------------
|
|
|
|
|
yyyy-MM-'000'dd
|
|
|
|
|
--------------------------------------------------
|
|
|
|
|
--
|
|
|
|
|
|
|
|
|
|
`e` (Name of day)::
|
|
|
|
|
+
|
|
|
|
|
--
|
|
|
|
|
In java time, `e` is still interpreted as "name of day" but does not parse
|
|
|
|
|
short- or full-text forms.
|
|
|
|
|
|
|
|
|
|
For example, the joda-time date format `EEE YYYY-MM` accepts both
|
|
|
|
|
`Wed 2020-01` and `Wednesday 2020-01`.
|
|
|
|
|
|
|
|
|
|
To accept both of these dates in java time, you must specify each format using
|
|
|
|
|
the `||` delimiter:
|
|
|
|
|
|
|
|
|
|
[source,txt]
|
|
|
|
|
--------------------------------------------------
|
|
|
|
|
cccc yyyy-MM||ccc yyyy-MM
|
|
|
|
|
--------------------------------------------------
|
|
|
|
|
|
|
|
|
|
The joda-time literal `E` is interpreted as "day of week."
|
|
|
|
|
The java-time literal `c` is interpreted as "localized day of week."
|
|
|
|
|
`E` does not accept full-text day formats, such as `Wednesday`.
|
|
|
|
|
--
|
|
|
|
|
|
|
|
|
|
`EEEE` and similar text forms::
|
|
|
|
|
+
|
|
|
|
|
--
|
|
|
|
|
Support for full-text forms depends on the locale data provided with your Java
|
|
|
|
|
Development Kit (JDK) and other implementation details. We recommend you
|
|
|
|
|
test formats containing these patterns carefully before upgrading.
|
|
|
|
|
--
|
|
|
|
|
|
|
|
|
|
`z` (Time zone text)::
|
|
|
|
|
+
|
|
|
|
|
--
|
|
|
|
|
In java time, `z` outputs 'Z' for Zulu when given a UTC timezone.
|
|
|
|
|
--
|
|
|
|
|
|
|
|
|
|
[[java-time-migration-test]]
|
|
|
|
|
===== Test with your data
|
|
|
|
|
|
|
|
|
|
We strongly recommend you test any date format changes using real data before
|
|
|
|
|
deploying in production.
|
|
|
|
|
|
|
|
|
|
For help with date debugging, consider using
|
|
|
|
|
https://esddd.herokuapp.com/[https://esddd.herokuapp.com/.]
|
|
|
|
|
|
|
|
|
|
[[java-time-migrate-update-mappings]]
|
|
|
|
|
==== Update index mappings
|
|
|
|
|
To update joda-time date formats in index mappings, you must create a new index
|
|
|
|
|
with an updated mapping and reindex your data to it.
|
|
|
|
|
You can however update your pipelines or templates.
|
|
|
|
|
|
|
|
|
|
The following `my_index_1` index contains a mapping for the `datetime` field, a
|
|
|
|
|
`date` field with a custom joda-time date format.
|
|
|
|
|
////
|
|
|
|
|
[source,console]
|
|
|
|
|
--------------------------------------------------
|
|
|
|
|
PUT my_index_1
|
|
|
|
|
{
|
|
|
|
|
"mappings" : {
|
|
|
|
|
"properties" : {
|
|
|
|
|
"datetime": {
|
|
|
|
|
"type": "date",
|
|
|
|
|
"format": "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis"
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
--------------------------------------------------
|
|
|
|
|
////
|
|
|
|
|
|
|
|
|
|
[source,console]
|
|
|
|
|
--------------------------------------------------
|
|
|
|
|
GET my_index_1/_mapping
|
|
|
|
|
--------------------------------------------------
|
|
|
|
|
// TEST[continued]
|
|
|
|
|
|
|
|
|
|
[source,console-result]
|
|
|
|
|
--------------------------------------------------
|
|
|
|
|
{
|
|
|
|
|
"my_index_1" : {
|
|
|
|
|
"mappings" : {
|
|
|
|
|
"properties" : {
|
|
|
|
|
"datetime": {
|
|
|
|
|
"type": "date",
|
|
|
|
|
"format": "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis"
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
--------------------------------------------------
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
To change the date format for the `datetime` field, create a separate index
|
|
|
|
|
containing an updated mapping and date format.
|
|
|
|
|
|
|
|
|
|
For example, the following `my_index_2` index changes the `datetime` field's
|
|
|
|
|
date format to `8uuuu/MM/dd HH:mm:ss||uuuu/MM/dd||epoch_millis`. The `8` prefix
|
|
|
|
|
indicates this date format uses java time.
|
|
|
|
|
|
|
|
|
|
[source,console]
|
|
|
|
|
--------------------------------------------------
|
|
|
|
|
PUT my_index_2
|
|
|
|
|
{
|
|
|
|
|
"mappings": {
|
|
|
|
|
"properties": {
|
|
|
|
|
"datetime": {
|
|
|
|
|
"type": "date",
|
|
|
|
|
"format": "8uuuu/MM/dd HH:mm:ss||uuuu/MM/dd||epoch_millis"
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
--------------------------------------------------
|
|
|
|
|
// TEST[continued]
|
|
|
|
|
|
|
|
|
|
Next, reindex data from the old index to the new index.
|
|
|
|
|
|
|
|
|
|
The following <<docs-reindex,reindex>> API request reindexes data from
|
|
|
|
|
`my_index_1` to `my_index_2`.
|
|
|
|
|
|
|
|
|
|
[source,console]
|
|
|
|
|
--------------------------------------------------
|
|
|
|
|
POST _reindex
|
|
|
|
|
{
|
|
|
|
|
"source": {
|
|
|
|
|
"index": "my_index_1"
|
|
|
|
|
},
|
|
|
|
|
"dest": {
|
|
|
|
|
"index": "my_index_2"
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
--------------------------------------------------
|
|
|
|
|
// TEST[continued]
|
|
|
|
|
|
|
|
|
|
If you use index aliases, update them to point to the new index.
|
|
|
|
|
|
|
|
|
|
[source,console]
|
|
|
|
|
--------------------------------------------------
|
|
|
|
|
POST /_aliases
|
|
|
|
|
{
|
|
|
|
|
"actions" : [
|
|
|
|
|
{ "remove" : { "index" : "my_index_1", "alias" : "my_index" } },
|
|
|
|
|
{ "add" : { "index" : "my_index_2", "alias" : "my_index" } }
|
|
|
|
|
]
|
|
|
|
|
}
|
|
|
|
|
--------------------------------------------------
|
|
|
|
|
// TEST[continued]
|
|
|
|
|
|
|
|
|
|
[[java-time-migration-update-ingest-pipelines]]
|
|
|
|
|
===== Update ingest pipelines
|
|
|
|
|
If your ingest pipelines contain joda-time date formats, you can update them
|
|
|
|
|
using the <<put-pipeline-api,put ingest pipeline>> API.
|
|
|
|
|
|
|
|
|
|
[source,console]
|
|
|
|
|
--------------------------------------------------
|
|
|
|
|
PUT _ingest/pipeline/my_pipeline
|
|
|
|
|
{
|
|
|
|
|
"description": "Pipeline for routing data to specific index",
|
|
|
|
|
"processors": [
|
|
|
|
|
{
|
|
|
|
|
"date": {
|
|
|
|
|
"field": "createdTime",
|
|
|
|
|
"formats": [
|
|
|
|
|
"8uuuu-w"
|
|
|
|
|
]
|
|
|
|
|
},
|
|
|
|
|
"date_index_name": {
|
|
|
|
|
"field": "@timestamp",
|
|
|
|
|
"date_rounding": "d",
|
|
|
|
|
"index_name_prefix": "x-",
|
|
|
|
|
"index_name_format": "8uuuu-w"
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
]
|
|
|
|
|
}
|
|
|
|
|
--------------------------------------------------
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
[[java-time-migration-update-index-templates]]
|
|
|
|
|
===== Update index templates
|
|
|
|
|
|
|
|
|
|
If your index templates contain joda-time date formats, you can update them
|
|
|
|
|
using the <<indices-templates,put index template>> API.
|
|
|
|
|
|
|
|
|
|
[source,console]
|
|
|
|
|
--------------------------------------------------
|
|
|
|
|
PUT _template/template_1
|
|
|
|
|
{
|
|
|
|
|
"index_patterns": [
|
|
|
|
|
"te*",
|
|
|
|
|
"bar*"
|
|
|
|
|
],
|
|
|
|
|
"settings": {
|
|
|
|
|
"number_of_shards": 1
|
|
|
|
|
},
|
|
|
|
|
"mappings": {
|
|
|
|
|
"_source": {
|
|
|
|
|
"enabled": false
|
|
|
|
|
},
|
|
|
|
|
"properties": {
|
|
|
|
|
"host_name": {
|
|
|
|
|
"type": "keyword"
|
|
|
|
|
},
|
|
|
|
|
"created_at": {
|
|
|
|
|
"type": "date",
|
|
|
|
|
"format": "8EEE MMM dd HH:mm:ss Z yyyy"
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
--------------------------------------------------
|
|
|
|
|
|
|
|
|
|
////
|
|
|
|
|
[source,console]
|
|
|
|
|
--------------------------------------------------
|
|
|
|
|
DELETE /_template/template_1
|
|
|
|
|
--------------------------------------------------
|
|
|
|
|
// TEST[continued]
|
|
|
|
|
////
|
|
|
|
|
|
|
|
|
|
[[java-time-migration-update-external-tools-templates]]
|
|
|
|
|
===== Update external tools and templates
|
|
|
|
|
Ensure you also update any date formats in templates or tools outside of {es}.
|
|
|
|
|
This can include tools such as {beats-ref}/getting-started.html[{beats}] or
|
|
|
|
|
{logstash-ref}/index.html[Logstash].
|
Przemyslaw Gomulka
GitHub