2013-11-24 06:13:08 -05:00
[[search-aggregations-bucket-datehistogram-aggregation]]
2019-11-22 11:31:30 -05:00
=== Date histogram aggregation
2013-11-24 06:13:08 -05:00
2018-10-31 10:16:36 -04:00
This multi-bucket aggregation is similar to the normal
<<search-aggregations-bucket-histogram-aggregation,histogram>>, but it can
2020-07-20 10:51:23 -04:00
only be used with date or date range values. Because dates are represented internally in
2018-10-31 10:16:36 -04:00
Elasticsearch as long values, it is possible, but not as accurate, to use the
normal `histogram` on dates as well. The main difference in the two APIs is
that here the interval can be specified using date/time expressions. Time-based
data requires special support because time-based intervals are not always a
fixed length.
2020-05-29 17:40:40 -04:00
Like the histogram, values are rounded *down* into the closest bucket. For
example, if the interval is a calendar day, `2020-01-03T07:00:01Z` is rounded to
`2020-01-03T00:00:00Z`. Values are rounded as follows:
[source,java]
----
bucket_key = Math.floor(value / interval) * interval)
----
2019-11-22 11:31:30 -05:00
[[calendar_and_fixed_intervals]]
==== Calendar and fixed intervals
2018-10-31 10:16:36 -04:00
[7.x Backport] Force selection of calendar or fixed intervals (#41906)
The date_histogram accepts an interval which can be either a calendar
interval (DST-aware, leap seconds, arbitrary length of months, etc) or
fixed interval (strict multiples of SI units). Unfortunately this is inferred
by first trying to parse as a calendar interval, then falling back to fixed
if that fails.
This leads to confusing arrangement where `1d` == calendar, but
`2d` == fixed. And if you want a day of fixed time, you have to
specify `24h` (e.g. the next smallest unit). This arrangement is very
error-prone for users.
This PR adds `calendar_interval` and `fixed_interval` parameters to any
code that uses intervals (date_histogram, rollup, composite, datafeed, etc).
Calendar only accepts calendar intervals, fixed accepts any combination of
units (meaning `1d` can be used to specify `24h` in fixed time), and both
are mutually exclusive.
The old interval behavior is deprecated and will throw a deprecation warning.
It is also mutually exclusive with the two new parameters. In the future the
old dual-purpose interval will be removed.
The change applies to both REST and java clients.
2019-05-20 12:07:29 -04:00
When configuring a date histogram aggregation, the interval can be specified
in two manners: calendar-aware time intervals, and fixed time intervals.
2018-10-31 10:16:36 -04:00
[7.x Backport] Force selection of calendar or fixed intervals (#41906)
The date_histogram accepts an interval which can be either a calendar
interval (DST-aware, leap seconds, arbitrary length of months, etc) or
fixed interval (strict multiples of SI units). Unfortunately this is inferred
by first trying to parse as a calendar interval, then falling back to fixed
if that fails.
This leads to confusing arrangement where `1d` == calendar, but
`2d` == fixed. And if you want a day of fixed time, you have to
specify `24h` (e.g. the next smallest unit). This arrangement is very
error-prone for users.
This PR adds `calendar_interval` and `fixed_interval` parameters to any
code that uses intervals (date_histogram, rollup, composite, datafeed, etc).
Calendar only accepts calendar intervals, fixed accepts any combination of
units (meaning `1d` can be used to specify `24h` in fixed time), and both
are mutually exclusive.
The old interval behavior is deprecated and will throw a deprecation warning.
It is also mutually exclusive with the two new parameters. In the future the
old dual-purpose interval will be removed.
The change applies to both REST and java clients.
2019-05-20 12:07:29 -04:00
Calendar-aware intervals understand that daylight savings changes the length
of specific days, months have different amounts of days, and leap seconds can
be tacked onto a particular year.
2018-10-31 10:16:36 -04:00
[7.x Backport] Force selection of calendar or fixed intervals (#41906)
The date_histogram accepts an interval which can be either a calendar
interval (DST-aware, leap seconds, arbitrary length of months, etc) or
fixed interval (strict multiples of SI units). Unfortunately this is inferred
by first trying to parse as a calendar interval, then falling back to fixed
if that fails.
This leads to confusing arrangement where `1d` == calendar, but
`2d` == fixed. And if you want a day of fixed time, you have to
specify `24h` (e.g. the next smallest unit). This arrangement is very
error-prone for users.
This PR adds `calendar_interval` and `fixed_interval` parameters to any
code that uses intervals (date_histogram, rollup, composite, datafeed, etc).
Calendar only accepts calendar intervals, fixed accepts any combination of
units (meaning `1d` can be used to specify `24h` in fixed time), and both
are mutually exclusive.
The old interval behavior is deprecated and will throw a deprecation warning.
It is also mutually exclusive with the two new parameters. In the future the
old dual-purpose interval will be removed.
The change applies to both REST and java clients.
2019-05-20 12:07:29 -04:00
Fixed intervals are, by contrast, always multiples of SI units and do not change
based on calendaring context.
[NOTE]
.Combined `interval` field is deprecated
==================================
deprecated[7.2, `interval` field is deprecated] Historically both calendar and fixed
intervals were configured in a single `interval` field, which led to confusing
semantics. Specifying `1d` would be assumed as a calendar-aware time,
whereas `2d` would be interpreted as fixed time. To get "one day" of fixed time,
the user would need to specify the next smaller unit (in this case, `24h`).
This combined behavior was often unknown to users, and even when knowledgeable about
the behavior it was difficult to use and confusing.
This behavior has been deprecated in favor of two new, explicit fields: `calendar_interval`
and `fixed_interval`.
By forcing a choice between calendar and intervals up front, the semantics of the interval
are clear to the user immediately and there is no ambiguity. The old `interval` field
will be removed in the future.
==================================
2019-11-22 11:31:30 -05:00
[[calendar_intervals]]
===== Calendar intervals
[7.x Backport] Force selection of calendar or fixed intervals (#41906)
The date_histogram accepts an interval which can be either a calendar
interval (DST-aware, leap seconds, arbitrary length of months, etc) or
fixed interval (strict multiples of SI units). Unfortunately this is inferred
by first trying to parse as a calendar interval, then falling back to fixed
if that fails.
This leads to confusing arrangement where `1d` == calendar, but
`2d` == fixed. And if you want a day of fixed time, you have to
specify `24h` (e.g. the next smallest unit). This arrangement is very
error-prone for users.
This PR adds `calendar_interval` and `fixed_interval` parameters to any
code that uses intervals (date_histogram, rollup, composite, datafeed, etc).
Calendar only accepts calendar intervals, fixed accepts any combination of
units (meaning `1d` can be used to specify `24h` in fixed time), and both
are mutually exclusive.
The old interval behavior is deprecated and will throw a deprecation warning.
It is also mutually exclusive with the two new parameters. In the future the
old dual-purpose interval will be removed.
The change applies to both REST and java clients.
2019-05-20 12:07:29 -04:00
Calendar-aware intervals are configured with the `calendar_interval` parameter.
2020-05-29 17:40:40 -04:00
You can specify calendar intervals using the unit name, such as `month`, or as a
single unit quantity, such as `1M`. For example, `day` and `1d` are equivalent.
Multiple quantities, such as `2d`, are not supported.
[7.x Backport] Force selection of calendar or fixed intervals (#41906)
The date_histogram accepts an interval which can be either a calendar
interval (DST-aware, leap seconds, arbitrary length of months, etc) or
fixed interval (strict multiples of SI units). Unfortunately this is inferred
by first trying to parse as a calendar interval, then falling back to fixed
if that fails.
This leads to confusing arrangement where `1d` == calendar, but
`2d` == fixed. And if you want a day of fixed time, you have to
specify `24h` (e.g. the next smallest unit). This arrangement is very
error-prone for users.
This PR adds `calendar_interval` and `fixed_interval` parameters to any
code that uses intervals (date_histogram, rollup, composite, datafeed, etc).
Calendar only accepts calendar intervals, fixed accepts any combination of
units (meaning `1d` can be used to specify `24h` in fixed time), and both
are mutually exclusive.
The old interval behavior is deprecated and will throw a deprecation warning.
It is also mutually exclusive with the two new parameters. In the future the
old dual-purpose interval will be removed.
The change applies to both REST and java clients.
2019-05-20 12:07:29 -04:00
2020-05-29 17:40:40 -04:00
The accepted calendar intervals are:
[7.x Backport] Force selection of calendar or fixed intervals (#41906)
The date_histogram accepts an interval which can be either a calendar
interval (DST-aware, leap seconds, arbitrary length of months, etc) or
fixed interval (strict multiples of SI units). Unfortunately this is inferred
by first trying to parse as a calendar interval, then falling back to fixed
if that fails.
This leads to confusing arrangement where `1d` == calendar, but
`2d` == fixed. And if you want a day of fixed time, you have to
specify `24h` (e.g. the next smallest unit). This arrangement is very
error-prone for users.
This PR adds `calendar_interval` and `fixed_interval` parameters to any
code that uses intervals (date_histogram, rollup, composite, datafeed, etc).
Calendar only accepts calendar intervals, fixed accepts any combination of
units (meaning `1d` can be used to specify `24h` in fixed time), and both
are mutually exclusive.
The old interval behavior is deprecated and will throw a deprecation warning.
It is also mutually exclusive with the two new parameters. In the future the
old dual-purpose interval will be removed.
The change applies to both REST and java clients.
2019-05-20 12:07:29 -04:00
2020-05-29 17:40:40 -04:00
`minute`, `1m` ::
2018-10-31 10:16:36 -04:00
2020-05-15 16:52:09 -04:00
All minutes begin at 00 seconds.
[7.x Backport] Force selection of calendar or fixed intervals (#41906)
The date_histogram accepts an interval which can be either a calendar
interval (DST-aware, leap seconds, arbitrary length of months, etc) or
fixed interval (strict multiples of SI units). Unfortunately this is inferred
by first trying to parse as a calendar interval, then falling back to fixed
if that fails.
This leads to confusing arrangement where `1d` == calendar, but
`2d` == fixed. And if you want a day of fixed time, you have to
specify `24h` (e.g. the next smallest unit). This arrangement is very
error-prone for users.
This PR adds `calendar_interval` and `fixed_interval` parameters to any
code that uses intervals (date_histogram, rollup, composite, datafeed, etc).
Calendar only accepts calendar intervals, fixed accepts any combination of
units (meaning `1d` can be used to specify `24h` in fixed time), and both
are mutually exclusive.
The old interval behavior is deprecated and will throw a deprecation warning.
It is also mutually exclusive with the two new parameters. In the future the
old dual-purpose interval will be removed.
The change applies to both REST and java clients.
2019-05-20 12:07:29 -04:00
One minute is the interval between 00 seconds of the first minute and 00
2020-05-29 17:40:40 -04:00
seconds of the following minute in the specified time zone, compensating for any
[7.x Backport] Force selection of calendar or fixed intervals (#41906)
The date_histogram accepts an interval which can be either a calendar
interval (DST-aware, leap seconds, arbitrary length of months, etc) or
fixed interval (strict multiples of SI units). Unfortunately this is inferred
by first trying to parse as a calendar interval, then falling back to fixed
if that fails.
This leads to confusing arrangement where `1d` == calendar, but
`2d` == fixed. And if you want a day of fixed time, you have to
specify `24h` (e.g. the next smallest unit). This arrangement is very
error-prone for users.
This PR adds `calendar_interval` and `fixed_interval` parameters to any
code that uses intervals (date_histogram, rollup, composite, datafeed, etc).
Calendar only accepts calendar intervals, fixed accepts any combination of
units (meaning `1d` can be used to specify `24h` in fixed time), and both
are mutually exclusive.
The old interval behavior is deprecated and will throw a deprecation warning.
It is also mutually exclusive with the two new parameters. In the future the
old dual-purpose interval will be removed.
The change applies to both REST and java clients.
2019-05-20 12:07:29 -04:00
intervening leap seconds, so that the number of minutes and seconds past the
hour is the same at the start and end.
2018-10-31 10:16:36 -04:00
2020-05-29 17:40:40 -04:00
`hour`, `1h` ::
2018-10-31 10:16:36 -04:00
2020-05-15 16:52:09 -04:00
All hours begin at 00 minutes and 00 seconds.
[7.x Backport] Force selection of calendar or fixed intervals (#41906)
The date_histogram accepts an interval which can be either a calendar
interval (DST-aware, leap seconds, arbitrary length of months, etc) or
fixed interval (strict multiples of SI units). Unfortunately this is inferred
by first trying to parse as a calendar interval, then falling back to fixed
if that fails.
This leads to confusing arrangement where `1d` == calendar, but
`2d` == fixed. And if you want a day of fixed time, you have to
specify `24h` (e.g. the next smallest unit). This arrangement is very
error-prone for users.
This PR adds `calendar_interval` and `fixed_interval` parameters to any
code that uses intervals (date_histogram, rollup, composite, datafeed, etc).
Calendar only accepts calendar intervals, fixed accepts any combination of
units (meaning `1d` can be used to specify `24h` in fixed time), and both
are mutually exclusive.
The old interval behavior is deprecated and will throw a deprecation warning.
It is also mutually exclusive with the two new parameters. In the future the
old dual-purpose interval will be removed.
The change applies to both REST and java clients.
2019-05-20 12:07:29 -04:00
One hour (1h) is the interval between 00:00 minutes of the first hour and 00:00
2020-05-29 17:40:40 -04:00
minutes of the following hour in the specified time zone, compensating for any
2018-10-31 10:16:36 -04:00
intervening leap seconds, so that the number of minutes and seconds past the hour
[7.x Backport] Force selection of calendar or fixed intervals (#41906)
The date_histogram accepts an interval which can be either a calendar
interval (DST-aware, leap seconds, arbitrary length of months, etc) or
fixed interval (strict multiples of SI units). Unfortunately this is inferred
by first trying to parse as a calendar interval, then falling back to fixed
if that fails.
This leads to confusing arrangement where `1d` == calendar, but
`2d` == fixed. And if you want a day of fixed time, you have to
specify `24h` (e.g. the next smallest unit). This arrangement is very
error-prone for users.
This PR adds `calendar_interval` and `fixed_interval` parameters to any
code that uses intervals (date_histogram, rollup, composite, datafeed, etc).
Calendar only accepts calendar intervals, fixed accepts any combination of
units (meaning `1d` can be used to specify `24h` in fixed time), and both
are mutually exclusive.
The old interval behavior is deprecated and will throw a deprecation warning.
It is also mutually exclusive with the two new parameters. In the future the
old dual-purpose interval will be removed.
The change applies to both REST and java clients.
2019-05-20 12:07:29 -04:00
is the same at the start and end.
2018-10-31 10:16:36 -04:00
2020-05-29 17:40:40 -04:00
`day`, `1d` ::
[7.x Backport] Force selection of calendar or fixed intervals (#41906)
The date_histogram accepts an interval which can be either a calendar
interval (DST-aware, leap seconds, arbitrary length of months, etc) or
fixed interval (strict multiples of SI units). Unfortunately this is inferred
by first trying to parse as a calendar interval, then falling back to fixed
if that fails.
This leads to confusing arrangement where `1d` == calendar, but
`2d` == fixed. And if you want a day of fixed time, you have to
specify `24h` (e.g. the next smallest unit). This arrangement is very
error-prone for users.
This PR adds `calendar_interval` and `fixed_interval` parameters to any
code that uses intervals (date_histogram, rollup, composite, datafeed, etc).
Calendar only accepts calendar intervals, fixed accepts any combination of
units (meaning `1d` can be used to specify `24h` in fixed time), and both
are mutually exclusive.
The old interval behavior is deprecated and will throw a deprecation warning.
It is also mutually exclusive with the two new parameters. In the future the
old dual-purpose interval will be removed.
The change applies to both REST and java clients.
2019-05-20 12:07:29 -04:00
2018-10-31 10:16:36 -04:00
All days begin at the earliest possible time, which is usually 00:00:00
(midnight).
[7.x Backport] Force selection of calendar or fixed intervals (#41906)
The date_histogram accepts an interval which can be either a calendar
interval (DST-aware, leap seconds, arbitrary length of months, etc) or
fixed interval (strict multiples of SI units). Unfortunately this is inferred
by first trying to parse as a calendar interval, then falling back to fixed
if that fails.
This leads to confusing arrangement where `1d` == calendar, but
`2d` == fixed. And if you want a day of fixed time, you have to
specify `24h` (e.g. the next smallest unit). This arrangement is very
error-prone for users.
This PR adds `calendar_interval` and `fixed_interval` parameters to any
code that uses intervals (date_histogram, rollup, composite, datafeed, etc).
Calendar only accepts calendar intervals, fixed accepts any combination of
units (meaning `1d` can be used to specify `24h` in fixed time), and both
are mutually exclusive.
The old interval behavior is deprecated and will throw a deprecation warning.
It is also mutually exclusive with the two new parameters. In the future the
old dual-purpose interval will be removed.
The change applies to both REST and java clients.
2019-05-20 12:07:29 -04:00
One day (1d) is the interval between the start of the day and the start of
2020-05-29 17:40:40 -04:00
of the following day in the specified time zone, compensating for any intervening
2018-10-31 10:16:36 -04:00
time changes.
2020-05-29 17:40:40 -04:00
`week`, `1w` ::
2018-10-31 10:16:36 -04:00
[7.x Backport] Force selection of calendar or fixed intervals (#41906)
The date_histogram accepts an interval which can be either a calendar
interval (DST-aware, leap seconds, arbitrary length of months, etc) or
fixed interval (strict multiples of SI units). Unfortunately this is inferred
by first trying to parse as a calendar interval, then falling back to fixed
if that fails.
This leads to confusing arrangement where `1d` == calendar, but
`2d` == fixed. And if you want a day of fixed time, you have to
specify `24h` (e.g. the next smallest unit). This arrangement is very
error-prone for users.
This PR adds `calendar_interval` and `fixed_interval` parameters to any
code that uses intervals (date_histogram, rollup, composite, datafeed, etc).
Calendar only accepts calendar intervals, fixed accepts any combination of
units (meaning `1d` can be used to specify `24h` in fixed time), and both
are mutually exclusive.
The old interval behavior is deprecated and will throw a deprecation warning.
It is also mutually exclusive with the two new parameters. In the future the
old dual-purpose interval will be removed.
The change applies to both REST and java clients.
2019-05-20 12:07:29 -04:00
One week is the interval between the start day_of_week:hour:minute:second
and the same day of the week and time of the following week in the specified
2020-05-29 17:40:40 -04:00
time zone.
2018-10-31 10:16:36 -04:00
2020-05-29 17:40:40 -04:00
`month`, `1M` ::
2018-10-31 10:16:36 -04:00
[7.x Backport] Force selection of calendar or fixed intervals (#41906)
The date_histogram accepts an interval which can be either a calendar
interval (DST-aware, leap seconds, arbitrary length of months, etc) or
fixed interval (strict multiples of SI units). Unfortunately this is inferred
by first trying to parse as a calendar interval, then falling back to fixed
if that fails.
This leads to confusing arrangement where `1d` == calendar, but
`2d` == fixed. And if you want a day of fixed time, you have to
specify `24h` (e.g. the next smallest unit). This arrangement is very
error-prone for users.
This PR adds `calendar_interval` and `fixed_interval` parameters to any
code that uses intervals (date_histogram, rollup, composite, datafeed, etc).
Calendar only accepts calendar intervals, fixed accepts any combination of
units (meaning `1d` can be used to specify `24h` in fixed time), and both
are mutually exclusive.
The old interval behavior is deprecated and will throw a deprecation warning.
It is also mutually exclusive with the two new parameters. In the future the
old dual-purpose interval will be removed.
The change applies to both REST and java clients.
2019-05-20 12:07:29 -04:00
One month is the interval between the start day of the month and time of
2018-10-31 10:16:36 -04:00
day and the same day of the month and time of the following month in the specified
2020-05-29 17:40:40 -04:00
time zone, so that the day of the month and time of day are the same at the start
2018-10-31 10:16:36 -04:00
and end.
2020-05-29 17:40:40 -04:00
`quarter`, `1q` ::
2018-10-31 10:16:36 -04:00
2020-05-29 17:40:40 -04:00
One quarter is the interval between the start day of the month and
2018-10-31 10:16:36 -04:00
time of day and the same day of the month and time of day three months later,
so that the day of the month and time of day are the same at the start and end. +
2020-05-29 17:40:40 -04:00
`year`, `1y` ::
2018-10-31 10:16:36 -04:00
2020-05-29 17:40:40 -04:00
One year is the interval between the start day of the month and time of
[7.x Backport] Force selection of calendar or fixed intervals (#41906)
The date_histogram accepts an interval which can be either a calendar
interval (DST-aware, leap seconds, arbitrary length of months, etc) or
fixed interval (strict multiples of SI units). Unfortunately this is inferred
by first trying to parse as a calendar interval, then falling back to fixed
if that fails.
This leads to confusing arrangement where `1d` == calendar, but
`2d` == fixed. And if you want a day of fixed time, you have to
specify `24h` (e.g. the next smallest unit). This arrangement is very
error-prone for users.
This PR adds `calendar_interval` and `fixed_interval` parameters to any
code that uses intervals (date_histogram, rollup, composite, datafeed, etc).
Calendar only accepts calendar intervals, fixed accepts any combination of
units (meaning `1d` can be used to specify `24h` in fixed time), and both
are mutually exclusive.
The old interval behavior is deprecated and will throw a deprecation warning.
It is also mutually exclusive with the two new parameters. In the future the
old dual-purpose interval will be removed.
The change applies to both REST and java clients.
2019-05-20 12:07:29 -04:00
day and the same day of the month and time of day the following year in the
2020-05-29 17:40:40 -04:00
specified time zone, so that the date and time are the same at the start and end. +
2018-10-31 10:16:36 -04:00
2019-11-22 11:31:30 -05:00
[[calendar_interval_examples]]
===== Calendar interval examples
[7.x Backport] Force selection of calendar or fixed intervals (#41906)
The date_histogram accepts an interval which can be either a calendar
interval (DST-aware, leap seconds, arbitrary length of months, etc) or
fixed interval (strict multiples of SI units). Unfortunately this is inferred
by first trying to parse as a calendar interval, then falling back to fixed
if that fails.
This leads to confusing arrangement where `1d` == calendar, but
`2d` == fixed. And if you want a day of fixed time, you have to
specify `24h` (e.g. the next smallest unit). This arrangement is very
error-prone for users.
This PR adds `calendar_interval` and `fixed_interval` parameters to any
code that uses intervals (date_histogram, rollup, composite, datafeed, etc).
Calendar only accepts calendar intervals, fixed accepts any combination of
units (meaning `1d` can be used to specify `24h` in fixed time), and both
are mutually exclusive.
The old interval behavior is deprecated and will throw a deprecation warning.
It is also mutually exclusive with the two new parameters. In the future the
old dual-purpose interval will be removed.
The change applies to both REST and java clients.
2019-05-20 12:07:29 -04:00
As an example, here is an aggregation requesting bucket intervals of a month in calendar time:
2018-10-31 10:16:36 -04:00
2019-09-05 10:11:25 -04:00
[source,console]
[7.x Backport] Force selection of calendar or fixed intervals (#41906)
The date_histogram accepts an interval which can be either a calendar
interval (DST-aware, leap seconds, arbitrary length of months, etc) or
fixed interval (strict multiples of SI units). Unfortunately this is inferred
by first trying to parse as a calendar interval, then falling back to fixed
if that fails.
This leads to confusing arrangement where `1d` == calendar, but
`2d` == fixed. And if you want a day of fixed time, you have to
specify `24h` (e.g. the next smallest unit). This arrangement is very
error-prone for users.
This PR adds `calendar_interval` and `fixed_interval` parameters to any
code that uses intervals (date_histogram, rollup, composite, datafeed, etc).
Calendar only accepts calendar intervals, fixed accepts any combination of
units (meaning `1d` can be used to specify `24h` in fixed time), and both
are mutually exclusive.
The old interval behavior is deprecated and will throw a deprecation warning.
It is also mutually exclusive with the two new parameters. In the future the
old dual-purpose interval will be removed.
The change applies to both REST and java clients.
2019-05-20 12:07:29 -04:00
--------------------------------------------------
POST /sales/_search?size=0
{
2020-07-20 15:59:00 -04:00
"aggs": {
"sales_over_time": {
"date_histogram": {
"field": "date",
"calendar_interval": "month"
}
[7.x Backport] Force selection of calendar or fixed intervals (#41906)
The date_histogram accepts an interval which can be either a calendar
interval (DST-aware, leap seconds, arbitrary length of months, etc) or
fixed interval (strict multiples of SI units). Unfortunately this is inferred
by first trying to parse as a calendar interval, then falling back to fixed
if that fails.
This leads to confusing arrangement where `1d` == calendar, but
`2d` == fixed. And if you want a day of fixed time, you have to
specify `24h` (e.g. the next smallest unit). This arrangement is very
error-prone for users.
This PR adds `calendar_interval` and `fixed_interval` parameters to any
code that uses intervals (date_histogram, rollup, composite, datafeed, etc).
Calendar only accepts calendar intervals, fixed accepts any combination of
units (meaning `1d` can be used to specify `24h` in fixed time), and both
are mutually exclusive.
The old interval behavior is deprecated and will throw a deprecation warning.
It is also mutually exclusive with the two new parameters. In the future the
old dual-purpose interval will be removed.
The change applies to both REST and java clients.
2019-05-20 12:07:29 -04:00
}
2020-07-20 15:59:00 -04:00
}
[7.x Backport] Force selection of calendar or fixed intervals (#41906)
The date_histogram accepts an interval which can be either a calendar
interval (DST-aware, leap seconds, arbitrary length of months, etc) or
fixed interval (strict multiples of SI units). Unfortunately this is inferred
by first trying to parse as a calendar interval, then falling back to fixed
if that fails.
This leads to confusing arrangement where `1d` == calendar, but
`2d` == fixed. And if you want a day of fixed time, you have to
specify `24h` (e.g. the next smallest unit). This arrangement is very
error-prone for users.
This PR adds `calendar_interval` and `fixed_interval` parameters to any
code that uses intervals (date_histogram, rollup, composite, datafeed, etc).
Calendar only accepts calendar intervals, fixed accepts any combination of
units (meaning `1d` can be used to specify `24h` in fixed time), and both
are mutually exclusive.
The old interval behavior is deprecated and will throw a deprecation warning.
It is also mutually exclusive with the two new parameters. In the future the
old dual-purpose interval will be removed.
The change applies to both REST and java clients.
2019-05-20 12:07:29 -04:00
}
--------------------------------------------------
// TEST[setup:sales]
2018-10-31 10:16:36 -04:00
[7.x Backport] Force selection of calendar or fixed intervals (#41906)
The date_histogram accepts an interval which can be either a calendar
interval (DST-aware, leap seconds, arbitrary length of months, etc) or
fixed interval (strict multiples of SI units). Unfortunately this is inferred
by first trying to parse as a calendar interval, then falling back to fixed
if that fails.
This leads to confusing arrangement where `1d` == calendar, but
`2d` == fixed. And if you want a day of fixed time, you have to
specify `24h` (e.g. the next smallest unit). This arrangement is very
error-prone for users.
This PR adds `calendar_interval` and `fixed_interval` parameters to any
code that uses intervals (date_histogram, rollup, composite, datafeed, etc).
Calendar only accepts calendar intervals, fixed accepts any combination of
units (meaning `1d` can be used to specify `24h` in fixed time), and both
are mutually exclusive.
The old interval behavior is deprecated and will throw a deprecation warning.
It is also mutually exclusive with the two new parameters. In the future the
old dual-purpose interval will be removed.
The change applies to both REST and java clients.
2019-05-20 12:07:29 -04:00
If you attempt to use multiples of calendar units, the aggregation will fail because only
singular calendar units are supported:
2018-10-31 10:16:36 -04:00
2019-09-05 10:11:25 -04:00
[source,console]
[7.x Backport] Force selection of calendar or fixed intervals (#41906)
The date_histogram accepts an interval which can be either a calendar
interval (DST-aware, leap seconds, arbitrary length of months, etc) or
fixed interval (strict multiples of SI units). Unfortunately this is inferred
by first trying to parse as a calendar interval, then falling back to fixed
if that fails.
This leads to confusing arrangement where `1d` == calendar, but
`2d` == fixed. And if you want a day of fixed time, you have to
specify `24h` (e.g. the next smallest unit). This arrangement is very
error-prone for users.
This PR adds `calendar_interval` and `fixed_interval` parameters to any
code that uses intervals (date_histogram, rollup, composite, datafeed, etc).
Calendar only accepts calendar intervals, fixed accepts any combination of
units (meaning `1d` can be used to specify `24h` in fixed time), and both
are mutually exclusive.
The old interval behavior is deprecated and will throw a deprecation warning.
It is also mutually exclusive with the two new parameters. In the future the
old dual-purpose interval will be removed.
The change applies to both REST and java clients.
2019-05-20 12:07:29 -04:00
--------------------------------------------------
POST /sales/_search?size=0
{
2020-07-20 15:59:00 -04:00
"aggs": {
"sales_over_time": {
"date_histogram": {
"field": "date",
"calendar_interval": "2d"
}
[7.x Backport] Force selection of calendar or fixed intervals (#41906)
The date_histogram accepts an interval which can be either a calendar
interval (DST-aware, leap seconds, arbitrary length of months, etc) or
fixed interval (strict multiples of SI units). Unfortunately this is inferred
by first trying to parse as a calendar interval, then falling back to fixed
if that fails.
This leads to confusing arrangement where `1d` == calendar, but
`2d` == fixed. And if you want a day of fixed time, you have to
specify `24h` (e.g. the next smallest unit). This arrangement is very
error-prone for users.
This PR adds `calendar_interval` and `fixed_interval` parameters to any
code that uses intervals (date_histogram, rollup, composite, datafeed, etc).
Calendar only accepts calendar intervals, fixed accepts any combination of
units (meaning `1d` can be used to specify `24h` in fixed time), and both
are mutually exclusive.
The old interval behavior is deprecated and will throw a deprecation warning.
It is also mutually exclusive with the two new parameters. In the future the
old dual-purpose interval will be removed.
The change applies to both REST and java clients.
2019-05-20 12:07:29 -04:00
}
2020-07-20 15:59:00 -04:00
}
[7.x Backport] Force selection of calendar or fixed intervals (#41906)
The date_histogram accepts an interval which can be either a calendar
interval (DST-aware, leap seconds, arbitrary length of months, etc) or
fixed interval (strict multiples of SI units). Unfortunately this is inferred
by first trying to parse as a calendar interval, then falling back to fixed
if that fails.
This leads to confusing arrangement where `1d` == calendar, but
`2d` == fixed. And if you want a day of fixed time, you have to
specify `24h` (e.g. the next smallest unit). This arrangement is very
error-prone for users.
This PR adds `calendar_interval` and `fixed_interval` parameters to any
code that uses intervals (date_histogram, rollup, composite, datafeed, etc).
Calendar only accepts calendar intervals, fixed accepts any combination of
units (meaning `1d` can be used to specify `24h` in fixed time), and both
are mutually exclusive.
The old interval behavior is deprecated and will throw a deprecation warning.
It is also mutually exclusive with the two new parameters. In the future the
old dual-purpose interval will be removed.
The change applies to both REST and java clients.
2019-05-20 12:07:29 -04:00
}
--------------------------------------------------
// TEST[setup:sales]
// TEST[catch:bad_request]
2018-10-31 10:16:36 -04:00
[7.x Backport] Force selection of calendar or fixed intervals (#41906)
The date_histogram accepts an interval which can be either a calendar
interval (DST-aware, leap seconds, arbitrary length of months, etc) or
fixed interval (strict multiples of SI units). Unfortunately this is inferred
by first trying to parse as a calendar interval, then falling back to fixed
if that fails.
This leads to confusing arrangement where `1d` == calendar, but
`2d` == fixed. And if you want a day of fixed time, you have to
specify `24h` (e.g. the next smallest unit). This arrangement is very
error-prone for users.
This PR adds `calendar_interval` and `fixed_interval` parameters to any
code that uses intervals (date_histogram, rollup, composite, datafeed, etc).
Calendar only accepts calendar intervals, fixed accepts any combination of
units (meaning `1d` can be used to specify `24h` in fixed time), and both
are mutually exclusive.
The old interval behavior is deprecated and will throw a deprecation warning.
It is also mutually exclusive with the two new parameters. In the future the
old dual-purpose interval will be removed.
The change applies to both REST and java clients.
2019-05-20 12:07:29 -04:00
[source,js]
--------------------------------------------------
{
"error" : {
"root_cause" : [...],
"type" : "x_content_parse_exception",
"reason" : "[1:82] [date_histogram] failed to parse field [calendar_interval]",
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "The supplied interval [2d] could not be parsed as a calendar interval.",
"stack_trace" : "java.lang.IllegalArgumentException: The supplied interval [2d] could not be parsed as a calendar interval."
}
}
}
--------------------------------------------------
// NOTCONSOLE
2019-11-22 11:31:30 -05:00
[[fixed_intervals]]
===== Fixed intervals
[7.x Backport] Force selection of calendar or fixed intervals (#41906)
The date_histogram accepts an interval which can be either a calendar
interval (DST-aware, leap seconds, arbitrary length of months, etc) or
fixed interval (strict multiples of SI units). Unfortunately this is inferred
by first trying to parse as a calendar interval, then falling back to fixed
if that fails.
This leads to confusing arrangement where `1d` == calendar, but
`2d` == fixed. And if you want a day of fixed time, you have to
specify `24h` (e.g. the next smallest unit). This arrangement is very
error-prone for users.
This PR adds `calendar_interval` and `fixed_interval` parameters to any
code that uses intervals (date_histogram, rollup, composite, datafeed, etc).
Calendar only accepts calendar intervals, fixed accepts any combination of
units (meaning `1d` can be used to specify `24h` in fixed time), and both
are mutually exclusive.
The old interval behavior is deprecated and will throw a deprecation warning.
It is also mutually exclusive with the two new parameters. In the future the
old dual-purpose interval will be removed.
The change applies to both REST and java clients.
2019-05-20 12:07:29 -04:00
Fixed intervals are configured with the `fixed_interval` parameter.
In contrast to calendar-aware intervals, fixed intervals are a fixed number of SI
units and never deviate, regardless of where they fall on the calendar. One second
2020-05-29 17:40:40 -04:00
is always composed of `1000ms`. This allows fixed intervals to be specified in
[7.x Backport] Force selection of calendar or fixed intervals (#41906)
The date_histogram accepts an interval which can be either a calendar
interval (DST-aware, leap seconds, arbitrary length of months, etc) or
fixed interval (strict multiples of SI units). Unfortunately this is inferred
by first trying to parse as a calendar interval, then falling back to fixed
if that fails.
This leads to confusing arrangement where `1d` == calendar, but
`2d` == fixed. And if you want a day of fixed time, you have to
specify `24h` (e.g. the next smallest unit). This arrangement is very
error-prone for users.
This PR adds `calendar_interval` and `fixed_interval` parameters to any
code that uses intervals (date_histogram, rollup, composite, datafeed, etc).
Calendar only accepts calendar intervals, fixed accepts any combination of
units (meaning `1d` can be used to specify `24h` in fixed time), and both
are mutually exclusive.
The old interval behavior is deprecated and will throw a deprecation warning.
It is also mutually exclusive with the two new parameters. In the future the
old dual-purpose interval will be removed.
The change applies to both REST and java clients.
2019-05-20 12:07:29 -04:00
any multiple of the supported units.
However, it means fixed intervals cannot express other units such as months,
since the duration of a month is not a fixed quantity. Attempting to specify
a calendar interval like month or quarter will throw an exception.
The accepted units for fixed intervals are:
2020-05-29 17:40:40 -04:00
milliseconds (`ms`) ::
A single millisecond. This is a very, very small interval.
[7.x Backport] Force selection of calendar or fixed intervals (#41906)
The date_histogram accepts an interval which can be either a calendar
interval (DST-aware, leap seconds, arbitrary length of months, etc) or
fixed interval (strict multiples of SI units). Unfortunately this is inferred
by first trying to parse as a calendar interval, then falling back to fixed
if that fails.
This leads to confusing arrangement where `1d` == calendar, but
`2d` == fixed. And if you want a day of fixed time, you have to
specify `24h` (e.g. the next smallest unit). This arrangement is very
error-prone for users.
This PR adds `calendar_interval` and `fixed_interval` parameters to any
code that uses intervals (date_histogram, rollup, composite, datafeed, etc).
Calendar only accepts calendar intervals, fixed accepts any combination of
units (meaning `1d` can be used to specify `24h` in fixed time), and both
are mutually exclusive.
The old interval behavior is deprecated and will throw a deprecation warning.
It is also mutually exclusive with the two new parameters. In the future the
old dual-purpose interval will be removed.
The change applies to both REST and java clients.
2019-05-20 12:07:29 -04:00
2020-05-29 17:40:40 -04:00
seconds (`s`) ::
Defined as 1000 milliseconds each.
[7.x Backport] Force selection of calendar or fixed intervals (#41906)
The date_histogram accepts an interval which can be either a calendar
interval (DST-aware, leap seconds, arbitrary length of months, etc) or
fixed interval (strict multiples of SI units). Unfortunately this is inferred
by first trying to parse as a calendar interval, then falling back to fixed
if that fails.
This leads to confusing arrangement where `1d` == calendar, but
`2d` == fixed. And if you want a day of fixed time, you have to
specify `24h` (e.g. the next smallest unit). This arrangement is very
error-prone for users.
This PR adds `calendar_interval` and `fixed_interval` parameters to any
code that uses intervals (date_histogram, rollup, composite, datafeed, etc).
Calendar only accepts calendar intervals, fixed accepts any combination of
units (meaning `1d` can be used to specify `24h` in fixed time), and both
are mutually exclusive.
The old interval behavior is deprecated and will throw a deprecation warning.
It is also mutually exclusive with the two new parameters. In the future the
old dual-purpose interval will be removed.
The change applies to both REST and java clients.
2019-05-20 12:07:29 -04:00
2020-05-29 17:40:40 -04:00
minutes (`m`) ::
Defined as 60 seconds each (60,000 milliseconds).
[7.x Backport] Force selection of calendar or fixed intervals (#41906)
The date_histogram accepts an interval which can be either a calendar
interval (DST-aware, leap seconds, arbitrary length of months, etc) or
fixed interval (strict multiples of SI units). Unfortunately this is inferred
by first trying to parse as a calendar interval, then falling back to fixed
if that fails.
This leads to confusing arrangement where `1d` == calendar, but
`2d` == fixed. And if you want a day of fixed time, you have to
specify `24h` (e.g. the next smallest unit). This arrangement is very
error-prone for users.
This PR adds `calendar_interval` and `fixed_interval` parameters to any
code that uses intervals (date_histogram, rollup, composite, datafeed, etc).
Calendar only accepts calendar intervals, fixed accepts any combination of
units (meaning `1d` can be used to specify `24h` in fixed time), and both
are mutually exclusive.
The old interval behavior is deprecated and will throw a deprecation warning.
It is also mutually exclusive with the two new parameters. In the future the
old dual-purpose interval will be removed.
The change applies to both REST and java clients.
2019-05-20 12:07:29 -04:00
All minutes begin at 00 seconds.
2020-05-29 17:40:40 -04:00
hours (`h`) ::
Defined as 60 minutes each (3,600,000 milliseconds).
[7.x Backport] Force selection of calendar or fixed intervals (#41906)
The date_histogram accepts an interval which can be either a calendar
interval (DST-aware, leap seconds, arbitrary length of months, etc) or
fixed interval (strict multiples of SI units). Unfortunately this is inferred
by first trying to parse as a calendar interval, then falling back to fixed
if that fails.
This leads to confusing arrangement where `1d` == calendar, but
`2d` == fixed. And if you want a day of fixed time, you have to
specify `24h` (e.g. the next smallest unit). This arrangement is very
error-prone for users.
This PR adds `calendar_interval` and `fixed_interval` parameters to any
code that uses intervals (date_histogram, rollup, composite, datafeed, etc).
Calendar only accepts calendar intervals, fixed accepts any combination of
units (meaning `1d` can be used to specify `24h` in fixed time), and both
are mutually exclusive.
The old interval behavior is deprecated and will throw a deprecation warning.
It is also mutually exclusive with the two new parameters. In the future the
old dual-purpose interval will be removed.
The change applies to both REST and java clients.
2019-05-20 12:07:29 -04:00
All hours begin at 00 minutes and 00 seconds.
2020-05-29 17:40:40 -04:00
days (`d`) ::
Defined as 24 hours (86,400,000 milliseconds).
[7.x Backport] Force selection of calendar or fixed intervals (#41906)
The date_histogram accepts an interval which can be either a calendar
interval (DST-aware, leap seconds, arbitrary length of months, etc) or
fixed interval (strict multiples of SI units). Unfortunately this is inferred
by first trying to parse as a calendar interval, then falling back to fixed
if that fails.
This leads to confusing arrangement where `1d` == calendar, but
`2d` == fixed. And if you want a day of fixed time, you have to
specify `24h` (e.g. the next smallest unit). This arrangement is very
error-prone for users.
This PR adds `calendar_interval` and `fixed_interval` parameters to any
code that uses intervals (date_histogram, rollup, composite, datafeed, etc).
Calendar only accepts calendar intervals, fixed accepts any combination of
units (meaning `1d` can be used to specify `24h` in fixed time), and both
are mutually exclusive.
The old interval behavior is deprecated and will throw a deprecation warning.
It is also mutually exclusive with the two new parameters. In the future the
old dual-purpose interval will be removed.
The change applies to both REST and java clients.
2019-05-20 12:07:29 -04:00
All days begin at the earliest possible time, which is usually 00:00:00
(midnight).
2019-11-22 11:31:30 -05:00
[[fixed_interval_examples]]
===== Fixed interval examples
[7.x Backport] Force selection of calendar or fixed intervals (#41906)
The date_histogram accepts an interval which can be either a calendar
interval (DST-aware, leap seconds, arbitrary length of months, etc) or
fixed interval (strict multiples of SI units). Unfortunately this is inferred
by first trying to parse as a calendar interval, then falling back to fixed
if that fails.
This leads to confusing arrangement where `1d` == calendar, but
`2d` == fixed. And if you want a day of fixed time, you have to
specify `24h` (e.g. the next smallest unit). This arrangement is very
error-prone for users.
This PR adds `calendar_interval` and `fixed_interval` parameters to any
code that uses intervals (date_histogram, rollup, composite, datafeed, etc).
Calendar only accepts calendar intervals, fixed accepts any combination of
units (meaning `1d` can be used to specify `24h` in fixed time), and both
are mutually exclusive.
The old interval behavior is deprecated and will throw a deprecation warning.
It is also mutually exclusive with the two new parameters. In the future the
old dual-purpose interval will be removed.
The change applies to both REST and java clients.
2019-05-20 12:07:29 -04:00
If we try to recreate the "month" `calendar_interval` from earlier, we can approximate that with
30 fixed days:
2013-11-24 06:13:08 -05:00
2019-09-05 10:11:25 -04:00
[source,console]
2013-11-24 06:13:08 -05:00
--------------------------------------------------
2017-01-20 16:23:28 -05:00
POST /sales/_search?size=0
2013-11-24 06:13:08 -05:00
{
2020-07-20 15:59:00 -04:00
"aggs": {
"sales_over_time": {
"date_histogram": {
"field": "date",
"fixed_interval": "30d"
}
2013-11-24 06:13:08 -05:00
}
2020-07-20 15:59:00 -04:00
}
2013-11-24 06:13:08 -05:00
}
--------------------------------------------------
2017-01-20 16:23:28 -05:00
// TEST[setup:sales]
2013-11-24 06:13:08 -05:00
[7.x Backport] Force selection of calendar or fixed intervals (#41906)
The date_histogram accepts an interval which can be either a calendar
interval (DST-aware, leap seconds, arbitrary length of months, etc) or
fixed interval (strict multiples of SI units). Unfortunately this is inferred
by first trying to parse as a calendar interval, then falling back to fixed
if that fails.
This leads to confusing arrangement where `1d` == calendar, but
`2d` == fixed. And if you want a day of fixed time, you have to
specify `24h` (e.g. the next smallest unit). This arrangement is very
error-prone for users.
This PR adds `calendar_interval` and `fixed_interval` parameters to any
code that uses intervals (date_histogram, rollup, composite, datafeed, etc).
Calendar only accepts calendar intervals, fixed accepts any combination of
units (meaning `1d` can be used to specify `24h` in fixed time), and both
are mutually exclusive.
The old interval behavior is deprecated and will throw a deprecation warning.
It is also mutually exclusive with the two new parameters. In the future the
old dual-purpose interval will be removed.
The change applies to both REST and java clients.
2019-05-20 12:07:29 -04:00
But if we try to use a calendar unit that is not supported, such as weeks, we'll get an exception:
2013-11-24 06:13:08 -05:00
2019-09-05 10:11:25 -04:00
[source,console]
2013-11-24 06:13:08 -05:00
--------------------------------------------------
2017-01-20 16:23:28 -05:00
POST /sales/_search?size=0
2013-11-24 06:13:08 -05:00
{
2020-07-20 15:59:00 -04:00
"aggs": {
"sales_over_time": {
"date_histogram": {
"field": "date",
"fixed_interval": "2w"
}
2013-11-24 06:13:08 -05:00
}
2020-07-20 15:59:00 -04:00
}
2013-11-24 06:13:08 -05:00
}
--------------------------------------------------
2017-01-20 16:23:28 -05:00
// TEST[setup:sales]
[7.x Backport] Force selection of calendar or fixed intervals (#41906)
The date_histogram accepts an interval which can be either a calendar
interval (DST-aware, leap seconds, arbitrary length of months, etc) or
fixed interval (strict multiples of SI units). Unfortunately this is inferred
by first trying to parse as a calendar interval, then falling back to fixed
if that fails.
This leads to confusing arrangement where `1d` == calendar, but
`2d` == fixed. And if you want a day of fixed time, you have to
specify `24h` (e.g. the next smallest unit). This arrangement is very
error-prone for users.
This PR adds `calendar_interval` and `fixed_interval` parameters to any
code that uses intervals (date_histogram, rollup, composite, datafeed, etc).
Calendar only accepts calendar intervals, fixed accepts any combination of
units (meaning `1d` can be used to specify `24h` in fixed time), and both
are mutually exclusive.
The old interval behavior is deprecated and will throw a deprecation warning.
It is also mutually exclusive with the two new parameters. In the future the
old dual-purpose interval will be removed.
The change applies to both REST and java clients.
2019-05-20 12:07:29 -04:00
// TEST[catch:bad_request]
[source,js]
--------------------------------------------------
{
"error" : {
"root_cause" : [...],
"type" : "x_content_parse_exception",
"reason" : "[1:82] [date_histogram] failed to parse field [fixed_interval]",
"caused_by" : {
"type" : "illegal_argument_exception",
"reason" : "failed to parse setting [date_histogram.fixedInterval] with value [2w] as a time value: unit is missing or unrecognized",
"stack_trace" : "java.lang.IllegalArgumentException: failed to parse setting [date_histogram.fixedInterval] with value [2w] as a time value: unit is missing or unrecognized"
}
}
}
--------------------------------------------------
// NOTCONSOLE
2020-08-12 14:01:10 -04:00
[[datehistogram-aggregation-notes]]
[7.x Backport] Force selection of calendar or fixed intervals (#41906)
The date_histogram accepts an interval which can be either a calendar
interval (DST-aware, leap seconds, arbitrary length of months, etc) or
fixed interval (strict multiples of SI units). Unfortunately this is inferred
by first trying to parse as a calendar interval, then falling back to fixed
if that fails.
This leads to confusing arrangement where `1d` == calendar, but
`2d` == fixed. And if you want a day of fixed time, you have to
specify `24h` (e.g. the next smallest unit). This arrangement is very
error-prone for users.
This PR adds `calendar_interval` and `fixed_interval` parameters to any
code that uses intervals (date_histogram, rollup, composite, datafeed, etc).
Calendar only accepts calendar intervals, fixed accepts any combination of
units (meaning `1d` can be used to specify `24h` in fixed time), and both
are mutually exclusive.
The old interval behavior is deprecated and will throw a deprecation warning.
It is also mutually exclusive with the two new parameters. In the future the
old dual-purpose interval will be removed.
The change applies to both REST and java clients.
2019-05-20 12:07:29 -04:00
===== Notes
In all cases, when the specified end time does not exist, the actual end time is
the closest available time after the specified end.
Widely distributed applications must also consider vagaries such as countries that
2020-07-20 10:51:23 -04:00
start and stop daylight savings time at 12:01 A.M., so end up with one minute of
[7.x Backport] Force selection of calendar or fixed intervals (#41906)
The date_histogram accepts an interval which can be either a calendar
interval (DST-aware, leap seconds, arbitrary length of months, etc) or
fixed interval (strict multiples of SI units). Unfortunately this is inferred
by first trying to parse as a calendar interval, then falling back to fixed
if that fails.
This leads to confusing arrangement where `1d` == calendar, but
`2d` == fixed. And if you want a day of fixed time, you have to
specify `24h` (e.g. the next smallest unit). This arrangement is very
error-prone for users.
This PR adds `calendar_interval` and `fixed_interval` parameters to any
code that uses intervals (date_histogram, rollup, composite, datafeed, etc).
Calendar only accepts calendar intervals, fixed accepts any combination of
units (meaning `1d` can be used to specify `24h` in fixed time), and both
are mutually exclusive.
The old interval behavior is deprecated and will throw a deprecation warning.
It is also mutually exclusive with the two new parameters. In the future the
old dual-purpose interval will be removed.
The change applies to both REST and java clients.
2019-05-20 12:07:29 -04:00
Sunday followed by an additional 59 minutes of Saturday once a year, and countries
that decide to move across the international date line. Situations like
2020-07-20 10:51:23 -04:00
that can make irregular time zone offsets seem easy.
[7.x Backport] Force selection of calendar or fixed intervals (#41906)
The date_histogram accepts an interval which can be either a calendar
interval (DST-aware, leap seconds, arbitrary length of months, etc) or
fixed interval (strict multiples of SI units). Unfortunately this is inferred
by first trying to parse as a calendar interval, then falling back to fixed
if that fails.
This leads to confusing arrangement where `1d` == calendar, but
`2d` == fixed. And if you want a day of fixed time, you have to
specify `24h` (e.g. the next smallest unit). This arrangement is very
error-prone for users.
This PR adds `calendar_interval` and `fixed_interval` parameters to any
code that uses intervals (date_histogram, rollup, composite, datafeed, etc).
Calendar only accepts calendar intervals, fixed accepts any combination of
units (meaning `1d` can be used to specify `24h` in fixed time), and both
are mutually exclusive.
The old interval behavior is deprecated and will throw a deprecation warning.
It is also mutually exclusive with the two new parameters. In the future the
old dual-purpose interval will be removed.
The change applies to both REST and java clients.
2019-05-20 12:07:29 -04:00
As always, rigorous testing, especially around time-change events, will ensure
that your time interval specification is
what you intend it to be.
WARNING:
To avoid unexpected results, all connected servers and clients must sync to a
reliable network time service.
NOTE: fractional time values are not supported, but you can address this by
shifting to another time unit (e.g., `1.5h` could instead be specified as `90m`).
NOTE: You can also specify time values using abbreviations supported by
<<time-units,time units>> parsing.
2013-11-24 06:13:08 -05:00
2018-10-31 10:16:36 -04:00
===== Keys
2013-11-24 06:13:08 -05:00
2015-09-07 13:53:50 -04:00
Internally, a date is represented as a 64 bit number representing a timestamp
2018-10-31 10:16:36 -04:00
in milliseconds-since-the-epoch (01/01/1970 midnight UTC). These timestamps are
returned as the ++key++ name of the bucket. The `key_as_string` is the same
timestamp converted to a formatted
2018-11-23 09:13:44 -05:00
date string using the `format` parameter specification:
2015-09-07 13:53:50 -04:00
2018-10-31 10:16:36 -04:00
TIP: If you don't specify `format`, the first date
<<mapping-date-format,format>> specified in the field mapping is used.
2013-11-24 06:13:08 -05:00
2019-09-05 10:11:25 -04:00
[source,console]
2013-11-24 06:13:08 -05:00
--------------------------------------------------
2017-01-20 16:23:28 -05:00
POST /sales/_search?size=0
2013-11-24 06:13:08 -05:00
{
2020-07-20 15:59:00 -04:00
"aggs": {
"sales_over_time": {
"date_histogram": {
"field": "date",
"calendar_interval": "1M",
"format": "yyyy-MM-dd" <1>
}
2013-11-24 06:13:08 -05:00
}
2020-07-20 15:59:00 -04:00
}
2013-11-24 06:13:08 -05:00
}
--------------------------------------------------
2017-01-20 16:23:28 -05:00
// TEST[setup:sales]
2013-11-24 06:13:08 -05:00
<1> Supports expressive date <<date-format-pattern,format pattern>>
Response:
2019-09-06 16:09:09 -04:00
[source,console-result]
2013-11-24 06:13:08 -05:00
--------------------------------------------------
{
2020-07-20 15:59:00 -04:00
...
"aggregations": {
"sales_over_time": {
"buckets": [
{
"key_as_string": "2015-01-01",
"key": 1420070400000,
"doc_count": 3
},
{
"key_as_string": "2015-02-01",
"key": 1422748800000,
"doc_count": 2
},
{
"key_as_string": "2015-03-01",
"key": 1425168000000,
"doc_count": 2
2014-01-28 11:46:26 -05:00
}
2020-07-20 15:59:00 -04:00
]
2013-11-24 06:13:08 -05:00
}
2020-07-20 15:59:00 -04:00
}
2013-11-24 06:13:08 -05:00
}
--------------------------------------------------
2017-01-20 16:23:28 -05:00
// TESTRESPONSE[s/\.\.\./"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/]
2013-11-24 06:13:08 -05:00
2020-05-29 17:40:40 -04:00
===== Time zone
2015-09-07 13:53:50 -04:00
2020-05-29 17:40:40 -04:00
{es} stores date-times in Coordinated Universal Time (UTC). By default, all bucketing and
2018-10-31 10:16:36 -04:00
rounding is also done in UTC. Use the `time_zone` parameter to indicate
2020-05-29 17:40:40 -04:00
that bucketing should use a different time zone.
For example, if the interval is a calendar day and the time zone is
`America/New_York` then `2020-01-03T01:00:01Z` is :
# Converted to `2020-01-02T18:00:01`
# Rounded down to `2020-01-02T00:00:00`
# Then converted back to UTC to produce `2020-01-02T05:00:00:00Z`
# Finally, when the bucket is turned into a string key it is printed in
`America/New_York` so it'll display as `"2020-01-02T00:00:00"`.
It looks like:
[source,java]
----
bucket_key = localToUtc(Math.floor(utcToLocal(value) / interval) * interval))
----
2015-09-07 13:53:50 -04:00
2020-05-29 17:40:40 -04:00
You can specify time zones as an ISO 8601 UTC offset (e.g. `+01:00` or
`-08:00`) or as an IANA time zone ID,
such as `America/Los_Angeles`.
2015-09-07 13:53:50 -04:00
Consider the following example:
2019-09-05 10:11:25 -04:00
[source,console]
2015-09-07 13:53:50 -04:00
---------------------------------
2020-07-27 15:58:26 -04:00
PUT my-index-000001/_doc/1?refresh
2015-09-07 13:53:50 -04:00
{
"date": "2015-10-01T00:30:00Z"
}
2020-07-27 15:58:26 -04:00
PUT my-index-000001/_doc/2?refresh
2015-09-07 13:53:50 -04:00
{
"date": "2015-10-01T01:30:00Z"
}
2020-07-27 15:58:26 -04:00
GET my-index-000001/_search?size=0
2015-09-07 13:53:50 -04:00
{
"aggs": {
"by_day": {
"date_histogram": {
"field": "date",
[7.x Backport] Force selection of calendar or fixed intervals (#41906)
The date_histogram accepts an interval which can be either a calendar
interval (DST-aware, leap seconds, arbitrary length of months, etc) or
fixed interval (strict multiples of SI units). Unfortunately this is inferred
by first trying to parse as a calendar interval, then falling back to fixed
if that fails.
This leads to confusing arrangement where `1d` == calendar, but
`2d` == fixed. And if you want a day of fixed time, you have to
specify `24h` (e.g. the next smallest unit). This arrangement is very
error-prone for users.
This PR adds `calendar_interval` and `fixed_interval` parameters to any
code that uses intervals (date_histogram, rollup, composite, datafeed, etc).
Calendar only accepts calendar intervals, fixed accepts any combination of
units (meaning `1d` can be used to specify `24h` in fixed time), and both
are mutually exclusive.
The old interval behavior is deprecated and will throw a deprecation warning.
It is also mutually exclusive with the two new parameters. In the future the
old dual-purpose interval will be removed.
The change applies to both REST and java clients.
2019-05-20 12:07:29 -04:00
"calendar_interval": "day"
2015-09-07 13:53:50 -04:00
}
}
}
}
---------------------------------
2020-05-29 17:40:40 -04:00
If you don't specify a time zone, UTC is used. This would result in both of these
2015-09-07 13:53:50 -04:00
documents being placed into the same day bucket, which starts at midnight UTC
on 1 October 2015:
2019-09-06 16:09:09 -04:00
[source,console-result]
2015-09-07 13:53:50 -04:00
---------------------------------
2017-01-20 16:23:28 -05:00
{
...
"aggregations": {
"by_day": {
"buckets": [
{
"key_as_string": "2015-10-01T00:00:00.000Z",
"key": 1443657600000,
"doc_count": 2
}
]
}
2015-09-07 13:53:50 -04:00
}
}
---------------------------------
2017-01-20 16:23:28 -05:00
// TESTRESPONSE[s/\.\.\./"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/]
2015-09-07 13:53:50 -04:00
2020-05-29 17:40:40 -04:00
If you specify a `time_zone` of `-01:00`, midnight in that time zone is one hour
2018-10-31 10:16:36 -04:00
before midnight UTC:
2015-09-07 13:53:50 -04:00
2019-09-05 10:11:25 -04:00
[source,console]
2015-09-07 13:53:50 -04:00
---------------------------------
2020-07-27 15:58:26 -04:00
GET my-index-000001/_search?size=0
2015-09-07 13:53:50 -04:00
{
"aggs": {
"by_day": {
"date_histogram": {
"field": "date",
[7.x Backport] Force selection of calendar or fixed intervals (#41906)
The date_histogram accepts an interval which can be either a calendar
interval (DST-aware, leap seconds, arbitrary length of months, etc) or
fixed interval (strict multiples of SI units). Unfortunately this is inferred
by first trying to parse as a calendar interval, then falling back to fixed
if that fails.
This leads to confusing arrangement where `1d` == calendar, but
`2d` == fixed. And if you want a day of fixed time, you have to
specify `24h` (e.g. the next smallest unit). This arrangement is very
error-prone for users.
This PR adds `calendar_interval` and `fixed_interval` parameters to any
code that uses intervals (date_histogram, rollup, composite, datafeed, etc).
Calendar only accepts calendar intervals, fixed accepts any combination of
units (meaning `1d` can be used to specify `24h` in fixed time), and both
are mutually exclusive.
The old interval behavior is deprecated and will throw a deprecation warning.
It is also mutually exclusive with the two new parameters. In the future the
old dual-purpose interval will be removed.
The change applies to both REST and java clients.
2019-05-20 12:07:29 -04:00
"calendar_interval": "day",
2015-09-07 13:53:50 -04:00
"time_zone": "-01:00"
}
}
}
}
---------------------------------
2017-01-20 16:23:28 -05:00
// TEST[continued]
2015-09-07 13:53:50 -04:00
Now the first document falls into the bucket for 30 September 2015, while the
second document falls into the bucket for 1 October 2015:
2019-09-06 16:09:09 -04:00
[source,console-result]
2015-09-07 13:53:50 -04:00
---------------------------------
2017-01-20 16:23:28 -05:00
{
...
"aggregations": {
"by_day": {
"buckets": [
{
"key_as_string": "2015-09-30T00:00:00.000-01:00", <1>
"key": 1443574800000,
"doc_count": 1
},
{
"key_as_string": "2015-10-01T00:00:00.000-01:00", <1>
"key": 1443661200000,
"doc_count": 1
}
]
}
2015-09-07 13:53:50 -04:00
}
}
---------------------------------
2017-01-20 16:23:28 -05:00
// TESTRESPONSE[s/\.\.\./"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/]
2015-09-07 13:53:50 -04:00
<1> The `key_as_string` value represents midnight on each day
2020-05-29 17:40:40 -04:00
in the specified time zone.
2015-09-07 13:53:50 -04:00
2020-05-29 17:40:40 -04:00
WARNING: Many time zones shift their clocks for daylight savings time. Buckets
close to the moment when those changes happen can have slightly different sizes
than you would expect from the `calendar_interval` or `fixed_interval`.
2016-11-15 13:06:35 -05:00
For example, consider a DST start in the `CET` time zone: on 27 March 2016 at 2am,
2020-05-29 17:40:40 -04:00
clocks were turned forward 1 hour to 3am local time. If you use `day` as the
`calendar_interval`, the bucket covering that day will only hold data for 23
hours instead of the usual 24 hours for other buckets. The same is true for
shorter intervals, like a `fixed_interval` of `12h`, where you'll have only a 11h
bucket on the morning of 27 March when the DST shift happens.
2016-11-15 13:06:35 -05:00
2020-01-09 14:11:24 -05:00
[[search-aggregations-bucket-datehistogram-offset]]
2018-10-31 10:16:36 -04:00
===== Offset
2016-11-15 13:06:35 -05:00
2020-01-09 14:11:24 -05:00
// tag::offset-explanation[]
2018-10-31 10:16:36 -04:00
Use the `offset` parameter to change the start value of each bucket by the
2015-09-07 13:53:50 -04:00
specified positive (`+`) or negative offset (`-`) duration, such as `1h` for
2017-01-30 02:49:49 -05:00
an hour, or `1d` for a day. See <<time-units>> for more possible time
2015-09-07 13:53:50 -04:00
duration options.
2018-10-31 10:16:36 -04:00
For example, when using an interval of `day`, each bucket runs from midnight
2020-01-09 14:11:24 -05:00
to midnight. Setting the `offset` parameter to `+6h` changes each bucket
2015-09-07 13:53:50 -04:00
to run from 6am to 6am:
2020-01-09 14:11:24 -05:00
// end::offset-explanation[]
2015-09-07 13:53:50 -04:00
2019-09-05 10:11:25 -04:00
[source,console]
2015-09-07 13:53:50 -04:00
-----------------------------
2020-07-27 15:58:26 -04:00
PUT my-index-000001/_doc/1?refresh
2015-09-07 13:53:50 -04:00
{
"date": "2015-10-01T05:30:00Z"
}
2020-07-27 15:58:26 -04:00
PUT my-index-000001/_doc/2?refresh
2015-09-07 13:53:50 -04:00
{
"date": "2015-10-01T06:30:00Z"
}
2020-07-27 15:58:26 -04:00
GET my-index-000001/_search?size=0
2015-09-07 13:53:50 -04:00
{
"aggs": {
"by_day": {
"date_histogram": {
"field": "date",
[7.x Backport] Force selection of calendar or fixed intervals (#41906)
The date_histogram accepts an interval which can be either a calendar
interval (DST-aware, leap seconds, arbitrary length of months, etc) or
fixed interval (strict multiples of SI units). Unfortunately this is inferred
by first trying to parse as a calendar interval, then falling back to fixed
if that fails.
This leads to confusing arrangement where `1d` == calendar, but
`2d` == fixed. And if you want a day of fixed time, you have to
specify `24h` (e.g. the next smallest unit). This arrangement is very
error-prone for users.
This PR adds `calendar_interval` and `fixed_interval` parameters to any
code that uses intervals (date_histogram, rollup, composite, datafeed, etc).
Calendar only accepts calendar intervals, fixed accepts any combination of
units (meaning `1d` can be used to specify `24h` in fixed time), and both
are mutually exclusive.
The old interval behavior is deprecated and will throw a deprecation warning.
It is also mutually exclusive with the two new parameters. In the future the
old dual-purpose interval will be removed.
The change applies to both REST and java clients.
2019-05-20 12:07:29 -04:00
"calendar_interval": "day",
2015-09-07 13:53:50 -04:00
"offset": "+6h"
}
}
}
}
-----------------------------
2020-01-09 14:11:24 -05:00
// tag::offset-result-intro[]
2015-09-07 13:53:50 -04:00
Instead of a single bucket starting at midnight, the above request groups the
documents into buckets starting at 6am:
2020-01-09 14:11:24 -05:00
// end::offset-result-intro[]
2015-09-07 13:53:50 -04:00
2019-09-06 16:09:09 -04:00
[source,console-result]
2015-09-07 13:53:50 -04:00
-----------------------------
2017-01-20 16:23:28 -05:00
{
...
"aggregations": {
"by_day": {
"buckets": [
{
"key_as_string": "2015-09-30T06:00:00.000Z",
"key": 1443592800000,
"doc_count": 1
},
{
"key_as_string": "2015-10-01T06:00:00.000Z",
"key": 1443679200000,
"doc_count": 1
}
]
}
2015-09-07 13:53:50 -04:00
}
}
-----------------------------
2017-01-20 16:23:28 -05:00
// TESTRESPONSE[s/\.\.\./"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/]
2015-09-07 13:53:50 -04:00
2020-01-09 14:11:24 -05:00
// tag::offset-note[]
2018-10-31 10:16:36 -04:00
NOTE: The start `offset` of each bucket is calculated after `time_zone`
2015-09-07 13:53:50 -04:00
adjustments have been made.
2020-01-09 14:11:24 -05:00
// end::offset-note[]
2015-09-07 13:53:50 -04:00
2018-10-31 10:16:36 -04:00
===== Keyed Response
2017-04-18 09:57:50 -04:00
2018-10-31 10:16:36 -04:00
Setting the `keyed` flag to `true` associates a unique string key with each
bucket and returns the ranges as a hash rather than an array:
2017-04-18 09:57:50 -04:00
2019-09-05 10:11:25 -04:00
[source,console]
2017-04-18 09:57:50 -04:00
--------------------------------------------------
POST /sales/_search?size=0
{
2020-07-20 15:59:00 -04:00
"aggs": {
"sales_over_time": {
"date_histogram": {
"field": "date",
"calendar_interval": "1M",
"format": "yyyy-MM-dd",
"keyed": true
}
2017-04-18 09:57:50 -04:00
}
2020-07-20 15:59:00 -04:00
}
2017-04-18 09:57:50 -04:00
}
--------------------------------------------------
// TEST[setup:sales]
Response:
2019-09-06 16:09:09 -04:00
[source,console-result]
2017-04-18 09:57:50 -04:00
--------------------------------------------------
{
2020-07-20 15:59:00 -04:00
...
"aggregations": {
"sales_over_time": {
"buckets": {
"2015-01-01": {
"key_as_string": "2015-01-01",
"key": 1420070400000,
"doc_count": 3
},
"2015-02-01": {
"key_as_string": "2015-02-01",
"key": 1422748800000,
"doc_count": 2
},
"2015-03-01": {
"key_as_string": "2015-03-01",
"key": 1425168000000,
"doc_count": 2
2017-04-18 09:57:50 -04:00
}
2020-07-20 15:59:00 -04:00
}
2017-04-18 09:57:50 -04:00
}
2020-07-20 15:59:00 -04:00
}
2017-04-18 09:57:50 -04:00
}
--------------------------------------------------
// TESTRESPONSE[s/\.\.\./"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/]
2018-10-31 10:16:36 -04:00
===== Scripts
2015-09-07 13:53:50 -04:00
2018-10-31 10:16:36 -04:00
As with the normal <<search-aggregations-bucket-histogram-aggregation,histogram>>,
both document-level scripts and
value-level scripts are supported. You can control the order of the returned
buckets using the `order`
settings and filter the returned buckets based on a `min_doc_count` setting
(by default all buckets between the first
bucket that matches documents and the last one are returned). This histogram
also supports the `extended_bounds`
setting, which enables extending the bounds of the histogram beyond the data
2020-07-20 10:51:23 -04:00
itself, and `hard_bounds` that limits the histogram to specified bounds.
For more information, see
<<search-aggregations-bucket-histogram-aggregation-extended-bounds,`Extended Bounds`>> and
<<search-aggregations-bucket-histogram-aggregation-hard-bounds,`Hard Bounds`>>.
2015-05-07 10:46:40 -04:00
2018-10-31 10:16:36 -04:00
===== Missing value
2015-05-07 10:46:40 -04:00
2018-10-31 10:16:36 -04:00
The `missing` parameter defines how to treat documents that are missing a value.
By default, they are ignored, but it is also possible to treat them as if they
have a value.
2015-05-07 10:46:40 -04:00
2019-09-05 10:11:25 -04:00
[source,console]
2015-05-07 10:46:40 -04:00
--------------------------------------------------
2017-01-20 16:23:28 -05:00
POST /sales/_search?size=0
2015-05-07 10:46:40 -04:00
{
2020-07-20 15:59:00 -04:00
"aggs": {
"sale_date": {
"date_histogram": {
"field": "date",
"calendar_interval": "year",
"missing": "2000/01/01" <1>
}
2015-05-07 10:46:40 -04:00
}
2020-07-20 15:59:00 -04:00
}
2015-05-07 10:46:40 -04:00
}
--------------------------------------------------
2017-01-20 16:23:28 -05:00
// TEST[setup:sales]
2015-05-07 10:46:40 -04:00
2018-10-31 10:16:36 -04:00
<1> Documents without a value in the `publish_date` field will fall into the
same bucket as documents that have the value `2000-01-01`.
2017-05-11 13:06:26 -04:00
2018-10-31 10:16:36 -04:00
===== Order
2017-05-11 13:06:26 -04:00
2018-10-31 10:16:36 -04:00
By default the returned buckets are sorted by their `key` ascending, but you can
control the order using
the `order` setting. This setting supports the same `order` functionality as
<<search-aggregations-bucket-terms-aggregation-order,`Terms Aggregation`>>.
2017-05-11 13:06:26 -04:00
2018-10-31 10:16:36 -04:00
===== Using a script to aggregate by day of the week
2017-07-07 14:16:53 -04:00
2020-07-20 10:51:23 -04:00
When you need to aggregate the results by day of the week, use a script that
2018-10-31 10:16:36 -04:00
returns the day of the week:
2017-07-07 14:16:53 -04:00
2019-09-05 10:11:25 -04:00
[source,console]
2017-07-07 14:16:53 -04:00
--------------------------------------------------
POST /sales/_search?size=0
{
2020-07-20 15:59:00 -04:00
"aggs": {
"dayOfWeek": {
"terms": {
"script": {
"lang": "painless",
"source": "doc['date'].value.dayOfWeekEnum.value"
2017-07-07 14:16:53 -04:00
}
2020-07-20 15:59:00 -04:00
}
2017-07-07 14:16:53 -04:00
}
2020-07-20 15:59:00 -04:00
}
2017-07-07 14:16:53 -04:00
}
--------------------------------------------------
// TEST[setup:sales]
Response:
2019-09-06 16:09:09 -04:00
[source,console-result]
2017-07-07 14:16:53 -04:00
--------------------------------------------------
{
...
"aggregations": {
"dayOfWeek": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "7",
"doc_count": 4
},
{
"key": "4",
"doc_count": 3
}
]
}
}
}
--------------------------------------------------
// TESTRESPONSE[s/\.\.\./"took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,/]
2018-10-31 10:16:36 -04:00
The response will contain all the buckets having the relative day of
the week as key : 1 for Monday, 2 for Tuesday... 7 for Sunday.