248 lines
8.1 KiB
Plaintext
248 lines
8.1 KiB
Plaintext
[role="xpack"]
|
|
[testenv="basic"]
|
|
[[rollup-understanding-groups]]
|
|
=== Understanding groups
|
|
|
|
experimental[]
|
|
|
|
To preserve flexibility, Rollup Jobs are defined based on how future queries may need to use the data. Traditionally, systems force
|
|
the admin to make decisions about what metrics to rollup and on what interval. E.g. The average of `cpu_time` on an hourly basis. This
|
|
is limiting; if, at a future date, the admin wishes to see the average of `cpu_time` on an hourly basis _and partitioned by `host_name`_,
|
|
they are out of luck.
|
|
|
|
Of course, the admin can decide to rollup the `[hour, host]` tuple on an hourly basis, but as the number of grouping keys grows, so do the
|
|
number of tuples the admin needs to configure. Furthermore, these `[hours, host]` tuples are only useful for hourly rollups... daily, weekly,
|
|
or monthly rollups all require new configurations.
|
|
|
|
Rather than force the admin to decide ahead of time which individual tuples should be rolled up, Elasticsearch's Rollup jobs are configured
|
|
based on which groups are potentially useful to future queries. For example, this configuration:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
"groups" : {
|
|
"date_histogram": {
|
|
"field": "timestamp",
|
|
"fixed_interval": "1h",
|
|
"delay": "7d"
|
|
},
|
|
"terms": {
|
|
"fields": ["hostname", "datacenter"]
|
|
},
|
|
"histogram": {
|
|
"fields": ["load", "net_in", "net_out"],
|
|
"interval": 5
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// NOTCONSOLE
|
|
|
|
Allows `date_histogram`'s to be used on the `"timestamp"` field, `terms` aggregations to be used on the `"hostname"` and `"datacenter"`
|
|
fields, and `histograms` to be used on any of `"load"`, `"net_in"`, `"net_out"` fields.
|
|
|
|
Importantly, these aggs/fields can be used in any combination. This aggregation:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
"aggs" : {
|
|
"hourly": {
|
|
"date_histogram": {
|
|
"field": "timestamp",
|
|
"fixed_interval": "1h"
|
|
},
|
|
"aggs": {
|
|
"host_names": {
|
|
"terms": {
|
|
"field": "hostname"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// NOTCONSOLE
|
|
|
|
is just as valid as this aggregation:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
"aggs" : {
|
|
"hourly": {
|
|
"date_histogram": {
|
|
"field": "timestamp",
|
|
"fixed_interval": "1h"
|
|
},
|
|
"aggs": {
|
|
"data_center": {
|
|
"terms": {
|
|
"field": "datacenter"
|
|
}
|
|
},
|
|
"aggs": {
|
|
"host_names": {
|
|
"terms": {
|
|
"field": "hostname"
|
|
}
|
|
},
|
|
"aggs": {
|
|
"load_values": {
|
|
"histogram": {
|
|
"field": "load",
|
|
"interval": 5
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// NOTCONSOLE
|
|
|
|
|
|
You'll notice that the second aggregation is not only substantially larger, it also swapped the position of the terms aggregation on
|
|
`"hostname"`, illustrating how the order of aggregations does not matter to rollups. Similarly, while the `date_histogram` is required
|
|
for rolling up data, it isn't required while querying (although often used). For example, this is a valid aggregation for
|
|
Rollup Search to execute:
|
|
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
"aggs" : {
|
|
"host_names": {
|
|
"terms": {
|
|
"field": "hostname"
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// NOTCONSOLE
|
|
|
|
Ultimately, when configuring `groups` for a job, think in terms of how you might wish to partition data in a query at a future date...
|
|
then include those in the config. Because Rollup Search allows any order or combination of the grouped fields, you just need to decide
|
|
if a field is useful for aggregating later, and how you might wish to use it (terms, histogram, etc).
|
|
|
|
[[rollup-understanding-group-intervals]]
|
|
==== Calendar vs fixed time intervals
|
|
|
|
Each rollup-job must have a date histogram group with a defined interval. {es}
|
|
understands both
|
|
<<calendar_and_fixed_intervals,calendar and fixed time intervals>>. Fixed time
|
|
intervals are fairly easy to understand; `60s` means sixty seconds. But what
|
|
does `1M` mean? One month of time depends on which month we are talking about,
|
|
some months are longer or shorter than others. This is an example of calendar
|
|
time and the duration of that unit depends on context. Calendar units are also
|
|
affected by leap-seconds, leap-years, etc.
|
|
|
|
This is important because the buckets generated by rollup are in either calendar
|
|
or fixed intervals and this limits how you can query them later. See
|
|
<<rollup-search-limitations-intervals>>.
|
|
|
|
We recommend sticking with fixed time intervals, since they are easier to
|
|
understand and are more flexible at query time. It will introduce some drift in
|
|
your data during leap-events and you will have to think about months in a fixed
|
|
quantity (30 days) instead of the actual calendar length. However, it is often
|
|
easier than dealing with calendar units at query time.
|
|
|
|
Multiples of units are always "fixed". For example, `2h` is always the fixed
|
|
quantity `7200` seconds. Single units can be fixed or calendar depending on the
|
|
unit:
|
|
|
|
[options="header"]
|
|
|=======
|
|
|Unit |Calendar |Fixed
|
|
|millisecond |NA |`1ms`, `10ms`, etc
|
|
|second |NA |`1s`, `10s`, etc
|
|
|minute |`1m` |`2m`, `10m`, etc
|
|
|hour |`1h` |`2h`, `10h`, etc
|
|
|day |`1d` |`2d`, `10d`, etc
|
|
|week |`1w` |NA
|
|
|month |`1M` |NA
|
|
|quarter |`1q` |NA
|
|
|year |`1y` |NA
|
|
|=======
|
|
|
|
For some units where there are both fixed and calendar, you may need to express
|
|
the quantity in terms of the next smaller unit. For example, if you want a fixed
|
|
day (not a calendar day), you should specify `24h` instead of `1d`. Similarly,
|
|
if you want fixed hours, specify `60m` instead of `1h`. This is because the
|
|
single quantity entails calendar time, and limits you to querying by calendar
|
|
time in the future.
|
|
|
|
==== Grouping limitations with heterogeneous indices
|
|
|
|
There was previously a limitation in how Rollup could handle indices that had heterogeneous mappings (multiple, unrelated/non-overlapping
|
|
mappings). The recommendation at the time was to configure a separate job per data "type". For example, you might configure a separate
|
|
job for each Beats module that you had enabled (one for `process`, another for `filesystem`, etc).
|
|
|
|
This recommendation was driven by internal implementation details that caused document counts to be potentially incorrect if a single "merged"
|
|
job was used.
|
|
|
|
This limitation has since been alleviated. As of 6.4.0, it is now considered best practice to combine all rollup configurations
|
|
into a single job.
|
|
|
|
As an example, if your index has two types of documents:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
{
|
|
"timestamp": 1516729294000,
|
|
"temperature": 200,
|
|
"voltage": 5.2,
|
|
"node": "a"
|
|
}
|
|
--------------------------------------------------
|
|
// NOTCONSOLE
|
|
|
|
and
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
{
|
|
"timestamp": 1516729294000,
|
|
"price": 123,
|
|
"title": "Foo"
|
|
}
|
|
--------------------------------------------------
|
|
// NOTCONSOLE
|
|
|
|
the best practice is to combine them into a single rollup job which covers both of these document types, like this:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
PUT _rollup/job/combined
|
|
{
|
|
"index_pattern": "data-*",
|
|
"rollup_index": "data_rollup",
|
|
"cron": "*/30 * * * * ?",
|
|
"page_size": 1000,
|
|
"groups": {
|
|
"date_histogram": {
|
|
"field": "timestamp",
|
|
"fixed_interval": "1h",
|
|
"delay": "7d"
|
|
},
|
|
"terms": {
|
|
"fields": [ "node", "title" ]
|
|
}
|
|
},
|
|
"metrics": [
|
|
{
|
|
"field": "temperature",
|
|
"metrics": [ "min", "max", "sum" ]
|
|
},
|
|
{
|
|
"field": "price",
|
|
"metrics": [ "avg" ]
|
|
}
|
|
]
|
|
}
|
|
--------------------------------------------------
|
|
// NOTCONSOLE
|
|
|
|
==== Doc counts and overlapping jobs
|
|
|
|
There was previously an issue with document counts on "overlapping" job configurations, driven by the same internal implementation detail.
|
|
If there were two Rollup jobs saving to the same index, where one job is a "subset" of another job, it was possible that document counts
|
|
could be incorrect for certain aggregation arrangements.
|
|
|
|
This issue has also since been eliminated in 6.4.0. |