234 lines
11 KiB
Plaintext
234 lines
11 KiB
Plaintext
[role="xpack"]
|
|
[[rollup-job-config]]
|
|
=== Rollup Job Configuration
|
|
|
|
experimental[]
|
|
|
|
The Rollup Job Configuration contains all the details about how the rollup job should run, when it indexes documents,
|
|
and what future queries will be able to execute against the rollup index.
|
|
|
|
There are three main sections to the Job Configuration; the logistical details about the job (cron schedule, etc), what fields
|
|
should be grouped on, and what metrics to collect for each group.
|
|
|
|
A full job configuration might look like this:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
PUT _xpack/rollup/job/sensor
|
|
{
|
|
"index_pattern": "sensor-*",
|
|
"rollup_index": "sensor_rollup",
|
|
"cron": "*/30 * * * * ?",
|
|
"page_size" :1000,
|
|
"groups" : {
|
|
"date_histogram": {
|
|
"field": "timestamp",
|
|
"interval": "1h",
|
|
"delay": "7d"
|
|
},
|
|
"terms": {
|
|
"fields": ["hostname", "datacenter"]
|
|
},
|
|
"histogram": {
|
|
"fields": ["load", "net_in", "net_out"],
|
|
"interval": 5
|
|
}
|
|
},
|
|
"metrics": [
|
|
{
|
|
"field": "temperature",
|
|
"metrics": ["min", "max", "sum"]
|
|
},
|
|
{
|
|
"field": "voltage",
|
|
"metrics": ["avg"]
|
|
}
|
|
]
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
// TEST[setup:sensor_index]
|
|
|
|
==== Logistical Details
|
|
|
|
In the above example, there are several pieces of logistical configuration for the job itself.
|
|
|
|
`{job_id}` (required)::
|
|
(string) In the endpoint URL, you specify the name of the job (`sensor` in the above example). This can be any alphanumeric string,
|
|
and uniquely identifies the data that is associated with the rollup job. The ID is persistent, in that it is stored with the rolled
|
|
up data. So if you create a job, let it run for a while, then delete the job... the data that the job rolled up will still be
|
|
associated with this job ID. You will be unable to create a new job with the same ID, as that could lead to problems with mismatched
|
|
job configurations
|
|
|
|
`index_pattern` (required)::
|
|
(string) The index, or index pattern, that you wish to rollup. Supports wildcard-style patterns (`logstash-*`). The job will
|
|
attempt to rollup the entire index or index-pattern. Once the "backfill" is finished, it will periodically (as defined by the cron)
|
|
look for new data and roll that up too.
|
|
|
|
`rollup_index` (required)::
|
|
(string) The index that you wish to store rollup results into. All the rollup data that is generated by the job will be
|
|
stored in this index. When searching the rollup data, this index will be used in the <<rollup-search,Rollup Search>> endpoint's URL.
|
|
The rollup index be shared with other rollup jobs. The data is stored so that it doesn't interfere with unrelated jobs.
|
|
|
|
`cron` (required)::
|
|
(string) A cron string which defines when the rollup job should be executed. The cron string defines an interval of when to run
|
|
the job's indexer. When the interval triggers, the indexer will attempt to rollup the data in the index pattern. The cron pattern
|
|
is unrelated to the time interval of the data being rolled up. For example, you may wish to create hourly rollups of your document (as
|
|
defined in the <<rollup-groups-config,grouping configuration>>) but to only run the indexer on a daily basis at midnight, as defined by the cron.
|
|
The cron pattern is defined just like Watcher's Cron Schedule.
|
|
|
|
`page_size` (required)::
|
|
(int) The number of bucket results that should be processed on each iteration of the rollup indexer. A larger value
|
|
will tend to execute faster, but will require more memory during processing. This has no effect on how the data is rolled up, it is
|
|
merely used for tweaking the speed/memory cost of the indexer.
|
|
|
|
[[rollup-groups-config]]
|
|
==== Grouping Config
|
|
|
|
The `groups` section of the configuration is where you decide which fields should be grouped on, and with what aggregations. These
|
|
fields will then be available later for aggregating into buckets. For example, this configuration:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
"groups" : {
|
|
"date_histogram": {
|
|
"field": "timestamp",
|
|
"interval": "1h",
|
|
"delay": "7d"
|
|
},
|
|
"terms": {
|
|
"fields": ["hostname", "datacenter"]
|
|
},
|
|
"histogram": {
|
|
"fields": ["load", "net_in", "net_out"],
|
|
"interval": 5
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// NOTCONSOLE
|
|
|
|
Allows `date_histogram`'s to be used on the `"timestamp"` field, `terms` aggregations to be used on the `"hostname"` and `"datacenter"`
|
|
fields, and `histograms` to be used on any of `"load"`, `"net_in"`, `"net_out"` fields.
|
|
|
|
Importantly, these aggs/fields can be used in any combination. Think of the `groups` configuration as defining a set of tools that can
|
|
later be used in aggregations to partition the data. Unlike raw data, we have to think ahead to which fields and aggregations might be used.
|
|
But Rollups provide enough flexibility that you simply need to determine _which_ fields are needed, not _in what order_ they are needed.
|
|
|
|
There are three types of groupings currently available:
|
|
|
|
===== Date Histogram
|
|
|
|
A `date_histogram` group aggregates a `date` field into time-based buckets. The `date_histogram` group is *mandatory* -- you currently
|
|
cannot rollup documents without a timestamp and a `date_histogram` group.
|
|
|
|
The `date_histogram` group has several parameters:
|
|
|
|
`field` (required)::
|
|
The date field that is to be rolled up.
|
|
|
|
`interval` (required)::
|
|
The interval of time buckets to be generated when rolling up. E.g. `"1h"` will produce hourly rollups. This follows standard time formatting
|
|
syntax as used elsewhere in Elasticsearch. The `interval` defines the _minimum_ interval that can be aggregated only. If hourly (`"1h"`)
|
|
intervals are configured, <<rollup-search,Rollup Search>> can execute aggregations with 1hr or greater (weekly, monthly, etc) intervals.
|
|
So define the interval as the smallest unit that you wish to later query.
|
|
|
|
Note: smaller, more granular intervals take up proportionally more space.
|
|
|
|
`delay`::
|
|
How long to wait before rolling up new documents. By default, the indexer attempts to roll up all data that is available. However, it
|
|
is not uncommon for data to arrive out of order, sometimes even a few days late. The indexer is unable to deal with data that arrives
|
|
after a time-span has been rolled up (e.g. there is no provision to update already-existing rollups).
|
|
|
|
Instead, you should specify a `delay` that matches the longest period of time you expect out-of-order data to arrive. E.g. a `delay` of
|
|
`"1d"` will instruct the indexer to roll up documents up to `"now - 1d"`, which provides a day of buffer time for out-of-order documents
|
|
to arrive.
|
|
|
|
`time_zone`::
|
|
Defines what time_zone the rollup documents are stored as. Unlike raw data, which can shift timezones on the fly, rolled documents have
|
|
to be stored with a specific timezone. By default, rollup documents are stored in `UTC`, but this can be changed with the `time_zone`
|
|
parameter.
|
|
|
|
===== Terms
|
|
|
|
The `terms` group can be used on `keyword` or numeric fields, to allow bucketing via the `terms` aggregation at a later point. The `terms`
|
|
group is optional. If defined, the indexer will enumerate and store _all_ values of a field for each time-period. This can be potentially
|
|
costly for high-cardinality groups such as IP addresses, especially if the time-bucket is particularly sparse.
|
|
|
|
While it is unlikely that a rollup will ever be larger in size than the raw data, defining `terms` groups on multiple high-cardinality fields
|
|
can effectively reduce the compression of a rollup to a large extent. You should be judicious which high-cardinality fields are included
|
|
for that reason.
|
|
|
|
The `terms` group has a single parameter:
|
|
|
|
`fields` (required)::
|
|
The set of fields that you wish to collect terms for. This array can contain fields that are both `keyword` and numerics. Order
|
|
does not matter
|
|
|
|
|
|
===== Histogram
|
|
|
|
The `histogram` group aggregates one or more numeric fields into numeric histogram intervals. This group is optional
|
|
|
|
|
|
The `histogram` group has a two parameters:
|
|
|
|
`fields` (required)::
|
|
The set of fields that you wish to build histograms for. All fields specified must be some kind of numeric. Order does not matter
|
|
|
|
`interval` (required)::
|
|
The interval of histogram buckets to be generated when rolling up. E.g. `5` will create buckets that are five units wide
|
|
(`0-5`, `5-10`, etc). Note that only one interval can be specified in the `histogram` group, meaning that all fields being grouped via
|
|
the histogram must share the same interval.
|
|
|
|
[[rollup-metrics-config]]
|
|
==== Metrics Config
|
|
|
|
After defining which groups should be generated for the data, you next configure which metrics should be collected. By default, only
|
|
the doc_counts are collected for each group. To make rollup useful, you will often add metrics like averages, mins, maxes, etc.
|
|
|
|
Metrics are defined on a per-field basis, and for each field you configure which metric should be collected. For example:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
"metrics": [
|
|
{
|
|
"field": "temperature",
|
|
"metrics": ["min", "max", "sum"]
|
|
},
|
|
{
|
|
"field": "voltage",
|
|
"metrics": ["avg"]
|
|
}
|
|
]
|
|
--------------------------------------------------
|
|
// NOTCONSOLE
|
|
|
|
This configuration defines metrics over two fields, `"temperature` and `"voltage"`. For the `"temperature"` field, we are collecting
|
|
the min, max and sum of the temperature. For `"voltage"`, we are collecting the average. These metrics are collected in a way that makes
|
|
them compatible with any combination of defined groups.
|
|
|
|
The `metrics` configuration accepts an array of objects, where each object has two parameters:
|
|
|
|
`field` (required)::
|
|
The field to collect metrics for. This must be a numeric of some kind
|
|
|
|
`metrics` (required)::
|
|
An array of metrics to collect for the field. At least one metric must be configured. Acceptable metrics are min/max/sum/avg/value_count.
|
|
|
|
|
|
|
|
.Averages aren't composable?!
|
|
**********************************
|
|
If you've worked with rollups before, you may be cautious around averages. If an average is saved for a 10 minute
|
|
interval, it usually isn't useful for larger intervals. You cannot average six 10-minute averages to find a
|
|
hourly average (average of averages is not equal to the total average).
|
|
|
|
For this reason, other systems tend to either omit the ability to average, or store the average at multiple intervals
|
|
to support more flexible querying.
|
|
|
|
Instead, the Rollup feature saves the `count` and `sum` for the defined time interval. This allows us to reconstruct
|
|
the average at any interval greater-than or equal to the defined interval. This gives maximum flexibility for
|
|
minimal storage costs... and you don't have to worry about average accuracies (no average of averages here!)
|
|
**********************************
|
|
|