mirror of
https://github.com/honeymoose/OpenSearch.git
synced 2025-02-07 05:28:34 +00:00
bf1550a0b2
This adds a new Rollup module to XPack, which allows users to configure periodic "rollup jobs" to pre-aggregate data. That data is then available later for search through a special RollupSearch API, which mimics the DSL and functionality of regular search. Rollups are used to drastically reduce the on-disk footprint of metric-based data (e.g. timestamped document with numeric and keyword fields). It can also be used to speed up aggregations over large datasets, since the rolled data will be considerably smaller and fewer documents to search. The PR adds seven new endpoints to interact with Rollups; create/get/delete job, start/stop job, a capabilities API similar to field-caps, and a Rollup-enabled search. Original commit: elastic/x-pack-elasticsearch@dcde91aacf
180 lines
5.2 KiB
Plaintext
180 lines
5.2 KiB
Plaintext
[role="xpack"]
|
|
[[rollup-get-rollup-caps]]
|
|
=== Get Rollup Job Capabilities
|
|
++++
|
|
<titleabbrev>Get Rollup Caps</titleabbrev>
|
|
++++
|
|
|
|
This API returns the rollup capabilities that have been configured for an index or index pattern. This API is useful
|
|
because a rollup job is often configured to rollup only a subset of fields from the source index. Furthermore, only
|
|
certain aggregations can be configured for various fields, leading to a limited subset of functionality depending on
|
|
that configuration.
|
|
|
|
This API will allow you to inspect an index and determine:
|
|
|
|
1. Does this index have associated rollup data somewhere in the cluster?
|
|
2. If yes to the first question, what fields were rolled up, what aggregations can be performed, and where does the data
|
|
live?
|
|
|
|
==== Request
|
|
|
|
`GET _xpack/rollup/data/{index}`
|
|
|
|
//===== Description
|
|
|
|
==== Path Parameters
|
|
|
|
`index`::
|
|
(string) Index, indices or index-pattern to return rollup capabilities for. If omitted (or `_all` is used) all available
|
|
rollup job capabilities will be returned
|
|
|
|
|
|
==== Request Body
|
|
|
|
There is no request body for the Get Jobs API.
|
|
|
|
==== Authorization
|
|
|
|
You must have `monitor`, `monitor_rollup`, `manage` or `manage_rollup` cluster privileges to use this API.
|
|
For more information, see
|
|
{xpack-ref}/security-privileges.html[Security Privileges].
|
|
|
|
==== Examples
|
|
|
|
Imagine we have an index named `sensor-1` full of raw data. We know that the data will grow over time, so there
|
|
will be a `sensor-2`, `sensor-3`, etc. Let's create a Rollup job that targets the index pattern `sensor-*` to accomodate
|
|
this future scaling:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
PUT _xpack/rollup/job/sensor
|
|
{
|
|
"index_pattern": "sensor-*",
|
|
"rollup_index": "sensor_rollup",
|
|
"cron": "*/30 * * * * ?",
|
|
"size" :1000,
|
|
"groups" : {
|
|
"date_histogram": {
|
|
"field": "timestamp",
|
|
"interval": "1h",
|
|
"delay": "7d"
|
|
},
|
|
"terms": {
|
|
"fields": ["node"]
|
|
}
|
|
},
|
|
"metrics": [
|
|
{
|
|
"field": "temperature",
|
|
"metrics": ["min", "max", "sum"]
|
|
},
|
|
{
|
|
"field": "voltage",
|
|
"metrics": ["avg"]
|
|
}
|
|
]
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
// TEST[setup:sensor_index]
|
|
|
|
We can then retrieve the rollup capabilities of that index pattern (`sensor-*`) via the following command:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
GET _xpack/rollup/data/sensor-*
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
// TEST[continued]
|
|
|
|
Which will yield the following response:
|
|
|
|
[source,js]
|
|
----
|
|
{
|
|
"sensor-*" : {
|
|
"rollup_jobs" : [
|
|
{
|
|
"job_id" : "sensor",
|
|
"rollup_index" : "sensor_rollup",
|
|
"index_pattern" : "sensor-*",
|
|
"fields" : {
|
|
"node" : [
|
|
{
|
|
"agg" : "terms"
|
|
}
|
|
],
|
|
"temperature" : [
|
|
{
|
|
"agg" : "min"
|
|
},
|
|
{
|
|
"agg" : "max"
|
|
},
|
|
{
|
|
"agg" : "sum"
|
|
}
|
|
],
|
|
"timestamp" : [
|
|
{
|
|
"agg" : "date_histogram",
|
|
"time_zone" : "UTC",
|
|
"interval" : "1h",
|
|
"delay": "7d"
|
|
}
|
|
],
|
|
"voltage" : [
|
|
{
|
|
"agg" : "avg"
|
|
}
|
|
]
|
|
}
|
|
}
|
|
]
|
|
}
|
|
}
|
|
----
|
|
// TESTRESPONSE
|
|
|
|
The response that is returned contains information that is similar to the original Rollup configuration, but formatted
|
|
differently. First, there are some house-keeping details: the Rollup job's ID, the index that holds the rolled data,
|
|
the index pattern that the job was targeting.
|
|
|
|
Next it shows a list of fields that contain data eligible for rollup searches. Here we see four fields: `node`, `temperature`,
|
|
`timestamp` and `voltage`. Each of these fields list the aggregations that are possible. For example, you can use a min, max
|
|
or sum aggregation on the `temperature` field, but only a `date_histogram` on `timestamp`.
|
|
|
|
Note that the `rollup_jobs` element is an array; there can be multiple, independent jobs configured for a single index
|
|
or index pattern. Each of these jobs may have different configurations, so the API returns a list of all the various
|
|
configurations available.
|
|
|
|
We could also retrieve the same information with a request to `_all`:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
GET _xpack/rollup/data/_all
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
// TEST[continued]
|
|
|
|
But note that if we use the concrete index name (`sensor-1`), we'll retrieve no rollup capabilities:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
GET _xpack/rollup/data/sensor-1
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
// TEST[continued]
|
|
|
|
[source,js]
|
|
----
|
|
{
|
|
|
|
}
|
|
----
|
|
// TESTRESPONSE
|
|
|
|
Why is this? The original rollup job was configured against a specific index pattern (`sensor-*`) not a concrete index
|
|
(`sensor-1`). So while the index belongs to the pattern, the rollup job is only valid across the entirety of the pattern
|
|
not just one of it's containing indices. So for that reason, the Rollup Capabilities API only returns information based
|
|
on the originally configured index name or pattern. |