[DOCS] Reformat rollup API docs (#49397)

This commit is contained in:
Lisa Cawley 2019-11-20 10:43:53 -08:00 committed by lcawl
parent d9fd4cc351
commit 0f15736687
7 changed files with 176 additions and 133 deletions

View File

@ -18,9 +18,9 @@ experimental[]
[[rollup-get-job-prereqs]] [[rollup-get-job-prereqs]]
==== {api-prereq-title} ==== {api-prereq-title}
* You must have `monitor`, `monitor_rollup`, `manage` or `manage_rollup` cluster * If the {es} {security-features} are enabled, you must have `monitor`,
privileges to use this API. For more information, see `monitor_rollup`, `manage` or `manage_rollup` cluster privileges to use this API.
<<security-privileges>>. For more information, see <<security-privileges>>.
[[rollup-get-job-desc]] [[rollup-get-job-desc]]
==== {api-description-title} ==== {api-description-title}

View File

@ -32,7 +32,7 @@ Jobs are created in a `STOPPED` state. You can start them with the
[[rollup-put-job-api-path-params]] [[rollup-put-job-api-path-params]]
==== {api-path-parms-title} ==== {api-path-parms-title}
`job_id`:: `<job_id>`::
(Required, string) Identifier for the {rollup-job}. (Required, string) Identifier for the {rollup-job}.
[[rollup-put-job-api-request-body]] [[rollup-put-job-api-request-body]]

View File

@ -1,17 +1,32 @@
[role="xpack"] [role="xpack"]
[testenv="basic"] [testenv="basic"]
[[rollup-get-rollup-caps]] [[rollup-get-rollup-caps]]
=== Get rollup job capabilities API === Get {rollup-job} capabilities API
++++ ++++
<titleabbrev>Get rollup caps</titleabbrev> <titleabbrev>Get rollup caps</titleabbrev>
++++ ++++
Returns the capabilities of any {rollup-jobs} that have been configured for a
specific index or index pattern.
experimental[] experimental[]
This API returns the capabilities of any rollup jobs that have been configured [[rollup-get-rollup-caps-request]]
for a specific index or index pattern. ==== {api-request-title}
This API is useful because a rollup job is often configured to rollup only a `GET _rollup/data/<index>`
[[rollup-get-rollup-caps-prereqs]]
==== {api-prereq-title}
* If the {es} {security-features} are enabled, you must have `monitor`,
`monitor_rollup`, `manage` or `manage_rollup` cluster privileges to use this API.
For more information, see <<security-privileges>>.
[[rollup-get-rollup-caps-desc]]
==== {api-description-title}
This API is useful because a {rollup-job} is often configured to rollup only a
subset of fields from the source index. Furthermore, only certain aggregations subset of fields from the source index. Furthermore, only certain aggregations
can be configured for various fields, leading to a limited subset of can be configured for various fields, leading to a limited subset of
functionality depending on that configuration. functionality depending on that configuration.
@ -22,33 +37,20 @@ This API enables you to inspect an index and determine:
2. If yes to the first question, what fields were rolled up, what aggregations 2. If yes to the first question, what fields were rolled up, what aggregations
can be performed, and where does the data live? can be performed, and where does the data live?
==== Request [[rollup-get-rollup-path-params]]
==== {api-path-parms-title}
`GET _rollup/data/{index}` `<index>`::
(string) Index, indices or index-pattern to return rollup capabilities for.
//===== Description `_all` may be used to fetch rollup capabilities from all jobs.
==== Path Parameters
`index`::
(string) Index, indices or index-pattern to return rollup capabilities for. `_all` may be used to fetch
rollup capabilities from all jobs
==== Request Body [[rollup-get-rollup-example]]
==== {api-examples-title}
There is no request body for the Get Rollup Caps API. Imagine we have an index named `sensor-1` full of raw data. We know that the
data will grow over time, so there will be a `sensor-2`, `sensor-3`, etc. Let's
==== Authorization create a {rollup-job} that targets the index pattern `sensor-*` to accommodate
You must have `monitor`, `monitor_rollup`, `manage` or `manage_rollup` cluster privileges to use this API.
For more information, see
<<security-privileges>>.
==== Examples
Imagine we have an index named `sensor-1` full of raw data. We know that the data will grow over time, so there
will be a `sensor-2`, `sensor-3`, etc. Let's create a Rollup job that targets the index pattern `sensor-*` to accommodate
this future scaling: this future scaling:
[source,console] [source,console]
@ -83,7 +85,8 @@ PUT _rollup/job/sensor
-------------------------------------------------- --------------------------------------------------
// TEST[setup:sensor_index] // TEST[setup:sensor_index]
We can then retrieve the rollup capabilities of that index pattern (`sensor-*`) via the following command: We can then retrieve the rollup capabilities of that index pattern (`sensor-*`)
via the following command:
[source,console] [source,console]
-------------------------------------------------- --------------------------------------------------
@ -139,17 +142,21 @@ Which will yield the following response:
} }
---- ----
The response that is returned contains information that is similar to the original Rollup configuration, but formatted The response that is returned contains information that is similar to the
differently. First, there are some house-keeping details: the Rollup job's ID, the index that holds the rolled data, original rollup configuration, but formatted differently. First, there are some
the index pattern that the job was targeting. house-keeping details: the {rollup-job} ID, the index that holds the rolled data,
and the index pattern that the job was targeting.
Next it shows a list of fields that contain data eligible for rollup searches. Here we see four fields: `node`, `temperature`, Next it shows a list of fields that contain data eligible for rollup searches.
`timestamp` and `voltage`. Each of these fields list the aggregations that are possible. For example, you can use a min, max Here we see four fields: `node`, `temperature`, `timestamp` and `voltage`. Each
or sum aggregation on the `temperature` field, but only a `date_histogram` on `timestamp`. of these fields list the aggregations that are possible. For example, you can
use a min, max or sum aggregation on the `temperature` field, but only a
`date_histogram` on `timestamp`.
Note that the `rollup_jobs` element is an array; there can be multiple, independent jobs configured for a single index Note that the `rollup_jobs` element is an array; there can be multiple,
or index pattern. Each of these jobs may have different configurations, so the API returns a list of all the various independent jobs configured for a single index or index pattern. Each of these
configurations available. jobs may have different configurations, so the API returns a list of all the
various configurations available.
We could also retrieve the same information with a request to `_all`: We could also retrieve the same information with a request to `_all`:
@ -159,7 +166,8 @@ GET _rollup/data/_all
-------------------------------------------------- --------------------------------------------------
// TEST[continued] // TEST[continued]
But note that if we use the concrete index name (`sensor-1`), we'll retrieve no rollup capabilities: But note that if we use the concrete index name (`sensor-1`), we'll retrieve no
rollup capabilities:
[source,console] [source,console]
-------------------------------------------------- --------------------------------------------------
@ -174,7 +182,9 @@ GET _rollup/data/sensor-1
} }
---- ----
Why is this? The original rollup job was configured against a specific index pattern (`sensor-*`) not a concrete index Why is this? The original {rollup-job} was configured against a specific index
(`sensor-1`). So while the index belongs to the pattern, the rollup job is only valid across the entirety of the pattern pattern (`sensor-*`) not a concrete index (`sensor-1`). So while the index
not just one of it's containing indices. So for that reason, the Rollup Capabilities API only returns information based belongs to the pattern, the {rollup-job} is only valid across the entirety of
on the originally configured index name or pattern. the pattern not just one of it's containing indices. So for that reason, the
get rollup capabilities API only returns information based on the originally
configured index name or pattern.

View File

@ -6,41 +6,48 @@
<titleabbrev>Get rollup index caps</titleabbrev> <titleabbrev>Get rollup index caps</titleabbrev>
++++ ++++
Returns the rollup capabilities of all jobs inside of a rollup index (e.g. the
index where rollup data is stored).
experimental[] experimental[]
This API returns the rollup capabilities of all jobs inside of a rollup index (e.g. the index where rollup data is stored). [[rollup-get-rollup-index-caps-request]]
A single rollup index may store the data for multiple rollup jobs, and may have a variety of capabilities depending on those jobs. ==== {api-request-title}
`GET <index>/_rollup/data`
[[rollup-get-rollup-index-caps-prereqs]]
==== {api-prereq-title}
* If the {es} {security-features} are enabled, you must have the `read` index
privilege on the index that stores the rollup results. For more information, see
<<security-privileges>>.
[[rollup-get-rollup-index-caps-desc]]
==== {api-description-title}
A single rollup index may store the data for multiple {rollup-jobs}, and may
have a variety of capabilities depending on those jobs.
This API will allow you to determine: This API will allow you to determine:
1. What jobs are stored in an index (or indices specified via a pattern)? 1. What jobs are stored in an index (or indices specified via a pattern)?
2. What target indices were rolled up, what fields were used in those rollups and what aggregations can be performed on each job? 2. What target indices were rolled up, what fields were used in those rollups
and what aggregations can be performed on each job?
==== Request [[rollup-get-rollup-index-caps-path-params]]
==== {api-path-parms-title}
`GET {index}/_rollup/data` `<index>`::
(Required, string) Index or index-pattern of concrete rollup indices to check
for capabilities.
//===== Description [[rollup-get-rollup-index-caps-example]]
==== {api-examples-title}
==== Path Parameters Imagine we have an index named `sensor-1` full of raw data. We know that the
data will grow over time, so there will be a `sensor-2`, `sensor-3`, etc.
`index`:: Let's create a {rollup-job} that stores its data in `sensor_rollup`:
(string) Index or index-pattern of concrete rollup indices to check for capabilities.
==== Request Body
There is no request body for the Get Jobs API.
==== Authorization
You must have the `read` index privilege on the index that stores the rollup results.
For more information, see
<<security-privileges>>.
==== Examples
Imagine we have an index named `sensor-1` full of raw data. We know that the data will grow over time, so there
will be a `sensor-2`, `sensor-3`, etc. Let's create a Rollup job, which stores it's data in `sensor_rollup`:
[source,console] [source,console]
-------------------------------------------------- --------------------------------------------------
@ -74,8 +81,8 @@ PUT _rollup/job/sensor
-------------------------------------------------- --------------------------------------------------
// TEST[setup:sensor_index] // TEST[setup:sensor_index]
If at a later date, we'd like to determine what jobs and capabilities were stored in the `sensor_rollup` index, we can use the Get Rollup If at a later date, we'd like to determine what jobs and capabilities were
Index API: stored in the `sensor_rollup` index, we can use the get rollup index API:
[source,console] [source,console]
-------------------------------------------------- --------------------------------------------------
@ -83,8 +90,8 @@ GET /sensor_rollup/_rollup/data
-------------------------------------------------- --------------------------------------------------
// TEST[continued] // TEST[continued]
Note how we are requesting the concrete rollup index name (`sensor_rollup`) as the first part of the URL. Note how we are requesting the concrete rollup index name (`sensor_rollup`) as
This will yield the following response: the first part of the URL. This will yield the following response:
[source,console-result] [source,console-result]
---- ----
@ -133,20 +140,24 @@ This will yield the following response:
---- ----
The response that is returned contains information that is similar to the original Rollup configuration, but formatted The response that is returned contains information that is similar to the
differently. First, there are some house-keeping details: the Rollup job's ID, the index that holds the rolled data, original rollup configuration, but formatted differently. First, there are some
house-keeping details: the {rollup-job} ID, the index that holds the rolled data,
the index pattern that the job was targeting. the index pattern that the job was targeting.
Next it shows a list of fields that contain data eligible for rollup searches. Here we see four fields: `node`, `temperature`, Next it shows a list of fields that contain data eligible for rollup searches.
`timestamp` and `voltage`. Each of these fields list the aggregations that are possible. For example, you can use a min, max Here we see four fields: `node`, `temperature`, `timestamp` and `voltage`. Each
or sum aggregation on the `temperature` field, but only a `date_histogram` on `timestamp`. of these fields list the aggregations that are possible. For example, you can
use a min, max, or sum aggregation on the `temperature` field, but only a
`date_histogram` on `timestamp`.
Note that the `rollup_jobs` element is an array; there can be multiple, independent jobs configured for a single index Note that the `rollup_jobs` element is an array; there can be multiple,
or index pattern. Each of these jobs may have different configurations, so the API returns a list of all the various independent jobs configured for a single index or index pattern. Each of these
configurations available. jobs may have different configurations, so the API returns a list of all the
various configurations available.
Like other APIs that interact with indices, you can specify index patterns
Like other APIs that interact with indices, you can specify index patterns instead of explicit indices: instead of explicit indices:
[source,console] [source,console]
-------------------------------------------------- --------------------------------------------------

View File

@ -6,50 +6,66 @@
<titleabbrev>Rollup search</titleabbrev> <titleabbrev>Rollup search</titleabbrev>
++++ ++++
Enables searching rolled-up data using the standard query DSL.
experimental[] experimental[]
The Rollup Search endpoint allows searching rolled-up data using the standard query DSL. The Rollup Search endpoint [[rollup-search-request]]
is needed because, internally, rolled-up documents utilize a different document structure than the original data. The ==== {api-request-title}
Rollup Search endpoint rewrites standard query DSL into a format that matches the rollup documents, then takes the response
and rewrites it back to what a client would expect given the original query.
==== Request `GET <index>/_rollup_search`
`GET {index}/_rollup_search` [[rollup-search-desc]]
==== {api-description-title}
//===== Description The rollup search endpoint is needed because, internally, rolled-up documents
utilize a different document structure than the original data. The rollup search
endpoint rewrites standard query DSL into a format that matches the rollup
documents, then takes the response and rewrites it back to what a client would
expect given the original query.
==== Path Parameters [[rollup-search-path-params]]
==== {api-path-parms-title}
`index`:: `<index>`::
(string) Index, indices or index-pattern to execute a rollup search against. This can include both rollup and non-rollup (Required, string) Index, indices or index-pattern to execute a rollup search
indices. against. This can include both rollup and non-rollup indices.
Rules for the `index` parameter: Rules for the `index` parameter:
- At least one index/index-pattern must be specified. This can be either a rollup or non-rollup index. Omitting the index parameter, - At least one index/index-pattern must be specified. This can be either a
or using `_all`, is not permitted rollup or non-rollup index. Omitting the index parameter, or using `_all`, is
not permitted.
- Multiple non-rollup indices may be specified - Multiple non-rollup indices may be specified
- Only one rollup index may be specified. If more than one are supplied an exception will be thrown - Only one rollup index may be specified. If more than one are supplied, an
- Index patterns may be used, but if they match more than one rollup index an exception will be thrown. exception occurs.
- Index patterns may be used, but if they match more than one rollup index an
exception occurs.
==== Request Body [[rollup-search-request-body]]
==== {api-request-body-title}
The request body supports a subset of features from the regular Search API. It supports: The request body supports a subset of features from the regular Search API. It
supports:
- `query` param for specifying an DSL query, subject to some limitations (see <<rollup-search-limitations>> and <<rollup-agg-limitations>> - `query` param for specifying an DSL query, subject to some limitations
(see <<rollup-search-limitations>> and <<rollup-agg-limitations>>
- `aggregations` param for specifying aggregations - `aggregations` param for specifying aggregations
Functionality that is not available: Functionality that is not available:
- `size`: because rollups work on pre-aggregated data, no search hits can be returned and so size must be set to zero or - `size`: Because rollups work on pre-aggregated data, no search hits can be
omitted entirely. returned and so size must be set to zero or omitted entirely.
- `highlighter`, `suggestors`, `post_filter`, `profile`, `explain` are similarly disallowed - `highlighter`, `suggestors`, `post_filter`, `profile`, `explain`: These are
similarly disallowed.
[[rollup-search-example]]
==== {api-examples-title}
==== Historical-only search example ===== Historical-only search example
Imagine we have an index named `sensor-1` full of raw data, and we have created a rollup job with the following configuration: Imagine we have an index named `sensor-1` full of raw data, and we have created
a {rollup-job} with the following configuration:
[source,console] [source,console]
-------------------------------------------------- --------------------------------------------------
@ -83,9 +99,10 @@ PUT _rollup/job/sensor
-------------------------------------------------- --------------------------------------------------
// TEST[setup:sensor_index] // TEST[setup:sensor_index]
This rolls up the `sensor-*` pattern and stores the results in `sensor_rollup`. To search this rolled up data, we This rolls up the `sensor-*` pattern and stores the results in `sensor_rollup`.
need to use the `_rollup_search` endpoint. However, you'll notice that we can use regular query DSL to search the To search this rolled up data, we need to use the `_rollup_search` endpoint.
rolled-up data: However, you'll notice that we can use regular query DSL to search the rolled-up
data:
[source,console] [source,console]
-------------------------------------------------- --------------------------------------------------
@ -104,8 +121,9 @@ GET /sensor_rollup/_rollup_search
// TEST[setup:sensor_prefab_data] // TEST[setup:sensor_prefab_data]
// TEST[s/_rollup_search/_rollup_search?filter_path=took,timed_out,terminated_early,_shards,hits,aggregations/] // TEST[s/_rollup_search/_rollup_search?filter_path=took,timed_out,terminated_early,_shards,hits,aggregations/]
The query is targeting the `sensor_rollup` data, since this contains the rollup data as configured in the job. A `max` The query is targeting the `sensor_rollup` data, since this contains the rollup
aggregation has been used on the `temperature` field, yielding the following response: data as configured in the job. A `max` aggregation has been used on the
`temperature` field, yielding the following response:
[source,console-result] [source,console-result]
---- ----
@ -132,12 +150,14 @@ aggregation has been used on the `temperature` field, yielding the following res
// TESTRESPONSE[s/"took" : 102/"took" : $body.$_path/] // TESTRESPONSE[s/"took" : 102/"took" : $body.$_path/]
// TESTRESPONSE[s/"_shards" : \.\.\. /"_shards" : $body.$_path/] // TESTRESPONSE[s/"_shards" : \.\.\. /"_shards" : $body.$_path/]
The response is exactly as you'd expect from a regular query + aggregation; it provides some metadata about the request The response is exactly as you'd expect from a regular query + aggregation; it
(`took`, `_shards`, etc), the search hits (which is always empty for rollup searches), and the aggregation response. provides some metadata about the request (`took`, `_shards`, etc), the search
hits (which is always empty for rollup searches), and the aggregation response.
Rollup searches are limited to functionality that was configured in the rollup job. For example, we are not able to calculate Rollup searches are limited to functionality that was configured in the
the average temperature because `avg` was not one of the configured metrics for the `temperature` field. If we try {rollup-job}. For example, we are not able to calculate the average temperature
to execute that search: because `avg` was not one of the configured metrics for the `temperature` field.
If we try to execute that search:
[source,console] [source,console]
-------------------------------------------------- --------------------------------------------------
@ -176,11 +196,11 @@ GET sensor_rollup/_rollup_search
---- ----
// TESTRESPONSE[s/"stack_trace": \.\.\./"stack_trace": $body.$_path/] // TESTRESPONSE[s/"stack_trace": \.\.\./"stack_trace": $body.$_path/]
==== Searching both historical rollup and non-rollup data ===== Searching both historical rollup and non-rollup data
The Rollup Search API has the capability to search across both "live", non-rollup data as well as the aggregated rollup
data. This is done by simply adding the live indices to the URI:
The rollup search API has the capability to search across both "live"
non-rollup data and the aggregated rollup data. This is done by simply adding
the live indices to the URI:
[source,console] [source,console]
-------------------------------------------------- --------------------------------------------------
@ -200,16 +220,18 @@ GET sensor-1,sensor_rollup/_rollup_search <1>
// TEST[s/_rollup_search/_rollup_search?filter_path=took,timed_out,terminated_early,_shards,hits,aggregations/] // TEST[s/_rollup_search/_rollup_search?filter_path=took,timed_out,terminated_early,_shards,hits,aggregations/]
<1> Note the URI now searches `sensor-1` and `sensor_rollup` at the same time <1> Note the URI now searches `sensor-1` and `sensor_rollup` at the same time
When the search is executed, the Rollup Search endpoint will do two things: When the search is executed, the rollup search endpoint does two things:
1. The original request will be sent to the non-rollup index unaltered 1. The original request is sent to the non-rollup index unaltered.
2. A rewritten version of the original request will be sent to the rollup index. 2. A rewritten version of the original request is sent to the rollup index.
When the two responses are received, the endpoint will then rewrite the rollup response and merge the two together. When the two responses are received, the endpoint rewrites the rollup response
During the merging process, if there is any overlap in buckets between the two responses, the buckets from the non-rollup and merges the two together. During the merging process, if there is any overlap
index will be used. in buckets between the two responses, the buckets from the non-rollup index are
used.
The response to the above query will look as expected, despite spanning rollup and non-rollup indices: The response to the above query looks as expected, despite spanning rollup and
non-rollup indices:
[source,console-result] [source,console-result]
---- ----

View File

@ -19,8 +19,8 @@ experimental[]
[[rollup-start-job-prereqs]] [[rollup-start-job-prereqs]]
==== {api-prereq-title} ==== {api-prereq-title}
* You must have `manage` or `manage_rollup` cluster privileges to use this API. * If the {es} {security-features} are enabled, you must have `manage` or
For more information, see `manage_rollup` cluster privileges to use this API. For more information, see
<<security-privileges>>. <<security-privileges>>.
[[rollup-start-job-desc]] [[rollup-start-job-desc]]

View File

@ -19,8 +19,8 @@ experimental[]
[[rollup-stop-job-prereqs]] [[rollup-stop-job-prereqs]]
==== {api-prereq-title} ==== {api-prereq-title}
* You must have `manage` or `manage_rollup` cluster privileges to use this API. * If the {es} {security-features} are enabled, you must have `manage` or
For more information, see `manage_rollup` cluster privileges to use this API. For more information, see
<<security-privileges>>. <<security-privileges>>.
[[rollup-stop-job-desc]] [[rollup-stop-job-desc]]