[DOCS] Reformat rollup API docs (#49397)

This commit is contained in:
Lisa Cawley 2019-11-20 10:43:53 -08:00 committed by lcawl
parent d9fd4cc351
commit 0f15736687
7 changed files with 176 additions and 133 deletions

View File

@ -18,9 +18,9 @@ experimental[]
[[rollup-get-job-prereqs]]
==== {api-prereq-title}
* You must have `monitor`, `monitor_rollup`, `manage` or `manage_rollup` cluster
privileges to use this API. For more information, see
<<security-privileges>>.
* If the {es} {security-features} are enabled, you must have `monitor`,
`monitor_rollup`, `manage` or `manage_rollup` cluster privileges to use this API.
For more information, see <<security-privileges>>.
[[rollup-get-job-desc]]
==== {api-description-title}

View File

@ -32,7 +32,7 @@ Jobs are created in a `STOPPED` state. You can start them with the
[[rollup-put-job-api-path-params]]
==== {api-path-parms-title}
`job_id`::
`<job_id>`::
(Required, string) Identifier for the {rollup-job}.
[[rollup-put-job-api-request-body]]

View File

@ -1,17 +1,32 @@
[role="xpack"]
[testenv="basic"]
[[rollup-get-rollup-caps]]
=== Get rollup job capabilities API
=== Get {rollup-job} capabilities API
++++
<titleabbrev>Get rollup caps</titleabbrev>
++++
Returns the capabilities of any {rollup-jobs} that have been configured for a
specific index or index pattern.
experimental[]
This API returns the capabilities of any rollup jobs that have been configured
for a specific index or index pattern.
[[rollup-get-rollup-caps-request]]
==== {api-request-title}
This API is useful because a rollup job is often configured to rollup only a
`GET _rollup/data/<index>`
[[rollup-get-rollup-caps-prereqs]]
==== {api-prereq-title}
* If the {es} {security-features} are enabled, you must have `monitor`,
`monitor_rollup`, `manage` or `manage_rollup` cluster privileges to use this API.
For more information, see <<security-privileges>>.
[[rollup-get-rollup-caps-desc]]
==== {api-description-title}
This API is useful because a {rollup-job} is often configured to rollup only a
subset of fields from the source index. Furthermore, only certain aggregations
can be configured for various fields, leading to a limited subset of
functionality depending on that configuration.
@ -22,33 +37,20 @@ This API enables you to inspect an index and determine:
2. If yes to the first question, what fields were rolled up, what aggregations
can be performed, and where does the data live?
==== Request
[[rollup-get-rollup-path-params]]
==== {api-path-parms-title}
`GET _rollup/data/{index}`
//===== Description
==== Path Parameters
`index`::
(string) Index, indices or index-pattern to return rollup capabilities for. `_all` may be used to fetch
rollup capabilities from all jobs
`<index>`::
(string) Index, indices or index-pattern to return rollup capabilities for.
`_all` may be used to fetch rollup capabilities from all jobs.
==== Request Body
[[rollup-get-rollup-example]]
==== {api-examples-title}
There is no request body for the Get Rollup Caps API.
==== Authorization
You must have `monitor`, `monitor_rollup`, `manage` or `manage_rollup` cluster privileges to use this API.
For more information, see
<<security-privileges>>.
==== Examples
Imagine we have an index named `sensor-1` full of raw data. We know that the data will grow over time, so there
will be a `sensor-2`, `sensor-3`, etc. Let's create a Rollup job that targets the index pattern `sensor-*` to accommodate
Imagine we have an index named `sensor-1` full of raw data. We know that the
data will grow over time, so there will be a `sensor-2`, `sensor-3`, etc. Let's
create a {rollup-job} that targets the index pattern `sensor-*` to accommodate
this future scaling:
[source,console]
@ -83,7 +85,8 @@ PUT _rollup/job/sensor
--------------------------------------------------
// TEST[setup:sensor_index]
We can then retrieve the rollup capabilities of that index pattern (`sensor-*`) via the following command:
We can then retrieve the rollup capabilities of that index pattern (`sensor-*`)
via the following command:
[source,console]
--------------------------------------------------
@ -139,17 +142,21 @@ Which will yield the following response:
}
----
The response that is returned contains information that is similar to the original Rollup configuration, but formatted
differently. First, there are some house-keeping details: the Rollup job's ID, the index that holds the rolled data,
the index pattern that the job was targeting.
The response that is returned contains information that is similar to the
original rollup configuration, but formatted differently. First, there are some
house-keeping details: the {rollup-job} ID, the index that holds the rolled data,
and the index pattern that the job was targeting.
Next it shows a list of fields that contain data eligible for rollup searches. Here we see four fields: `node`, `temperature`,
`timestamp` and `voltage`. Each of these fields list the aggregations that are possible. For example, you can use a min, max
or sum aggregation on the `temperature` field, but only a `date_histogram` on `timestamp`.
Next it shows a list of fields that contain data eligible for rollup searches.
Here we see four fields: `node`, `temperature`, `timestamp` and `voltage`. Each
of these fields list the aggregations that are possible. For example, you can
use a min, max or sum aggregation on the `temperature` field, but only a
`date_histogram` on `timestamp`.
Note that the `rollup_jobs` element is an array; there can be multiple, independent jobs configured for a single index
or index pattern. Each of these jobs may have different configurations, so the API returns a list of all the various
configurations available.
Note that the `rollup_jobs` element is an array; there can be multiple,
independent jobs configured for a single index or index pattern. Each of these
jobs may have different configurations, so the API returns a list of all the
various configurations available.
We could also retrieve the same information with a request to `_all`:
@ -159,7 +166,8 @@ GET _rollup/data/_all
--------------------------------------------------
// TEST[continued]
But note that if we use the concrete index name (`sensor-1`), we'll retrieve no rollup capabilities:
But note that if we use the concrete index name (`sensor-1`), we'll retrieve no
rollup capabilities:
[source,console]
--------------------------------------------------
@ -174,7 +182,9 @@ GET _rollup/data/sensor-1
}
----
Why is this? The original rollup job was configured against a specific index pattern (`sensor-*`) not a concrete index
(`sensor-1`). So while the index belongs to the pattern, the rollup job is only valid across the entirety of the pattern
not just one of it's containing indices. So for that reason, the Rollup Capabilities API only returns information based
on the originally configured index name or pattern.
Why is this? The original {rollup-job} was configured against a specific index
pattern (`sensor-*`) not a concrete index (`sensor-1`). So while the index
belongs to the pattern, the {rollup-job} is only valid across the entirety of
the pattern not just one of it's containing indices. So for that reason, the
get rollup capabilities API only returns information based on the originally
configured index name or pattern.

View File

@ -6,41 +6,48 @@
<titleabbrev>Get rollup index caps</titleabbrev>
++++
Returns the rollup capabilities of all jobs inside of a rollup index (e.g. the
index where rollup data is stored).
experimental[]
This API returns the rollup capabilities of all jobs inside of a rollup index (e.g. the index where rollup data is stored).
A single rollup index may store the data for multiple rollup jobs, and may have a variety of capabilities depending on those jobs.
[[rollup-get-rollup-index-caps-request]]
==== {api-request-title}
`GET <index>/_rollup/data`
[[rollup-get-rollup-index-caps-prereqs]]
==== {api-prereq-title}
* If the {es} {security-features} are enabled, you must have the `read` index
privilege on the index that stores the rollup results. For more information, see
<<security-privileges>>.
[[rollup-get-rollup-index-caps-desc]]
==== {api-description-title}
A single rollup index may store the data for multiple {rollup-jobs}, and may
have a variety of capabilities depending on those jobs.
This API will allow you to determine:
1. What jobs are stored in an index (or indices specified via a pattern)?
2. What target indices were rolled up, what fields were used in those rollups and what aggregations can be performed on each job?
2. What target indices were rolled up, what fields were used in those rollups
and what aggregations can be performed on each job?
==== Request
[[rollup-get-rollup-index-caps-path-params]]
==== {api-path-parms-title}
`GET {index}/_rollup/data`
`<index>`::
(Required, string) Index or index-pattern of concrete rollup indices to check
for capabilities.
//===== Description
[[rollup-get-rollup-index-caps-example]]
==== {api-examples-title}
==== Path Parameters
`index`::
(string) Index or index-pattern of concrete rollup indices to check for capabilities.
==== Request Body
There is no request body for the Get Jobs API.
==== Authorization
You must have the `read` index privilege on the index that stores the rollup results.
For more information, see
<<security-privileges>>.
==== Examples
Imagine we have an index named `sensor-1` full of raw data. We know that the data will grow over time, so there
will be a `sensor-2`, `sensor-3`, etc. Let's create a Rollup job, which stores it's data in `sensor_rollup`:
Imagine we have an index named `sensor-1` full of raw data. We know that the
data will grow over time, so there will be a `sensor-2`, `sensor-3`, etc.
Let's create a {rollup-job} that stores its data in `sensor_rollup`:
[source,console]
--------------------------------------------------
@ -74,8 +81,8 @@ PUT _rollup/job/sensor
--------------------------------------------------
// TEST[setup:sensor_index]
If at a later date, we'd like to determine what jobs and capabilities were stored in the `sensor_rollup` index, we can use the Get Rollup
Index API:
If at a later date, we'd like to determine what jobs and capabilities were
stored in the `sensor_rollup` index, we can use the get rollup index API:
[source,console]
--------------------------------------------------
@ -83,8 +90,8 @@ GET /sensor_rollup/_rollup/data
--------------------------------------------------
// TEST[continued]
Note how we are requesting the concrete rollup index name (`sensor_rollup`) as the first part of the URL.
This will yield the following response:
Note how we are requesting the concrete rollup index name (`sensor_rollup`) as
the first part of the URL. This will yield the following response:
[source,console-result]
----
@ -133,20 +140,24 @@ This will yield the following response:
----
The response that is returned contains information that is similar to the original Rollup configuration, but formatted
differently. First, there are some house-keeping details: the Rollup job's ID, the index that holds the rolled data,
The response that is returned contains information that is similar to the
original rollup configuration, but formatted differently. First, there are some
house-keeping details: the {rollup-job} ID, the index that holds the rolled data,
the index pattern that the job was targeting.
Next it shows a list of fields that contain data eligible for rollup searches. Here we see four fields: `node`, `temperature`,
`timestamp` and `voltage`. Each of these fields list the aggregations that are possible. For example, you can use a min, max
or sum aggregation on the `temperature` field, but only a `date_histogram` on `timestamp`.
Next it shows a list of fields that contain data eligible for rollup searches.
Here we see four fields: `node`, `temperature`, `timestamp` and `voltage`. Each
of these fields list the aggregations that are possible. For example, you can
use a min, max, or sum aggregation on the `temperature` field, but only a
`date_histogram` on `timestamp`.
Note that the `rollup_jobs` element is an array; there can be multiple, independent jobs configured for a single index
or index pattern. Each of these jobs may have different configurations, so the API returns a list of all the various
configurations available.
Note that the `rollup_jobs` element is an array; there can be multiple,
independent jobs configured for a single index or index pattern. Each of these
jobs may have different configurations, so the API returns a list of all the
various configurations available.
Like other APIs that interact with indices, you can specify index patterns instead of explicit indices:
Like other APIs that interact with indices, you can specify index patterns
instead of explicit indices:
[source,console]
--------------------------------------------------

View File

@ -6,50 +6,66 @@
<titleabbrev>Rollup search</titleabbrev>
++++
Enables searching rolled-up data using the standard query DSL.
experimental[]
The Rollup Search endpoint allows searching rolled-up data using the standard query DSL. The Rollup Search endpoint
is needed because, internally, rolled-up documents utilize a different document structure than the original data. The
Rollup Search endpoint rewrites standard query DSL into a format that matches the rollup documents, then takes the response
and rewrites it back to what a client would expect given the original query.
[[rollup-search-request]]
==== {api-request-title}
==== Request
`GET <index>/_rollup_search`
`GET {index}/_rollup_search`
[[rollup-search-desc]]
==== {api-description-title}
//===== Description
The rollup search endpoint is needed because, internally, rolled-up documents
utilize a different document structure than the original data. The rollup search
endpoint rewrites standard query DSL into a format that matches the rollup
documents, then takes the response and rewrites it back to what a client would
expect given the original query.
==== Path Parameters
[[rollup-search-path-params]]
==== {api-path-parms-title}
`index`::
(string) Index, indices or index-pattern to execute a rollup search against. This can include both rollup and non-rollup
indices.
`<index>`::
(Required, string) Index, indices or index-pattern to execute a rollup search
against. This can include both rollup and non-rollup indices.
Rules for the `index` parameter:
- At least one index/index-pattern must be specified. This can be either a rollup or non-rollup index. Omitting the index parameter,
or using `_all`, is not permitted
- At least one index/index-pattern must be specified. This can be either a
rollup or non-rollup index. Omitting the index parameter, or using `_all`, is
not permitted.
- Multiple non-rollup indices may be specified
- Only one rollup index may be specified. If more than one are supplied an exception will be thrown
- Index patterns may be used, but if they match more than one rollup index an exception will be thrown.
- Only one rollup index may be specified. If more than one are supplied, an
exception occurs.
- Index patterns may be used, but if they match more than one rollup index an
exception occurs.
==== Request Body
[[rollup-search-request-body]]
==== {api-request-body-title}
The request body supports a subset of features from the regular Search API. It supports:
The request body supports a subset of features from the regular Search API. It
supports:
- `query` param for specifying an DSL query, subject to some limitations (see <<rollup-search-limitations>> and <<rollup-agg-limitations>>
- `query` param for specifying an DSL query, subject to some limitations
(see <<rollup-search-limitations>> and <<rollup-agg-limitations>>
- `aggregations` param for specifying aggregations
Functionality that is not available:
- `size`: because rollups work on pre-aggregated data, no search hits can be returned and so size must be set to zero or
omitted entirely.
- `highlighter`, `suggestors`, `post_filter`, `profile`, `explain` are similarly disallowed
- `size`: Because rollups work on pre-aggregated data, no search hits can be
returned and so size must be set to zero or omitted entirely.
- `highlighter`, `suggestors`, `post_filter`, `profile`, `explain`: These are
similarly disallowed.
[[rollup-search-example]]
==== {api-examples-title}
==== Historical-only search example
===== Historical-only search example
Imagine we have an index named `sensor-1` full of raw data, and we have created a rollup job with the following configuration:
Imagine we have an index named `sensor-1` full of raw data, and we have created
a {rollup-job} with the following configuration:
[source,console]
--------------------------------------------------
@ -83,9 +99,10 @@ PUT _rollup/job/sensor
--------------------------------------------------
// TEST[setup:sensor_index]
This rolls up the `sensor-*` pattern and stores the results in `sensor_rollup`. To search this rolled up data, we
need to use the `_rollup_search` endpoint. However, you'll notice that we can use regular query DSL to search the
rolled-up data:
This rolls up the `sensor-*` pattern and stores the results in `sensor_rollup`.
To search this rolled up data, we need to use the `_rollup_search` endpoint.
However, you'll notice that we can use regular query DSL to search the rolled-up
data:
[source,console]
--------------------------------------------------
@ -104,8 +121,9 @@ GET /sensor_rollup/_rollup_search
// TEST[setup:sensor_prefab_data]
// TEST[s/_rollup_search/_rollup_search?filter_path=took,timed_out,terminated_early,_shards,hits,aggregations/]
The query is targeting the `sensor_rollup` data, since this contains the rollup data as configured in the job. A `max`
aggregation has been used on the `temperature` field, yielding the following response:
The query is targeting the `sensor_rollup` data, since this contains the rollup
data as configured in the job. A `max` aggregation has been used on the
`temperature` field, yielding the following response:
[source,console-result]
----
@ -132,12 +150,14 @@ aggregation has been used on the `temperature` field, yielding the following res
// TESTRESPONSE[s/"took" : 102/"took" : $body.$_path/]
// TESTRESPONSE[s/"_shards" : \.\.\. /"_shards" : $body.$_path/]
The response is exactly as you'd expect from a regular query + aggregation; it provides some metadata about the request
(`took`, `_shards`, etc), the search hits (which is always empty for rollup searches), and the aggregation response.
The response is exactly as you'd expect from a regular query + aggregation; it
provides some metadata about the request (`took`, `_shards`, etc), the search
hits (which is always empty for rollup searches), and the aggregation response.
Rollup searches are limited to functionality that was configured in the rollup job. For example, we are not able to calculate
the average temperature because `avg` was not one of the configured metrics for the `temperature` field. If we try
to execute that search:
Rollup searches are limited to functionality that was configured in the
{rollup-job}. For example, we are not able to calculate the average temperature
because `avg` was not one of the configured metrics for the `temperature` field.
If we try to execute that search:
[source,console]
--------------------------------------------------
@ -176,11 +196,11 @@ GET sensor_rollup/_rollup_search
----
// TESTRESPONSE[s/"stack_trace": \.\.\./"stack_trace": $body.$_path/]
==== Searching both historical rollup and non-rollup data
The Rollup Search API has the capability to search across both "live", non-rollup data as well as the aggregated rollup
data. This is done by simply adding the live indices to the URI:
===== Searching both historical rollup and non-rollup data
The rollup search API has the capability to search across both "live"
non-rollup data and the aggregated rollup data. This is done by simply adding
the live indices to the URI:
[source,console]
--------------------------------------------------
@ -200,16 +220,18 @@ GET sensor-1,sensor_rollup/_rollup_search <1>
// TEST[s/_rollup_search/_rollup_search?filter_path=took,timed_out,terminated_early,_shards,hits,aggregations/]
<1> Note the URI now searches `sensor-1` and `sensor_rollup` at the same time
When the search is executed, the Rollup Search endpoint will do two things:
When the search is executed, the rollup search endpoint does two things:
1. The original request will be sent to the non-rollup index unaltered
2. A rewritten version of the original request will be sent to the rollup index.
1. The original request is sent to the non-rollup index unaltered.
2. A rewritten version of the original request is sent to the rollup index.
When the two responses are received, the endpoint will then rewrite the rollup response and merge the two together.
During the merging process, if there is any overlap in buckets between the two responses, the buckets from the non-rollup
index will be used.
When the two responses are received, the endpoint rewrites the rollup response
and merges the two together. During the merging process, if there is any overlap
in buckets between the two responses, the buckets from the non-rollup index are
used.
The response to the above query will look as expected, despite spanning rollup and non-rollup indices:
The response to the above query looks as expected, despite spanning rollup and
non-rollup indices:
[source,console-result]
----

View File

@ -19,8 +19,8 @@ experimental[]
[[rollup-start-job-prereqs]]
==== {api-prereq-title}
* You must have `manage` or `manage_rollup` cluster privileges to use this API.
For more information, see
* If the {es} {security-features} are enabled, you must have `manage` or
`manage_rollup` cluster privileges to use this API. For more information, see
<<security-privileges>>.
[[rollup-start-job-desc]]

View File

@ -19,8 +19,8 @@ experimental[]
[[rollup-stop-job-prereqs]]
==== {api-prereq-title}
* You must have `manage` or `manage_rollup` cluster privileges to use this API.
For more information, see
* If the {es} {security-features} are enabled, you must have `manage` or
`manage_rollup` cluster privileges to use this API. For more information, see
<<security-privileges>>.
[[rollup-stop-job-desc]]