[Docs] All Rollup docs experimental, agg limitations, clarify DeleteJob (#31299)

- All rollup pages should be marked as experimental instead of just
the top page
- While the job config docs state which aggregations are allowed, adding
a section which specifically details this in one place is more convenient
for the user
- Add a clarification that the DeleteJob API does not delete the rollup
data, just the rollup job.
This commit is contained in:
Zachary Tong 2018-06-13 15:42:20 -04:00 committed by GitHub
parent 664903a70a
commit d4262de83a
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
15 changed files with 81 additions and 3 deletions

View File

@ -5,9 +5,37 @@
<titleabbrev>Delete Job</titleabbrev>
++++
experimental[]
This API deletes an existing rollup job. The job can be started or stopped, in both cases it will be deleted. Attempting
to delete a non-existing job will throw an exception
.Deleting the job does not delete rolled up data
**********************************
When a job is deleted, that only removes the process that is actively monitoring and rolling up data.
It does not delete any previously rolled up data. This is by design; a user may wish to roll up a static dataset. Because
the dataset is static, once it has been fully rolled up there is no need to keep the indexing Rollup job around (as there
will be no new data). So the job may be deleted, leaving behind the rolled up data for analysis.
If you wish to also remove the rollup data, and the rollup index only contains the data for a single job, you can simply
delete the whole rollup index. If the rollup index stores data from several jobs, you must issue a Delete-By-Query that
targets the Rollup job's ID in the rollup index:
[source,js]
--------------------------------------------------
POST my_rollup_index/_delete_by_query
{
"query": {
"term": {
"_rollup.id": "the_rollup_job_id"
}
}
}
--------------------------------------------------
// NOTCONSOLE
**********************************
==== Request
`DELETE _xpack/rollup/job/<job_id>`

View File

@ -5,6 +5,8 @@
<titleabbrev>Get Job</titleabbrev>
++++
experimental[]
This API returns the configuration, stats and status of rollup jobs. The API can return the details for a single job,
or for all jobs.

View File

@ -5,6 +5,8 @@
<titleabbrev>Create Job</titleabbrev>
++++
experimental[]
This API enables you to create a rollup job. The job will be created in a `STOPPED` state, and must be
started with the <<rollup-start-job,Start Job API>>.

View File

@ -5,6 +5,8 @@
<titleabbrev>Get Rollup Caps</titleabbrev>
++++
experimental[]
This API returns the rollup capabilities that have been configured for an index or index pattern. This API is useful
because a rollup job is often configured to rollup only a subset of fields from the source index. Furthermore, only
certain aggregations can be configured for various fields, leading to a limited subset of functionality depending on

View File

@ -2,6 +2,8 @@
[[rollup-job-config]]
=== Rollup Job Configuration
experimental[]
The Rollup Job Configuration contains all the details about how the rollup job should run, when it indexes documents,
and what future queries will be able to execute against the rollup index.

View File

@ -5,6 +5,8 @@
<titleabbrev>Rollup Search</titleabbrev>
++++
experimental[]
The Rollup Search endpoint allows searching rolled-up data using the standard query DSL. The Rollup Search endpoint
is needed because, internally, rolled-up documents utilize a different document structure than the original data. The
Rollup Search endpoint rewrites standard query DSL into a format that matches the rollup documents, then takes the response

View File

@ -5,6 +5,8 @@
<titleabbrev>Start Job</titleabbrev>
++++
experimental[]
This API starts an existing, stopped rollup job. If the job does not exist an exception will be thrown.
Starting an already started job has no action.

View File

@ -5,6 +5,8 @@
<titleabbrev>Stop Job</titleabbrev>
++++
experimental[]
This API stops an existing, started rollup job. If the job does not exist an exception will be thrown.
Stopping an already stopped job has no action.

View File

@ -1,6 +1,8 @@
[[rollup-api-quickref]]
== API Quick Reference
experimental[]
Most {rollup} endpoints have the following base:
[source,js]

View File

@ -18,7 +18,8 @@ for analysis, but at a fraction of the storage cost of raw data.
* <<rollup-getting-started,Getting Started>>
* <<rollup-api-quickref, API Quick Reference>>
* <<rollup-understanding-groups,Understanding Rollup Grouping>>
* <<rollup-search-limitations,Limitations of Rollup Search>>
* <<rollup-agg-limitations,Rollup aggregation limitations>>
* <<rollup-search-limitations,Rollup Search limitations>>
--
@ -27,4 +28,5 @@ include::overview.asciidoc[]
include::api-quickref.asciidoc[]
include::rollup-getting-started.asciidoc[]
include::understanding-groups.asciidoc[]
include::rollup-agg-limitations.asciidoc[]
include::rollup-search-limitations.asciidoc[]

View File

@ -1,6 +1,8 @@
[[rollup-overview]]
== Overview
experimental[]
Time-based data (documents that are predominantly identified by their timestamp) often have associated retention policies
to manage data growth. For example, your system may be generating 500,000 documents every second. That will generate
43 million documents per day, and nearly 16 billion documents a year.

View File

@ -0,0 +1,24 @@
[[rollup-agg-limitations]]
== Rollup Aggregation Limitations
experimental[]
There are some limitations to how fields can be rolled up / aggregated. This page highlights the major limitations so that
you are aware of them.
[float]
=== Limited aggregation components
The Rollup functionality allows fields to be grouped with the following aggregations:
- Date Histogram aggregation
- Histogram aggregation
- Terms aggregation
And the following metrics are allowed to be specified for numeric fields:
- Min aggregation
- Max aggregation
- Sum aggregation
- Average aggregation
- Value Count aggregation

View File

@ -1,6 +1,8 @@
[[rollup-getting-started]]
== Getting Started
experimental[]
To use the Rollup feature, you need to create one or more "Rollup Jobs". These jobs run continuously in the background
and rollup the index or indices that you specify, placing the rolled documents in a secondary index (also of your choosing).

View File

@ -1,6 +1,8 @@
[[rollup-search-limitations]]
== Rollup Search Limitations
experimental[]
While we feel the Rollup function is extremely flexible, the nature of summarizing data means there will be some limitations. Once
live data is thrown away, you will always lose some flexibility.
@ -100,8 +102,8 @@ The Rollup functionality allows `query`'s in the search request, but with a limi
- MatchAll Query
- Any compound query (Boolean, Boosting, ConstantScore, etc)
Furthermore, these queries can only use fields that were also saved in the rollup job. If you wish to filter on a keyword `hostname` field,
that field must have been configured in the rollup job under a `terms` grouping.
Furthermore, these queries can only use fields that were also saved in the rollup job as a `group`.
If you wish to filter on a keyword `hostname` field, that field must have been configured in the rollup job under a `terms` grouping.
If you attempt to use an unsupported query, or the query references a field that wasn't configured in the rollup job, an exception will be
thrown. We expect the list of support queries to grow over time as more are implemented.

View File

@ -1,6 +1,8 @@
[[rollup-understanding-groups]]
== Understanding Groups
experimental[]
To preserve flexibility, Rollup Jobs are defined based on how future queries may need to use the data. Traditionally, systems force
the admin to make decisions about what metrics to rollup and on what interval. E.g. The average of `cpu_time` on an hourly basis. This
is limiting; if, at a future date, the admin wishes to see the average of `cpu_time` on an hourly basis _and partitioned by `host_name`_,