[DOCS] Adds collapsible sections to rollup APIs (#54690)
This commit is contained in:
Lisa Cawley
lcawl
parent
7666276b09
commit
11afead21e
|
|
@ -40,21 +40,32 @@ For details about a historical {rollup-job}, the
|
|||
`<job_id>`::
|
||||
(Optional, string) Identifier for the {rollup-job}. If it is `_all` or omitted,
|
||||
the API returns all {rollup-jobs}.
|
||||
|
||||
|
||||
[role="child_attributes"]
|
||||
[[rollup-get-job-response-body]]
|
||||
==== {api-response-body-title}
|
||||
|
||||
`jobs`::
|
||||
(array) An array of {rollup-job} resources.
|
||||
`config`:::
|
||||
(object) Contains the configuration for the {rollup-job}. This information
|
||||
is identical to the configuration that was supplied when creating the job
|
||||
via the <<rollup-put-job,create job API>>.
|
||||
`status`:::
|
||||
(object) Contains the current status of the indexer for the {rollup-job}.
|
||||
The possible values and their meanings are:
|
||||
(array) An array of {rollup-job} resources.
|
||||
+
|
||||
.Properties of {rollup-job} resources
|
||||
[%collapsible%open]
|
||||
====
|
||||
`config`:::
|
||||
(object) Contains the configuration for the {rollup-job}. This information is
|
||||
identical to the configuration that was supplied when creating the job via the
|
||||
<<rollup-put-job,create job API>>.
|
||||
|
||||
`stats`:::
|
||||
(object) Contains transient statistics about the {rollup-job}, such as how many
|
||||
documents have been processed and how many rollup summary docs have been
|
||||
indexed. These stats are not persisted. If a node is restarted, these stats are
|
||||
reset.
|
||||
|
||||
`status`:::
|
||||
(object) Contains the current status of the indexer for the {rollup-job}. The
|
||||
possible values and their meanings are:
|
||||
+
|
||||
--
|
||||
- `stopped` means the indexer is paused and will not process data, even if its
|
||||
cron interval triggers.
|
||||
- `started` means the indexer is running, but not actively indexing data. When
|
||||
|
|
@ -66,12 +77,7 @@ be ignored because the job is already active with the prior trigger.
|
|||
is used if the task needs to be shut down for some reason (job has been deleted,
|
||||
an unrecoverable error has been encountered, etc). Shortly after the `abort`
|
||||
state is set, the job will remove itself from the cluster.
|
||||
--
|
||||
`stats`:::
|
||||
(object) Contains transient statistics about the {rollup-job}, such as how
|
||||
many documents have been processed and how many rollup summary docs have
|
||||
been indexed. These stats are not persisted. If a node is restarted, these
|
||||
stats will be reset.
|
||||
====
|
||||
|
||||
[[rollup-get-job-example]]
|
||||
==== {api-examples-title}
|
||||
|
|
|
|||
|
|
@ -49,54 +49,59 @@ Jobs are created in a `STOPPED` state. You can start them with the
|
|||
create a new job with the same ID since that could lead to problems with
|
||||
mismatched job configurations.
|
||||
|
||||
[role="child_attributes"]
|
||||
[[rollup-put-job-api-request-body]]
|
||||
==== {api-request-body-title}
|
||||
|
||||
`cron`::
|
||||
(Required, string) A cron string which defines the intervals when the
|
||||
{rollup-job} should be executed. When the interval triggers, the indexer
|
||||
attempts to rollup the data in the index pattern. The cron pattern is
|
||||
unrelated to the time interval of the data being rolled up. For example, you
|
||||
may wish to create hourly rollups of your document but to only run the indexer
|
||||
on a daily basis at midnight, as defined by the cron. The cron pattern is
|
||||
defined just like a {watcher} cron schedule.
|
||||
(Required, string) A cron string which defines the intervals when the
|
||||
{rollup-job} should be executed. When the interval triggers, the indexer
|
||||
attempts to rollup the data in the index pattern. The cron pattern is
|
||||
unrelated to the time interval of the data being rolled up. For example, you
|
||||
may wish to create hourly rollups of your document but to only run the indexer
|
||||
on a daily basis at midnight, as defined by the cron. The cron pattern is
|
||||
defined just like a {watcher} cron schedule.
|
||||
|
||||
//Begin groups
|
||||
[[rollup-groups-config]]
|
||||
`groups`::
|
||||
(Required, object) Defines the grouping fields and aggregations that are
|
||||
defined for this {rollup-job}. These fields will then be available later for
|
||||
aggregating into buckets.
|
||||
(Required, object) Defines the grouping fields and aggregations that are
|
||||
defined for this {rollup-job}. These fields will then be available later for
|
||||
aggregating into buckets.
|
||||
+
|
||||
--
|
||||
These aggs and fields can be used in any combination. Think of the `groups`
|
||||
configuration as defining a set of tools that can later be used in aggregations
|
||||
to partition the data. Unlike raw data, we have to think ahead to which fields
|
||||
and aggregations might be used. Rollups provide enough flexibility that you
|
||||
simply need to determine _which_ fields are needed, not _in what order_ they are
|
||||
needed.
|
||||
|
||||
There are three types of groupings currently available:
|
||||
--
|
||||
|
||||
+
|
||||
There are three types of groupings currently available: `date_histogram`,
|
||||
`histogram`, and `terms`.
|
||||
+
|
||||
.Properties of `groups`
|
||||
[%collapsible%open]
|
||||
====
|
||||
//Begin date_histogram
|
||||
`date_histogram`:::
|
||||
(Required, object) A date histogram group aggregates a `date` field into
|
||||
time-based buckets. This group is *mandatory*; you currently cannot rollup
|
||||
documents without a timestamp and a `date_histogram` group. The
|
||||
`date_histogram` group has several parameters:
|
||||
|
||||
`field`::::
|
||||
(Required, string) The date field that is to be rolled up.
|
||||
|
||||
(Required, object) A date histogram group aggregates a `date` field into
|
||||
time-based buckets. This group is *mandatory*; you currently cannot rollup
|
||||
documents without a timestamp and a `date_histogram` group. The
|
||||
`date_histogram` group has several parameters:
|
||||
+
|
||||
.Properties of `date_histogram`
|
||||
[%collapsible%open]
|
||||
=====
|
||||
`calendar_interval` or `fixed_interval`::::
|
||||
(Required, <<time-units,time units>>) The interval of time buckets to be
|
||||
generated when rolling up. For example, `60m` produces 60 minute (hourly)
|
||||
rollups. This follows standard time formatting syntax as used elsewhere in
|
||||
{es}. The interval defines the _minimum_ interval that can be aggregated only.
|
||||
If hourly (`60m`) intervals are configured, <<rollup-search,rollup search>>
|
||||
can execute aggregations with 60m or greater (weekly, monthly, etc) intervals.
|
||||
So define the interval as the smallest unit that you wish to later query. For
|
||||
more information about the difference between calendar and fixed time
|
||||
intervals, see <<rollup-understanding-group-intervals>>.
|
||||
(Required, <<time-units,time units>>) The interval of time buckets to be
|
||||
generated when rolling up. For example, `60m` produces 60 minute (hourly)
|
||||
rollups. This follows standard time formatting syntax as used elsewhere in {es}.
|
||||
The interval defines the _minimum_ interval that can be aggregated only. If
|
||||
hourly (`60m`) intervals are configured, <<rollup-search,rollup search>>
|
||||
can execute aggregations with 60m or greater (weekly, monthly, etc) intervals.
|
||||
So define the interval as the smallest unit that you wish to later query. For
|
||||
more information about the difference between calendar and fixed time
|
||||
intervals, see <<rollup-understanding-group-intervals>>.
|
||||
+
|
||||
--
|
||||
NOTE: Smaller, more granular intervals take up proportionally more space.
|
||||
|
|
@ -104,12 +109,12 @@ NOTE: Smaller, more granular intervals take up proportionally more space.
|
|||
--
|
||||
|
||||
`delay`::::
|
||||
(Optional,<<time-units,time units>>) How long to wait before rolling up new
|
||||
documents. By default, the indexer attempts to roll up all data that is
|
||||
available. However, it is not uncommon for data to arrive out of order,
|
||||
sometimes even a few days late. The indexer is unable to deal with data that
|
||||
arrives after a time-span has been rolled up. That is to say, there is no
|
||||
provision to update already-existing rollups.
|
||||
(Optional,<<time-units,time units>>) How long to wait before rolling up new
|
||||
documents. By default, the indexer attempts to roll up all data that is
|
||||
available. However, it is not uncommon for data to arrive out of order,
|
||||
sometimes even a few days late. The indexer is unable to deal with data that
|
||||
arrives after a time-span has been rolled up. That is to say, there is no
|
||||
provision to update already-existing rollups.
|
||||
+
|
||||
--
|
||||
Instead, you should specify a `delay` that matches the longest period of time
|
||||
|
|
@ -118,18 +123,45 @@ instructs the indexer to roll up documents up to `now - 1d`, which provides
|
|||
a day of buffer time for out-of-order documents to arrive.
|
||||
--
|
||||
|
||||
`time_zone`::::
|
||||
(Optional, string) Defines what time_zone the rollup documents are stored as.
|
||||
Unlike raw data, which can shift timezones on the fly, rolled documents have
|
||||
to be stored with a specific timezone. By default, rollup documents are stored
|
||||
in `UTC`.
|
||||
`field`::::
|
||||
(Required, string) The date field that is to be rolled up.
|
||||
|
||||
`time_zone`::::
|
||||
(Optional, string) Defines what time_zone the rollup documents are stored as.
|
||||
Unlike raw data, which can shift timezones on the fly, rolled documents have to
|
||||
be stored with a specific timezone. By default, rollup documents are stored
|
||||
in `UTC`.
|
||||
=====
|
||||
//End date_histogram
|
||||
|
||||
//Begin histogram
|
||||
`histogram`:::
|
||||
(Optional, object) The histogram group aggregates one or more numeric fields
|
||||
into numeric histogram intervals.
|
||||
+
|
||||
.Properties of `histogram`
|
||||
[%collapsible%open]
|
||||
=====
|
||||
`fields`::::
|
||||
(Required, array) The set of fields that you wish to build histograms for. All
|
||||
fields specified must be some kind of numeric. Order does not matter.
|
||||
|
||||
`interval`::::
|
||||
(Required, integer) The interval of histogram buckets to be generated when
|
||||
rolling up. For example, a value of `5` creates buckets that are five units wide
|
||||
(`0-5`, `5-10`, etc). Note that only one interval can be specified in the
|
||||
`histogram` group, meaning that all fields being grouped via the histogram
|
||||
must share the same interval.
|
||||
=====
|
||||
//End histogram
|
||||
|
||||
//Begin terms
|
||||
`terms`:::
|
||||
(Optional, object) The terms group can be used on `keyword` or numeric fields
|
||||
to allow bucketing via the `terms` aggregation at a later point. The indexer
|
||||
enumerates and stores _all_ values of a field for each time-period. This can
|
||||
be potentially costly for high-cardinality groups such as IP addresses,
|
||||
especially if the time-bucket is particularly sparse.
|
||||
(Optional, object) The terms group can be used on `keyword` or numeric fields to
|
||||
allow bucketing via the `terms` aggregation at a later point. The indexer
|
||||
enumerates and stores _all_ values of a field for each time-period. This can be
|
||||
potentially costly for high-cardinality groups such as IP addresses, especially
|
||||
if the time-bucket is particularly sparse.
|
||||
+
|
||||
--
|
||||
TIP: While it is unlikely that a rollup will ever be larger in size than the raw
|
||||
|
|
@ -137,37 +169,25 @@ data, defining `terms` groups on multiple high-cardinality fields can
|
|||
effectively reduce the compression of a rollup to a large extent. You should be
|
||||
judicious which high-cardinality fields are included for that reason.
|
||||
|
||||
The `terms` group has a single parameter:
|
||||
--
|
||||
|
||||
`fields`::::
|
||||
(Required, string) The set of fields that you wish to collect terms for. This
|
||||
array can contain fields that are both `keyword` and numerics. Order does not
|
||||
matter.
|
||||
|
||||
`histogram`:::
|
||||
(Optional, object) The histogram group aggregates one or more numeric fields
|
||||
into numeric histogram intervals.
|
||||
+
|
||||
--
|
||||
The `histogram` group has a two parameters:
|
||||
--
|
||||
.Properties of `terms`
|
||||
[%collapsible%open]
|
||||
=====
|
||||
|
||||
`fields`::::
|
||||
(Required, array) The set of fields that you wish to build histograms for. All fields
|
||||
specified must be some kind of numeric. Order does not matter.
|
||||
|
||||
`interval`::::
|
||||
(Required, integer) The interval of histogram buckets to be generated when
|
||||
rolling up. For example, a value of `5` creates buckets that are five units
|
||||
wide (`0-5`, `5-10`, etc). Note that only one interval can be specified in the
|
||||
`histogram` group, meaning that all fields being grouped via the histogram
|
||||
must share the same interval.
|
||||
(Required, string) The set of fields that you wish to collect terms for. This
|
||||
array can contain fields that are both `keyword` and numerics. Order does not
|
||||
matter.
|
||||
=====
|
||||
//End terms
|
||||
====
|
||||
//End groups
|
||||
|
||||
`index_pattern`::
|
||||
(Required, string) The index or index pattern to roll up. Supports
|
||||
wildcard-style patterns (`logstash-*`). The job will
|
||||
attempt to rollup the entire index or index-pattern.
|
||||
(Required, string) The index or index pattern to roll up. Supports
|
||||
wildcard-style patterns (`logstash-*`). The job attempts to rollup the entire
|
||||
index or index-pattern.
|
||||
+
|
||||
--
|
||||
NOTE: The `index_pattern` cannot be a pattern that would also match the
|
||||
|
|
@ -179,39 +199,43 @@ prevent this behavior.
|
|||
|
||||
--
|
||||
|
||||
//Begin metrics
|
||||
[[rollup-metrics-config]]
|
||||
`metrics`::
|
||||
(Optional, object) Defines the metrics to collect for each grouping tuple.
|
||||
By default, only the doc_counts are collected for each group. To make rollup
|
||||
useful, you will often add metrics like averages, mins, maxes, etc. Metrics
|
||||
are defined on a per-field basis and for each field you configure which metric
|
||||
should be collected.
|
||||
(Optional, object) Defines the metrics to collect for each grouping tuple. By
|
||||
default, only the doc_counts are collected for each group. To make rollup useful,
|
||||
you will often add metrics like averages, mins, maxes, etc. Metrics are defined
|
||||
on a per-field basis and for each field you configure which metric should be
|
||||
collected.
|
||||
+
|
||||
--
|
||||
The `metrics` configuration accepts an array of objects, where each object has
|
||||
two parameters:
|
||||
--
|
||||
|
||||
two parameters.
|
||||
+
|
||||
.Properties of metric objects
|
||||
[%collapsible%open]
|
||||
====
|
||||
`field`:::
|
||||
(Required, string) The field to collect metrics for. This must be a numeric
|
||||
of some kind.
|
||||
(Required, string) The field to collect metrics for. This must be a numeric of
|
||||
some kind.
|
||||
|
||||
`metrics`:::
|
||||
(Required, array) An array of metrics to collect for the field. At least one
|
||||
metric must be configured. Acceptable metrics are `min`,`max`,`sum`,`avg`, and
|
||||
`value_count`.
|
||||
(Required, array) An array of metrics to collect for the field. At least one
|
||||
metric must be configured. Acceptable metrics are `min`,`max`,`sum`,`avg`, and
|
||||
`value_count`.
|
||||
====
|
||||
//End metrics
|
||||
|
||||
`page_size`::
|
||||
(Required, integer) The number of bucket results that are processed on each
|
||||
iteration of the rollup indexer. A larger value tends to execute faster, but
|
||||
requires more memory during processing. This value has no effect on how the
|
||||
data is rolled up; it is merely used for tweaking the speed or memory cost of
|
||||
the indexer.
|
||||
(Required, integer) The number of bucket results that are processed on each
|
||||
iteration of the rollup indexer. A larger value tends to execute faster, but
|
||||
requires more memory during processing. This value has no effect on how the data
|
||||
is rolled up; it is merely used for tweaking the speed or memory cost of
|
||||
the indexer.
|
||||
|
||||
`rollup_index`::
|
||||
(Required, string) The index that contains the rollup results. The index can
|
||||
be shared with other {rollup-jobs}. The data is stored so that it doesn't
|
||||
interfere with unrelated jobs.
|
||||
(Required, string) The index that contains the rollup results. The index can
|
||||
be shared with other {rollup-jobs}. The data is stored so that it doesn't
|
||||
interfere with unrelated jobs.
|
||||
|
||||
[[rollup-put-job-api-example]]
|
||||
==== {api-example-title}
|
||||
|
|
|
|||
Loading…
Reference in New Issue