mirror of
https://github.com/honeymoose/OpenSearch.git
synced 2025-02-17 02:14:54 +00:00
[DOCS] Adds collapsible sections to rollup APIs (#54690)
This commit is contained in:
parent
7666276b09
commit
11afead21e
@ -40,21 +40,32 @@ For details about a historical {rollup-job}, the
|
|||||||
`<job_id>`::
|
`<job_id>`::
|
||||||
(Optional, string) Identifier for the {rollup-job}. If it is `_all` or omitted,
|
(Optional, string) Identifier for the {rollup-job}. If it is `_all` or omitted,
|
||||||
the API returns all {rollup-jobs}.
|
the API returns all {rollup-jobs}.
|
||||||
|
|
||||||
|
[role="child_attributes"]
|
||||||
[[rollup-get-job-response-body]]
|
[[rollup-get-job-response-body]]
|
||||||
==== {api-response-body-title}
|
==== {api-response-body-title}
|
||||||
|
|
||||||
`jobs`::
|
`jobs`::
|
||||||
(array) An array of {rollup-job} resources.
|
(array) An array of {rollup-job} resources.
|
||||||
`config`:::
|
+
|
||||||
(object) Contains the configuration for the {rollup-job}. This information
|
.Properties of {rollup-job} resources
|
||||||
is identical to the configuration that was supplied when creating the job
|
[%collapsible%open]
|
||||||
via the <<rollup-put-job,create job API>>.
|
====
|
||||||
`status`:::
|
`config`:::
|
||||||
(object) Contains the current status of the indexer for the {rollup-job}.
|
(object) Contains the configuration for the {rollup-job}. This information is
|
||||||
The possible values and their meanings are:
|
identical to the configuration that was supplied when creating the job via the
|
||||||
|
<<rollup-put-job,create job API>>.
|
||||||
|
|
||||||
|
`stats`:::
|
||||||
|
(object) Contains transient statistics about the {rollup-job}, such as how many
|
||||||
|
documents have been processed and how many rollup summary docs have been
|
||||||
|
indexed. These stats are not persisted. If a node is restarted, these stats are
|
||||||
|
reset.
|
||||||
|
|
||||||
|
`status`:::
|
||||||
|
(object) Contains the current status of the indexer for the {rollup-job}. The
|
||||||
|
possible values and their meanings are:
|
||||||
+
|
+
|
||||||
--
|
|
||||||
- `stopped` means the indexer is paused and will not process data, even if its
|
- `stopped` means the indexer is paused and will not process data, even if its
|
||||||
cron interval triggers.
|
cron interval triggers.
|
||||||
- `started` means the indexer is running, but not actively indexing data. When
|
- `started` means the indexer is running, but not actively indexing data. When
|
||||||
@ -66,12 +77,7 @@ be ignored because the job is already active with the prior trigger.
|
|||||||
is used if the task needs to be shut down for some reason (job has been deleted,
|
is used if the task needs to be shut down for some reason (job has been deleted,
|
||||||
an unrecoverable error has been encountered, etc). Shortly after the `abort`
|
an unrecoverable error has been encountered, etc). Shortly after the `abort`
|
||||||
state is set, the job will remove itself from the cluster.
|
state is set, the job will remove itself from the cluster.
|
||||||
--
|
====
|
||||||
`stats`:::
|
|
||||||
(object) Contains transient statistics about the {rollup-job}, such as how
|
|
||||||
many documents have been processed and how many rollup summary docs have
|
|
||||||
been indexed. These stats are not persisted. If a node is restarted, these
|
|
||||||
stats will be reset.
|
|
||||||
|
|
||||||
[[rollup-get-job-example]]
|
[[rollup-get-job-example]]
|
||||||
==== {api-examples-title}
|
==== {api-examples-title}
|
||||||
|
@ -49,54 +49,59 @@ Jobs are created in a `STOPPED` state. You can start them with the
|
|||||||
create a new job with the same ID since that could lead to problems with
|
create a new job with the same ID since that could lead to problems with
|
||||||
mismatched job configurations.
|
mismatched job configurations.
|
||||||
|
|
||||||
|
[role="child_attributes"]
|
||||||
[[rollup-put-job-api-request-body]]
|
[[rollup-put-job-api-request-body]]
|
||||||
==== {api-request-body-title}
|
==== {api-request-body-title}
|
||||||
|
|
||||||
`cron`::
|
`cron`::
|
||||||
(Required, string) A cron string which defines the intervals when the
|
(Required, string) A cron string which defines the intervals when the
|
||||||
{rollup-job} should be executed. When the interval triggers, the indexer
|
{rollup-job} should be executed. When the interval triggers, the indexer
|
||||||
attempts to rollup the data in the index pattern. The cron pattern is
|
attempts to rollup the data in the index pattern. The cron pattern is
|
||||||
unrelated to the time interval of the data being rolled up. For example, you
|
unrelated to the time interval of the data being rolled up. For example, you
|
||||||
may wish to create hourly rollups of your document but to only run the indexer
|
may wish to create hourly rollups of your document but to only run the indexer
|
||||||
on a daily basis at midnight, as defined by the cron. The cron pattern is
|
on a daily basis at midnight, as defined by the cron. The cron pattern is
|
||||||
defined just like a {watcher} cron schedule.
|
defined just like a {watcher} cron schedule.
|
||||||
|
|
||||||
|
//Begin groups
|
||||||
[[rollup-groups-config]]
|
[[rollup-groups-config]]
|
||||||
`groups`::
|
`groups`::
|
||||||
(Required, object) Defines the grouping fields and aggregations that are
|
(Required, object) Defines the grouping fields and aggregations that are
|
||||||
defined for this {rollup-job}. These fields will then be available later for
|
defined for this {rollup-job}. These fields will then be available later for
|
||||||
aggregating into buckets.
|
aggregating into buckets.
|
||||||
+
|
+
|
||||||
--
|
|
||||||
These aggs and fields can be used in any combination. Think of the `groups`
|
These aggs and fields can be used in any combination. Think of the `groups`
|
||||||
configuration as defining a set of tools that can later be used in aggregations
|
configuration as defining a set of tools that can later be used in aggregations
|
||||||
to partition the data. Unlike raw data, we have to think ahead to which fields
|
to partition the data. Unlike raw data, we have to think ahead to which fields
|
||||||
and aggregations might be used. Rollups provide enough flexibility that you
|
and aggregations might be used. Rollups provide enough flexibility that you
|
||||||
simply need to determine _which_ fields are needed, not _in what order_ they are
|
simply need to determine _which_ fields are needed, not _in what order_ they are
|
||||||
needed.
|
needed.
|
||||||
|
+
|
||||||
There are three types of groupings currently available:
|
There are three types of groupings currently available: `date_histogram`,
|
||||||
--
|
`histogram`, and `terms`.
|
||||||
|
+
|
||||||
|
.Properties of `groups`
|
||||||
|
[%collapsible%open]
|
||||||
|
====
|
||||||
|
//Begin date_histogram
|
||||||
`date_histogram`:::
|
`date_histogram`:::
|
||||||
(Required, object) A date histogram group aggregates a `date` field into
|
(Required, object) A date histogram group aggregates a `date` field into
|
||||||
time-based buckets. This group is *mandatory*; you currently cannot rollup
|
time-based buckets. This group is *mandatory*; you currently cannot rollup
|
||||||
documents without a timestamp and a `date_histogram` group. The
|
documents without a timestamp and a `date_histogram` group. The
|
||||||
`date_histogram` group has several parameters:
|
`date_histogram` group has several parameters:
|
||||||
|
+
|
||||||
`field`::::
|
.Properties of `date_histogram`
|
||||||
(Required, string) The date field that is to be rolled up.
|
[%collapsible%open]
|
||||||
|
=====
|
||||||
`calendar_interval` or `fixed_interval`::::
|
`calendar_interval` or `fixed_interval`::::
|
||||||
(Required, <<time-units,time units>>) The interval of time buckets to be
|
(Required, <<time-units,time units>>) The interval of time buckets to be
|
||||||
generated when rolling up. For example, `60m` produces 60 minute (hourly)
|
generated when rolling up. For example, `60m` produces 60 minute (hourly)
|
||||||
rollups. This follows standard time formatting syntax as used elsewhere in
|
rollups. This follows standard time formatting syntax as used elsewhere in {es}.
|
||||||
{es}. The interval defines the _minimum_ interval that can be aggregated only.
|
The interval defines the _minimum_ interval that can be aggregated only. If
|
||||||
If hourly (`60m`) intervals are configured, <<rollup-search,rollup search>>
|
hourly (`60m`) intervals are configured, <<rollup-search,rollup search>>
|
||||||
can execute aggregations with 60m or greater (weekly, monthly, etc) intervals.
|
can execute aggregations with 60m or greater (weekly, monthly, etc) intervals.
|
||||||
So define the interval as the smallest unit that you wish to later query. For
|
So define the interval as the smallest unit that you wish to later query. For
|
||||||
more information about the difference between calendar and fixed time
|
more information about the difference between calendar and fixed time
|
||||||
intervals, see <<rollup-understanding-group-intervals>>.
|
intervals, see <<rollup-understanding-group-intervals>>.
|
||||||
+
|
+
|
||||||
--
|
--
|
||||||
NOTE: Smaller, more granular intervals take up proportionally more space.
|
NOTE: Smaller, more granular intervals take up proportionally more space.
|
||||||
@ -104,12 +109,12 @@ NOTE: Smaller, more granular intervals take up proportionally more space.
|
|||||||
--
|
--
|
||||||
|
|
||||||
`delay`::::
|
`delay`::::
|
||||||
(Optional,<<time-units,time units>>) How long to wait before rolling up new
|
(Optional,<<time-units,time units>>) How long to wait before rolling up new
|
||||||
documents. By default, the indexer attempts to roll up all data that is
|
documents. By default, the indexer attempts to roll up all data that is
|
||||||
available. However, it is not uncommon for data to arrive out of order,
|
available. However, it is not uncommon for data to arrive out of order,
|
||||||
sometimes even a few days late. The indexer is unable to deal with data that
|
sometimes even a few days late. The indexer is unable to deal with data that
|
||||||
arrives after a time-span has been rolled up. That is to say, there is no
|
arrives after a time-span has been rolled up. That is to say, there is no
|
||||||
provision to update already-existing rollups.
|
provision to update already-existing rollups.
|
||||||
+
|
+
|
||||||
--
|
--
|
||||||
Instead, you should specify a `delay` that matches the longest period of time
|
Instead, you should specify a `delay` that matches the longest period of time
|
||||||
@ -118,18 +123,45 @@ instructs the indexer to roll up documents up to `now - 1d`, which provides
|
|||||||
a day of buffer time for out-of-order documents to arrive.
|
a day of buffer time for out-of-order documents to arrive.
|
||||||
--
|
--
|
||||||
|
|
||||||
`time_zone`::::
|
`field`::::
|
||||||
(Optional, string) Defines what time_zone the rollup documents are stored as.
|
(Required, string) The date field that is to be rolled up.
|
||||||
Unlike raw data, which can shift timezones on the fly, rolled documents have
|
|
||||||
to be stored with a specific timezone. By default, rollup documents are stored
|
|
||||||
in `UTC`.
|
|
||||||
|
|
||||||
|
`time_zone`::::
|
||||||
|
(Optional, string) Defines what time_zone the rollup documents are stored as.
|
||||||
|
Unlike raw data, which can shift timezones on the fly, rolled documents have to
|
||||||
|
be stored with a specific timezone. By default, rollup documents are stored
|
||||||
|
in `UTC`.
|
||||||
|
=====
|
||||||
|
//End date_histogram
|
||||||
|
|
||||||
|
//Begin histogram
|
||||||
|
`histogram`:::
|
||||||
|
(Optional, object) The histogram group aggregates one or more numeric fields
|
||||||
|
into numeric histogram intervals.
|
||||||
|
+
|
||||||
|
.Properties of `histogram`
|
||||||
|
[%collapsible%open]
|
||||||
|
=====
|
||||||
|
`fields`::::
|
||||||
|
(Required, array) The set of fields that you wish to build histograms for. All
|
||||||
|
fields specified must be some kind of numeric. Order does not matter.
|
||||||
|
|
||||||
|
`interval`::::
|
||||||
|
(Required, integer) The interval of histogram buckets to be generated when
|
||||||
|
rolling up. For example, a value of `5` creates buckets that are five units wide
|
||||||
|
(`0-5`, `5-10`, etc). Note that only one interval can be specified in the
|
||||||
|
`histogram` group, meaning that all fields being grouped via the histogram
|
||||||
|
must share the same interval.
|
||||||
|
=====
|
||||||
|
//End histogram
|
||||||
|
|
||||||
|
//Begin terms
|
||||||
`terms`:::
|
`terms`:::
|
||||||
(Optional, object) The terms group can be used on `keyword` or numeric fields
|
(Optional, object) The terms group can be used on `keyword` or numeric fields to
|
||||||
to allow bucketing via the `terms` aggregation at a later point. The indexer
|
allow bucketing via the `terms` aggregation at a later point. The indexer
|
||||||
enumerates and stores _all_ values of a field for each time-period. This can
|
enumerates and stores _all_ values of a field for each time-period. This can be
|
||||||
be potentially costly for high-cardinality groups such as IP addresses,
|
potentially costly for high-cardinality groups such as IP addresses, especially
|
||||||
especially if the time-bucket is particularly sparse.
|
if the time-bucket is particularly sparse.
|
||||||
+
|
+
|
||||||
--
|
--
|
||||||
TIP: While it is unlikely that a rollup will ever be larger in size than the raw
|
TIP: While it is unlikely that a rollup will ever be larger in size than the raw
|
||||||
@ -137,37 +169,25 @@ data, defining `terms` groups on multiple high-cardinality fields can
|
|||||||
effectively reduce the compression of a rollup to a large extent. You should be
|
effectively reduce the compression of a rollup to a large extent. You should be
|
||||||
judicious which high-cardinality fields are included for that reason.
|
judicious which high-cardinality fields are included for that reason.
|
||||||
|
|
||||||
The `terms` group has a single parameter:
|
|
||||||
--
|
--
|
||||||
|
|
||||||
`fields`::::
|
|
||||||
(Required, string) The set of fields that you wish to collect terms for. This
|
|
||||||
array can contain fields that are both `keyword` and numerics. Order does not
|
|
||||||
matter.
|
|
||||||
|
|
||||||
`histogram`:::
|
|
||||||
(Optional, object) The histogram group aggregates one or more numeric fields
|
|
||||||
into numeric histogram intervals.
|
|
||||||
+
|
+
|
||||||
--
|
.Properties of `terms`
|
||||||
The `histogram` group has a two parameters:
|
[%collapsible%open]
|
||||||
--
|
=====
|
||||||
|
|
||||||
`fields`::::
|
`fields`::::
|
||||||
(Required, array) The set of fields that you wish to build histograms for. All fields
|
(Required, string) The set of fields that you wish to collect terms for. This
|
||||||
specified must be some kind of numeric. Order does not matter.
|
array can contain fields that are both `keyword` and numerics. Order does not
|
||||||
|
matter.
|
||||||
`interval`::::
|
=====
|
||||||
(Required, integer) The interval of histogram buckets to be generated when
|
//End terms
|
||||||
rolling up. For example, a value of `5` creates buckets that are five units
|
====
|
||||||
wide (`0-5`, `5-10`, etc). Note that only one interval can be specified in the
|
//End groups
|
||||||
`histogram` group, meaning that all fields being grouped via the histogram
|
|
||||||
must share the same interval.
|
|
||||||
|
|
||||||
`index_pattern`::
|
`index_pattern`::
|
||||||
(Required, string) The index or index pattern to roll up. Supports
|
(Required, string) The index or index pattern to roll up. Supports
|
||||||
wildcard-style patterns (`logstash-*`). The job will
|
wildcard-style patterns (`logstash-*`). The job attempts to rollup the entire
|
||||||
attempt to rollup the entire index or index-pattern.
|
index or index-pattern.
|
||||||
+
|
+
|
||||||
--
|
--
|
||||||
NOTE: The `index_pattern` cannot be a pattern that would also match the
|
NOTE: The `index_pattern` cannot be a pattern that would also match the
|
||||||
@ -179,39 +199,43 @@ prevent this behavior.
|
|||||||
|
|
||||||
--
|
--
|
||||||
|
|
||||||
|
//Begin metrics
|
||||||
[[rollup-metrics-config]]
|
[[rollup-metrics-config]]
|
||||||
`metrics`::
|
`metrics`::
|
||||||
(Optional, object) Defines the metrics to collect for each grouping tuple.
|
(Optional, object) Defines the metrics to collect for each grouping tuple. By
|
||||||
By default, only the doc_counts are collected for each group. To make rollup
|
default, only the doc_counts are collected for each group. To make rollup useful,
|
||||||
useful, you will often add metrics like averages, mins, maxes, etc. Metrics
|
you will often add metrics like averages, mins, maxes, etc. Metrics are defined
|
||||||
are defined on a per-field basis and for each field you configure which metric
|
on a per-field basis and for each field you configure which metric should be
|
||||||
should be collected.
|
collected.
|
||||||
+
|
+
|
||||||
--
|
|
||||||
The `metrics` configuration accepts an array of objects, where each object has
|
The `metrics` configuration accepts an array of objects, where each object has
|
||||||
two parameters:
|
two parameters.
|
||||||
--
|
+
|
||||||
|
.Properties of metric objects
|
||||||
|
[%collapsible%open]
|
||||||
|
====
|
||||||
`field`:::
|
`field`:::
|
||||||
(Required, string) The field to collect metrics for. This must be a numeric
|
(Required, string) The field to collect metrics for. This must be a numeric of
|
||||||
of some kind.
|
some kind.
|
||||||
|
|
||||||
`metrics`:::
|
`metrics`:::
|
||||||
(Required, array) An array of metrics to collect for the field. At least one
|
(Required, array) An array of metrics to collect for the field. At least one
|
||||||
metric must be configured. Acceptable metrics are `min`,`max`,`sum`,`avg`, and
|
metric must be configured. Acceptable metrics are `min`,`max`,`sum`,`avg`, and
|
||||||
`value_count`.
|
`value_count`.
|
||||||
|
====
|
||||||
|
//End metrics
|
||||||
|
|
||||||
`page_size`::
|
`page_size`::
|
||||||
(Required, integer) The number of bucket results that are processed on each
|
(Required, integer) The number of bucket results that are processed on each
|
||||||
iteration of the rollup indexer. A larger value tends to execute faster, but
|
iteration of the rollup indexer. A larger value tends to execute faster, but
|
||||||
requires more memory during processing. This value has no effect on how the
|
requires more memory during processing. This value has no effect on how the data
|
||||||
data is rolled up; it is merely used for tweaking the speed or memory cost of
|
is rolled up; it is merely used for tweaking the speed or memory cost of
|
||||||
the indexer.
|
the indexer.
|
||||||
|
|
||||||
`rollup_index`::
|
`rollup_index`::
|
||||||
(Required, string) The index that contains the rollup results. The index can
|
(Required, string) The index that contains the rollup results. The index can
|
||||||
be shared with other {rollup-jobs}. The data is stored so that it doesn't
|
be shared with other {rollup-jobs}. The data is stored so that it doesn't
|
||||||
interfere with unrelated jobs.
|
interfere with unrelated jobs.
|
||||||
|
|
||||||
[[rollup-put-job-api-example]]
|
[[rollup-put-job-api-example]]
|
||||||
==== {api-example-title}
|
==== {api-example-title}
|
||||||
|
Loading…
x
Reference in New Issue
Block a user