164 lines
6.5 KiB
Plaintext
164 lines
6.5 KiB
Plaintext
[[search-aggregations-pipeline]]
|
|
|
|
== Pipeline Aggregations
|
|
|
|
coming[2.0.0]
|
|
|
|
experimental[]
|
|
|
|
Pipeline aggregations work on the outputs produced from other aggregations rather than from document sets, adding
|
|
information to the output tree. There are many different types of pipeline aggregation, each computing different information from
|
|
other aggregations, but these types can broken down into two families:
|
|
|
|
_Parent_::
|
|
A family of pipeline aggregations that is provided with the output of its parent aggregation and is able
|
|
to compute new buckets or new aggregations to add to existing buckets.
|
|
|
|
_Sibling_::
|
|
Pipeline aggregations that are provided with the output of a sibling aggregation and are able to compute a
|
|
new aggregation which will be at the same level as the sibling aggregation.
|
|
|
|
Pipeline aggregations can reference the aggregations they need to perform their computation by using the `buckets_paths`
|
|
parameter to indicate the paths to the required metrics. The syntax for defining these paths can be found in the
|
|
<<bucket-path-syntax, `buckets_path` Syntax>> section below.
|
|
|
|
Pipeline aggregations cannot have sub-aggregations but depending on the type it can reference another pipeline in the `buckets_path`
|
|
allowing pipeline aggregations to be chained. For example, you can chain together two derivatives to calculate the second derivative
|
|
(e.g. a derivative of a derivative).
|
|
|
|
NOTE: Because pipeline aggregations only add to the output, when chaining pipeline aggregations the output of each pipeline aggregation
|
|
will be included in the final output.
|
|
|
|
[[bucket-path-syntax]]
|
|
[float]
|
|
=== `buckets_path` Syntax
|
|
|
|
Most pipeline aggregations require another aggregation as their input. The input aggregation is defined via the `buckets_path`
|
|
parameter, which follows a specific format:
|
|
|
|
--------------------------------------------------
|
|
AGG_SEPARATOR := '>'
|
|
METRIC_SEPARATOR := '.'
|
|
AGG_NAME := <the name of the aggregation>
|
|
METRIC := <the name of the metric (in case of multi-value metrics aggregation)>
|
|
PATH := <AGG_NAME>[<AGG_SEPARATOR><AGG_NAME>]*[<METRIC_SEPARATOR><METRIC>]
|
|
--------------------------------------------------
|
|
|
|
For example, the path `"my_bucket>my_stats.avg"` will path to the `avg` value in the `"my_stats"` metric, which is
|
|
contained in the `"my_bucket"` bucket aggregation.
|
|
|
|
Paths are relative from the position of the pipeline aggregation; they are not absolute paths, and the path cannot go back "up" the
|
|
aggregation tree. For example, this moving average is embedded inside a date_histogram and refers to a "sibling"
|
|
metric `"the_sum"`:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
{
|
|
"my_date_histo":{
|
|
"date_histogram":{
|
|
"field":"timestamp",
|
|
"interval":"day"
|
|
},
|
|
"aggs":{
|
|
"the_sum":{
|
|
"sum":{ "field": "lemmings" } <1>
|
|
},
|
|
"the_movavg":{
|
|
"moving_avg":{ "buckets_path": "the_sum" } <2>
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
<1> The metric is called `"the_sum"`
|
|
<2> The `buckets_path` refers to the metric via a relative path `"the_sum"`
|
|
|
|
`buckets_path` is also used for Sibling pipeline aggregations, where the aggregation is "next" to a series of buckets
|
|
instead of embedded "inside" them. For example, the `max_bucket` aggregation uses the `buckets_path` to specify
|
|
a metric embedded inside a sibling aggregation:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
{
|
|
"aggs" : {
|
|
"sales_per_month" : {
|
|
"date_histogram" : {
|
|
"field" : "date",
|
|
"interval" : "month"
|
|
},
|
|
"aggs": {
|
|
"sales": {
|
|
"sum": {
|
|
"field": "price"
|
|
}
|
|
}
|
|
}
|
|
},
|
|
"max_monthly_sales": {
|
|
"max_bucket": {
|
|
"buckets_paths": "sales_per_month>sales" <1>
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
<1> `bucket_paths` instructs this max_bucket aggregation that we want the maximum value of the `sales` aggregation in the
|
|
`sales_per_month` date histogram.
|
|
|
|
[float]
|
|
==== Special Paths
|
|
|
|
Instead of pathing to a metric, `buckets_path` can use a special `"_count"` path. This instructs
|
|
the pipeline aggregation to use the document count as it's input. For example, a moving average can be calculated on the document
|
|
count of each bucket, instead of a specific metric:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
{
|
|
"my_date_histo":{
|
|
"date_histogram":{
|
|
"field":"timestamp",
|
|
"interval":"day"
|
|
},
|
|
"aggs":{
|
|
"the_movavg":{
|
|
"moving_avg":{ "buckets_path": "_count" } <1>
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
<1> By using `_count` instead of a metric name, we can calculate the moving average of document counts in the histogram
|
|
|
|
[[gap-policy]]
|
|
[float]
|
|
=== Dealing with gaps in the data
|
|
|
|
There are a couple of reasons why the data output by the enclosing histogram may have gaps:
|
|
|
|
* There are no documents matching the query for some buckets
|
|
* The data for a metric is missing in all of the documents falling into a bucket (this is most likely with either a small interval
|
|
on the enclosing histogram or with a query matching only a small number of documents)
|
|
|
|
Where there is no data available in a bucket for a given metric it presents a problem for calculating the derivative value for both
|
|
the current bucket and the next bucket. In the derivative pipeline aggregation has a `gap policy` parameter to define what the behavior
|
|
should be when a gap in the data is found. There are currently two options for controlling the gap policy:
|
|
|
|
_skip_::
|
|
This option will not produce a derivative value for any buckets where the value in the current or previous bucket is
|
|
missing
|
|
|
|
_insert_zeros_::
|
|
This option will assume the missing value is `0` and calculate the derivative with the value `0`.
|
|
|
|
|
|
|
|
|
|
include::pipeline/avg-bucket-aggregation.asciidoc[]
|
|
include::pipeline/derivative-aggregation.asciidoc[]
|
|
include::pipeline/max-bucket-aggregation.asciidoc[]
|
|
include::pipeline/min-bucket-aggregation.asciidoc[]
|
|
include::pipeline/sum-bucket-aggregation.asciidoc[]
|
|
include::pipeline/movavg-aggregation.asciidoc[]
|
|
include::pipeline/series-arithmetic-aggregation.asciidoc[]
|