OpenSearch/docs/reference/aggregations/pipeline.asciidoc

[[search-aggregations-pipeline]]

== Pipeline Aggregations

coming[2.0.0]

experimental[]

Pipeline aggregations work on the outputs produced from other aggregations rather than from document sets, adding
information to the output tree. There are many different types of pipeline aggregation, each computing different information from
other aggregations, but these types can broken down into two families:

_Parent_::
                A family of pipeline aggregations that is provided with the output of its parent aggregation and is able
                to compute new buckets or new aggregations to add to existing buckets.

_Sibling_::
                Pipeline aggregations that are provided with the output of a sibling aggregation and are able to compute a
                new aggregation which will be at the same level as the sibling aggregation.

Pipeline aggregations can reference the aggregations they need to perform their computation by using the `buckets_paths`
parameter to indicate the paths to the required metrics. The syntax for defining these paths can be found in the
<<bucket-path-syntax, `buckets_path` Syntax>> section below.

Pipeline aggregations cannot have sub-aggregations but depending on the type it can reference another pipeline in the `buckets_path`
allowing pipeline aggregations to be chained.  For example, you can chain together two derivatives to calculate the second derivative
(e.g. a derivative of a derivative).

NOTE: Because pipeline aggregations only add to the output, when chaining pipeline aggregations the output of each pipeline aggregation 
will be included in the final output.

[[bucket-path-syntax]]
[float]
=== `buckets_path` Syntax

Most pipeline aggregations require another aggregation as their input.  The input aggregation is defined via the `buckets_path`
parameter, which follows a specific format:

--------------------------------------------------
AGG_SEPARATOR       :=  '>'
METRIC_SEPARATOR    :=  '.'
AGG_NAME            :=  <the name of the aggregation>
METRIC              :=  <the name of the metric (in case of multi-value metrics aggregation)>
PATH                :=  <AGG_NAME>[<AGG_SEPARATOR><AGG_NAME>]*[<METRIC_SEPARATOR><METRIC>]
--------------------------------------------------

For example, the path `"my_bucket>my_stats.avg"` will path to the `avg` value in the `"my_stats"` metric, which is
contained in the `"my_bucket"` bucket aggregation.

Paths are relative from the position of the pipeline aggregation; they are not absolute paths, and the path cannot go back "up" the
aggregation tree. For example, this moving average is embedded inside a date_histogram and refers to a "sibling"
metric `"the_sum"`:

[source,js]
--------------------------------------------------
{
    "my_date_histo":{
        "date_histogram":{
            "field":"timestamp",
            "interval":"day"
        },
        "aggs":{
            "the_sum":{
                "sum":{ "field": "lemmings" } <1>
            },
            "the_movavg":{
                "moving_avg":{ "buckets_path": "the_sum" } <2>
            }
        }
    }
}
--------------------------------------------------
<1> The metric is called `"the_sum"`
<2> The `buckets_path` refers to the metric via a relative path `"the_sum"`

`buckets_path` is also used for Sibling pipeline aggregations, where the aggregation is "next" to a series of buckets
instead of embedded "inside" them.  For example, the `max_bucket` aggregation uses the `buckets_path` to specify
a metric embedded inside a sibling aggregation:

[source,js]
--------------------------------------------------
{
    "aggs" : {
        "sales_per_month" : {
            "date_histogram" : {
                "field" : "date",
                "interval" : "month"
            },
            "aggs": {
                "sales": {
                    "sum": {
                        "field": "price"
                    }
                }
            }
        },
        "max_monthly_sales": {
            "max_bucket": {
                "buckets_paths": "sales_per_month>sales" <1>
            }
        }
    }
}
--------------------------------------------------
<1> `bucket_paths` instructs this max_bucket aggregation that we want the maximum value of the `sales` aggregation in the
`sales_per_month` date histogram.

[float]
==== Special Paths

Instead of pathing to a metric, `buckets_path` can use a special `"_count"` path.  This instructs
the pipeline aggregation to use the document count as it's input.  For example, a moving average can be calculated on the document
count of each bucket, instead of a specific metric:

[source,js]
--------------------------------------------------
{
    "my_date_histo":{
        "date_histogram":{
            "field":"timestamp",
            "interval":"day"
        },
        "aggs":{
            "the_movavg":{
                "moving_avg":{ "buckets_path": "_count" } <1>
            }
        }
    }
}
--------------------------------------------------
<1> By using `_count` instead of a metric name, we can calculate the moving average of document counts in the histogram

[[gap-policy]]
[float]
=== Dealing with gaps in the data

There are a couple of reasons why the data output by the enclosing histogram may have gaps:

* There are no documents matching the query for some buckets
* The data for a metric is missing in all of the documents falling into a bucket (this is most likely with either a small interval
on the enclosing histogram or with a query matching only a small number of documents)

Where there is no data available in a bucket for a given metric it presents a problem for calculating the derivative value for both
the current bucket and the next bucket. In the derivative pipeline aggregation has a `gap policy` parameter to define what the behavior
should be when a gap in the data is found. There are currently two options for controlling the gap policy:

_skip_::
                This option will not produce a derivative value for any buckets where the value in the current or previous bucket is
                missing

_insert_zeros_::
                This option will assume the missing value is `0` and calculate the derivative with the value `0`.




include::pipeline/avg-bucket-aggregation.asciidoc[]
include::pipeline/derivative-aggregation.asciidoc[]
include::pipeline/max-bucket-aggregation.asciidoc[]
include::pipeline/min-bucket-aggregation.asciidoc[]
include::pipeline/sum-bucket-aggregation.asciidoc[]
include::pipeline/movavg-aggregation.asciidoc[]