[role="xpack"] [[rollup-search]] === Rollup Search ++++ Rollup Search ++++ The Rollup Search endpoint allows searching rolled-up data using the standard query DSL. The Rollup Search endpoint is needed because, internally, rolled-up documents utilize a different document structure than the original data. The Rollup Search endpoint rewrites standard query DSL into a format that matches the rollup documents, then takes the response and rewrites it back to what a client would expect given the original query. ==== Request `GET {index}/_rollup_search` //===== Description ==== Path Parameters `index`:: (string) Index, indices or index-pattern to execute a rollup search against. This can include both rollup and non-rollup indices ==== Request Body The request body supports a subset of features from the regular Search API. It supports: - `query` param for specifying an DSL query, subject to some limitations - `aggregations` param for specifying aggregations Functionality that is not available: - `size`: because rollups work on pre-aggregated data, no search hits can be returned and so size must be set to zero or omitted entirely. - `highlighter`, `suggestors`, `post_filter`, `profile`, `explain` are similarly disallowed ==== Historical-only search example Imagine we have an index named `sensor-1` full of raw data, and we have created a rollup job with the following configuration: [source,js] -------------------------------------------------- PUT _xpack/rollup/job/sensor { "index_pattern": "sensor-*", "rollup_index": "sensor_rollup", "cron": "*/30 * * * * ?", "size" :1000, "groups" : { "date_histogram": { "field": "timestamp", "interval": "1h", "delay": "7d" }, "terms": { "fields": ["node"] } }, "metrics": [ { "field": "temperature", "metrics": ["min", "max", "sum"] }, { "field": "voltage", "metrics": ["avg"] } ] } -------------------------------------------------- // CONSOLE // NOTEST This rolls up the `sensor-*` pattern and stores the results in `sensor_rollup`. To search this rolled up data, we need to use the `_rollup_search` endpoint. However, you'll notice that we can use regular query DSL to search the rolled-up data: [source,js] -------------------------------------------------- GET /sensor_rollup/_rollup_search { "size": 0, "aggregations": { "max_temperature": { "max": { "field": "temperature" } } } } -------------------------------------------------- // CONSOLE // TEST[setup:sensor_prefab_data] The query is targeting the `sensor_rollup` data, since this contains the rollup data as configured in the job. A `max` aggregation has been used on the `temperature` field, yielding the following response: [source,js] ---- { "took" : 102, "timed_out" : false, "_shards" : ... , "hits" : { "total" : 0, "max_score" : 0.0, "hits" : [ ] }, "aggregations" : { "max_temperature" : { "value" : 202.0 } } } ---- // TESTRESPONSE[s/"took" : 102/"took" : $body.$_path/] // TESTRESPONSE[s/"_shards" : \.\.\. /"_shards" : $body.$_path/] The response is exactly as you'd expect from a regular query + aggregation; it provides some metadata about the request (`took`, `_shards`, etc), the search hits (which is always empty for rollup searches), and the aggregation response. Rollup searches are limited to functionality that was configured in the rollup job. For example, we are not able to calculate the average temperature because `avg` was not one of the configured metrics for the `temperature` field. If we try to execute that search: [source,js] -------------------------------------------------- GET sensor_rollup/_rollup_search { "size": 0, "aggregations": { "avg_temperature": { "avg": { "field": "temperature" } } } } -------------------------------------------------- // CONSOLE // TEST[continued] // TEST[catch:bad_request] [source,js] ---- { "error" : { "root_cause" : [ { "type" : "illegal_argument_exception", "reason" : "There is not a [avg] agg with name [temperature] configured in selected rollup indices, cannot translate aggregation.", "stack_trace": ... } ], "type" : "illegal_argument_exception", "reason" : "There is not a [avg] agg with name [temperature] configured in selected rollup indices, cannot translate aggregation.", "stack_trace": ... }, "status": 400 } ---- // TESTRESPONSE[s/"stack_trace": \.\.\./"stack_trace": $body.$_path/] ==== Searching both historical rollup and non-rollup data The Rollup Search API has the capability to search across both "live", non-rollup data as well as the aggregated rollup data. This is done by simply adding the live indices to the URI: [source,js] -------------------------------------------------- GET sensor-1,sensor_rollup/_rollup_search <1> { "size": 0, "aggregations": { "max_temperature": { "max": { "field": "temperature" } } } } -------------------------------------------------- // CONSOLE // TEST[continued] <1> Note the URI now searches `sensor-1` and `sensor_rollup` at the same time When the search is executed, the Rollup Search endpoint will do two things: 1. The original request will be sent to the non-rollup index unaltered 2. A rewritten version of the original request will be sent to the rollup index. When the two responses are received, the endpoint will then rewrite the rollup response and merge the two together. During the merging process, if there is any overlap in buckets between the two responses, the buckets from the non-rollup index will be used. The response to the above query will look as expected, despite spanning rollup and non-rollup indices: [source,js] ---- { "took" : 102, "timed_out" : false, "_shards" : ... , "hits" : { "total" : 0, "max_score" : 0.0, "hits" : [ ] }, "aggregations" : { "max_temperature" : { "value" : 202.0 } } } ---- // TESTRESPONSE[s/"took" : 102/"took" : $body.$_path/] // TESTRESPONSE[s/"_shards" : \.\.\. /"_shards" : $body.$_path/]