115 lines
5.1 KiB
Plaintext
115 lines
5.1 KiB
Plaintext
[[rollup-search-limitations]]
|
|
== Rollup Search Limitations
|
|
|
|
While we feel the Rollup function is extremely flexible, the nature of summarizing data means there will be some limitations. Once
|
|
live data is thrown away, you will always lose some flexibility.
|
|
|
|
This page highlights the major limitations so that you are aware of them.
|
|
|
|
[float]
|
|
=== Only one Rollup index per search
|
|
|
|
When using the <<rollup-search>> endpoint, the `index` parameter accepts one or more indices. These can be a mix of regular, non-rollup
|
|
indices and rollup indices. However, only one rollup index can be specified. The exact list of rules for the `index` parameter are as
|
|
follows:
|
|
|
|
- At least one index/index-pattern must be specified. This can be either a rollup or non-rollup index. Omitting the index parameter,
|
|
or using `_all`, is not permitted
|
|
- Multiple non-rollup indices may be specified
|
|
- Only one rollup index may be specified. If more than one are supplied an exception will be thrown
|
|
|
|
This limitation is driven by the logic that decides which jobs are the "best" for any given query. If you have ten jobs stored in a single
|
|
index, which cover the source data with varying degrees of completeness and different intervals, the query needs to determine which set
|
|
of jobs to actually search. Incorrect decisions can lead to inaccurate aggregation results (e.g. over-counting doc counts, or bad metrics).
|
|
Needless to say, this is a technically challenging piece of code.
|
|
|
|
To help simplify the problem, we have limited search to just one rollup index at a time (which may contain multiple jobs). In the future we
|
|
may be able to open this up to multiple rollup jobs.
|
|
|
|
[float]
|
|
=== Can only aggregate what's been stored
|
|
|
|
A perhaps obvious limitation, but rollups can only aggregate on data that has been stored in the rollups. If you don't configure the
|
|
rollup job to store metrics about the `price` field, you won't be able to use the `price` field in any query or aggregation.
|
|
|
|
For example, the `temperature` field in the following query has been stored in a rollup job... but not with an `avg` metric. Which means
|
|
the usage of `avg` here is not allowed:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
GET sensor_rollup/_rollup_search
|
|
{
|
|
"size": 0,
|
|
"aggregations": {
|
|
"avg_temperature": {
|
|
"avg": {
|
|
"field": "temperature"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
// TEST[setup:sensor_prefab_data]
|
|
// TEST[catch:/illegal_argument_exception/]
|
|
|
|
The response will tell you that the field and aggregation were not possible, because no rollup jobs were found which contained them:
|
|
|
|
[source,js]
|
|
----
|
|
{
|
|
"error" : {
|
|
"root_cause" : [
|
|
{
|
|
"type" : "illegal_argument_exception",
|
|
"reason" : "There is not a rollup job that has a [avg] agg with name [avg_temperature] which also satisfies all requirements of query.",
|
|
"stack_trace": ...
|
|
}
|
|
],
|
|
"type" : "illegal_argument_exception",
|
|
"reason" : "There is not a rollup job that has a [avg] agg with name [avg_temperature] which also satisfies all requirements of query.",
|
|
"stack_trace": ...
|
|
},
|
|
"status": 400
|
|
}
|
|
----
|
|
// TESTRESPONSE[s/"stack_trace": \.\.\./"stack_trace": $body.$_path/]
|
|
|
|
[float]
|
|
=== Interval Granularity
|
|
|
|
Rollups are stored at a certain granularity, as defined by the `date_histogram` group in the configuration. If data is rolled up at hourly
|
|
intervals, the <<rollup-search>> API can aggregate on any time interval hourly or greater. Intervals that are less than an hour will throw
|
|
an exception, since the data simply doesn't exist for finer granularities.
|
|
|
|
Because the RollupSearch endpoint can "upsample" intervals, there is no need to configure jobs with multiple intervals (hourly, daily, etc).
|
|
It's recommended to just configure a single job with the smallest granularity that is needed, and allow the search endpoint to upsample
|
|
as needed.
|
|
|
|
That said, if multiple jobs are present in a single rollup index with varying intervals, the search endpoint will identify and use the job(s)
|
|
with the largest interval to satisfy the search reques.
|
|
|
|
[float]
|
|
=== Limited querying components
|
|
|
|
The Rollup functionality allows `query`'s in the search request, but with a limited subset of components. The queries currently allowed are:
|
|
|
|
- Term Query
|
|
- Terms Query
|
|
- Range Query
|
|
- MatchAll Query
|
|
- Any compound query (Boolean, Boosting, ConstantScore, etc)
|
|
|
|
Furthermore, these queries can only use fields that were also saved in the rollup job. If you wish to filter on a keyword `hostname` field,
|
|
that field must have been configured in the rollup job under a `terms` grouping.
|
|
|
|
If you attempt to use an unsupported query, or the query references a field that wasn't configured in the rollup job, an exception will be
|
|
thrown. We expect the list of support queries to grow over time as more are implemented.
|
|
|
|
[float]
|
|
=== Timezones
|
|
|
|
Rollup documents are stored in the timezone of the `date_histogram` group configuration in the job. If no timezone is specified, the default
|
|
is to rollup timestamps in `UTC`.
|
|
|