e81804cfa4
Today if we search across a large amount of shards we hit every shard. Yet, it's quite common to search across an index pattern for time based indices but filtering will exclude all results outside a certain time range ie. `now-3d`. While the search can potentially hit hundreds of shards the majority of the shards might yield 0 results since there is not document that is within this date range. Kibana for instance does this regularly but used `_field_stats` to optimize the indexes they need to query. Now with the deprecation of `_field_stats` and it's upcoming removal a single dashboard in kibana can potentially turn into searches hitting hundreds or thousands of shards and that can easily cause search rejections even though the most of the requests are very likely super cheap and only need a query rewriting to early terminate with 0 results. This change adds a pre-filter phase for searches that can, if the number of shards are higher than a the `pre_filter_shard_size` threshold (defaults to 128 shards), fan out to the shards and check if the query can potentially match any documents at all. While false positives are possible, a negative response means that no matches are possible. These requests are not subject to rejection and can greatly reduce the number of shards a request needs to hit. The approach here is preferable to the kibana approach with field stats since it correctly handles aliases and uses the correct threadpools to execute these requests. Further it's completely transparent to the user and improves scalability of elasticsearch in general on large clusters. |
||
|---|---|---|
| .. | ||
| src/main/resources/rest-api-spec | ||
| .gitignore | ||
| README.markdown | ||
| build.gradle | ||
README.markdown
Elasticsearch REST API JSON specification
This repository contains a collection of JSON files which describe the Elasticsearch HTTP API.
Their purpose is to formalize and standardize the API, to facilitate development of libraries and integrations.
Example for the "Create Index" API:
{
"indices.create": {
"documentation": "http://www.elastic.co/guide/en/elasticsearch/reference/master/indices-create-index.html",
"methods": ["PUT", "POST"],
"url": {
"path": "/{index}",
"paths": ["/{index}"],
"parts": {
"index": {
"type" : "string",
"required" : true,
"description" : "The name of the index"
}
},
"params": {
"timeout": {
"type" : "time",
"description" : "Explicit operation timeout"
}
}
},
"body": {
"description" : "The configuration for the index (`settings` and `mappings`)"
}
}
}
The specification contains:
- The name of the API (
indices.create), which usually corresponds to the client calls - Link to the documentation at http://elastic.co
- List of HTTP methods for the endpoint
- URL specification: path, parts, parameters
- Whether body is allowed for the endpoint or not and its description
The methods and url.paths elements list all possible HTTP methods and URLs for the endpoint;
it is the responsibility of the developer to use this information for a sensible API on the target platform.
License
This software is licensed under the Apache License, version 2 ("ALv2").