e81804cfa4
Today if we search across a large amount of shards we hit every shard. Yet, it's quite common to search across an index pattern for time based indices but filtering will exclude all results outside a certain time range ie. `now-3d`. While the search can potentially hit hundreds of shards the majority of the shards might yield 0 results since there is not document that is within this date range. Kibana for instance does this regularly but used `_field_stats` to optimize the indexes they need to query. Now with the deprecation of `_field_stats` and it's upcoming removal a single dashboard in kibana can potentially turn into searches hitting hundreds or thousands of shards and that can easily cause search rejections even though the most of the requests are very likely super cheap and only need a query rewriting to early terminate with 0 results. This change adds a pre-filter phase for searches that can, if the number of shards are higher than a the `pre_filter_shard_size` threshold (defaults to 128 shards), fan out to the shards and check if the query can potentially match any documents at all. While false positives are possible, a negative response means that no matches are possible. These requests are not subject to rejection and can greatly reduce the number of shards a request needs to hit. The approach here is preferable to the kibana approach with field stats since it correctly handles aliases and uses the correct threadpools to execute these requests. Further it's completely transparent to the user and improves scalability of elasticsearch in general on large clusters. |
||
|---|---|---|
| .. | ||
| src/main/java/org/elasticsearch/plugin/noop | ||
| README.md | ||
| build.gradle | ||
README.md
Purpose
This plugin provides empty REST and transport endpoints for bulk indexing and search. It is used to avoid accidental server-side bottlenecks in client-side benchmarking.
Build Instructions
Build the plugin with gradle :client:client-benchmark-noop-api-plugin:assemble from the Elasticsearch root project directory.
Installation Instructions
After, the binary has been built, install it with bin/elasticsearch-plugin install file:///full/path/to/noop-plugin.zip.
Usage
The plugin provides two REST endpoints:
/_noop_bulkand all variations that the bulk endpoint provides (except that all no op endpoints are called_noop_bulkinstead of_bulk)_noop_searchand all variations that the search endpoint provides (except that all no op endpoints are called_noop_searchinstead of_search)
The corresponding transport actions are:
org.elasticsearch.plugin.noop.action.bulk.TransportNoopBulkActionorg.elasticsearch.plugin.noop.action.search.TransportNoopSearchAction