mirror of
https://github.com/honeymoose/OpenSearch.git
synced 2025-02-13 00:15:47 +00:00
e81804cfa4
Today if we search across a large amount of shards we hit every shard. Yet, it's quite common to search across an index pattern for time based indices but filtering will exclude all results outside a certain time range ie. `now-3d`. While the search can potentially hit hundreds of shards the majority of the shards might yield 0 results since there is not document that is within this date range. Kibana for instance does this regularly but used `_field_stats` to optimize the indexes they need to query. Now with the deprecation of `_field_stats` and it's upcoming removal a single dashboard in kibana can potentially turn into searches hitting hundreds or thousands of shards and that can easily cause search rejections even though the most of the requests are very likely super cheap and only need a query rewriting to early terminate with 0 results. This change adds a pre-filter phase for searches that can, if the number of shards are higher than a the `pre_filter_shard_size` threshold (defaults to 128 shards), fan out to the shards and check if the query can potentially match any documents at all. While false positives are possible, a negative response means that no matches are possible. These requests are not subject to rejection and can greatly reduce the number of shards a request needs to hit. The approach here is preferable to the kibana approach with field stats since it correctly handles aliases and uses the correct threadpools to execute these requests. Further it's completely transparent to the user and improves scalability of elasticsearch in general on large clusters.
110 lines
3.2 KiB
Plaintext
110 lines
3.2 KiB
Plaintext
[[search-count]]
|
|
== Count API
|
|
|
|
The count API allows to easily execute a query and get the number of
|
|
matches for that query. It can be executed across one or more indices
|
|
and across one or more types. The query can either be provided using a
|
|
simple query string as a parameter, or using the
|
|
<<query-dsl,Query DSL>> defined within the request
|
|
body. Here is an example:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
PUT /twitter/tweet/1?refresh
|
|
{
|
|
"user": "kimchy"
|
|
}
|
|
|
|
GET /twitter/tweet/_count?q=user:kimchy
|
|
|
|
GET /twitter/tweet/_count
|
|
{
|
|
"query" : {
|
|
"term" : { "user" : "kimchy" }
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
//CONSOLE
|
|
|
|
NOTE: The query being sent in the body must be nested in a `query` key, same as
|
|
the <<search-search,search api>> works
|
|
|
|
Both examples above do the same thing, which is count the number of
|
|
tweets from the twitter index for a certain user. The result is:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
{
|
|
"count" : 1,
|
|
"_shards" : {
|
|
"total" : 5,
|
|
"successful" : 5,
|
|
"skipped" : 0,
|
|
"failed" : 0
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// TESTRESPONSE
|
|
|
|
The query is optional, and when not provided, it will use `match_all` to
|
|
count all the docs.
|
|
|
|
[float]
|
|
=== Multi index, Multi type
|
|
|
|
The count API can be applied to <<search-multi-index-type,multiple types in multiple indices>>.
|
|
|
|
[float]
|
|
=== Request Parameters
|
|
|
|
When executing count using the query parameter `q`, the query passed is
|
|
a query string using Lucene query parser. There are additional
|
|
parameters that can be passed:
|
|
|
|
[cols="<,<",options="header",]
|
|
|=======================================================================
|
|
|Name |Description
|
|
|`df` |The default field to use when no field prefix is defined within the
|
|
query.
|
|
|
|
|`analyzer` |The analyzer name to be used when analyzing the query string.
|
|
|
|
|`default_operator` |The default operator to be used, can be `AND` or
|
|
`OR`. Defaults to `OR`.
|
|
|
|
|`lenient` |If set to true will cause format based failures (like
|
|
providing text to a numeric field) to be ignored. Defaults to false.
|
|
|
|
|`analyze_wildcard` |Should wildcard and prefix queries be analyzed or
|
|
not. Defaults to `false`.
|
|
|
|
|`terminate_after` |The maximum count for each shard, upon
|
|
reaching which the query execution will terminate early.
|
|
If set, the response will have a boolean field `terminated_early` to
|
|
indicate whether the query execution has actually terminated_early.
|
|
Defaults to no terminate_after.
|
|
|=======================================================================
|
|
|
|
[float]
|
|
=== Request Body
|
|
|
|
The count can use the <<query-dsl,Query DSL>> within
|
|
its body in order to express the query that should be executed. The body
|
|
content can also be passed as a REST parameter named `source`.
|
|
|
|
Both HTTP GET and HTTP POST can be used to execute count with body.
|
|
Since not all clients support GET with body, POST is allowed as well.
|
|
|
|
[float]
|
|
=== Distributed
|
|
|
|
The count operation is broadcast across all shards. For each shard id
|
|
group, a replica is chosen and executed against it. This means that
|
|
replicas increase the scalability of count.
|
|
|
|
[float]
|
|
=== Routing
|
|
|
|
The routing value (a comma separated list of the routing values) can be
|
|
specified to control which shards the count request will be executed on.
|