mirror of
https://github.com/honeymoose/OpenSearch.git
synced 2025-02-17 10:25:15 +00:00
Today we require users to prepare their indices for split operations. Yet, we can do this automatically when an index is created which would make the split feature a much more appealing option since it doesn't have any 3rd party prerequisites anymore. This change automatically sets the number of routinng shards such that an index is guaranteed to be able to split once into twice as many shards. The number of routing shards is scaled towards the default shard limit per index such that indices with a smaller amount of shards can be split more often than larger ones. For instance an index with 1 or 2 shards can be split 10x (until it approaches 1024 shards) while an index created with 128 shards can only be split 3x by a factor of 2. Please note this is just a default value and users can still prepare their indices with `index.number_of_routing_shards` for custom splitting. NOTE: this change has an impact on the document distribution since we are changing the hash space. Documents are still uniformly distributed across all shards but since we are artificually changing the number of buckets in the consistent hashign space document might be hashed into different shards compared to previous versions. This is a 7.0 only change.
260 lines
6.6 KiB
Plaintext
260 lines
6.6 KiB
Plaintext
[[search-validate]]
|
|
== Validate API
|
|
|
|
The validate API allows a user to validate a potentially expensive query
|
|
without executing it. We'll use the following test data to explain _validate:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
PUT twitter/tweet/_bulk?refresh
|
|
{"index":{"_id":1}}
|
|
{"user" : "kimchy", "post_date" : "2009-11-15T14:12:12", "message" : "trying out Elasticsearch"}
|
|
{"index":{"_id":2}}
|
|
{"user" : "kimchi", "post_date" : "2009-11-15T14:12:13", "message" : "My username is similar to @kimchy!"}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
// TESTSETUP
|
|
|
|
When sent a valid query:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
GET twitter/_validate/query?q=user:foo
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
|
|
The response contains `valid:true`:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
{"valid":true,"_shards":{"total":1,"successful":1,"failed":0}}
|
|
--------------------------------------------------
|
|
// TESTRESPONSE
|
|
|
|
[float]
|
|
=== Request Parameters
|
|
|
|
When executing exists using the query parameter `q`, the query passed is
|
|
a query string using Lucene query parser. There are additional
|
|
parameters that can be passed:
|
|
|
|
[cols="<,<",options="header",]
|
|
|=======================================================================
|
|
|Name |Description
|
|
|`df` |The default field to use when no field prefix is defined within the
|
|
query.
|
|
|
|
|`analyzer` |The analyzer name to be used when analyzing the query string.
|
|
|
|
|`default_operator` |The default operator to be used, can be `AND` or
|
|
`OR`. Defaults to `OR`.
|
|
|
|
|`lenient` |If set to true will cause format based failures (like
|
|
providing text to a numeric field) to be ignored. Defaults to false.
|
|
|
|
|`analyze_wildcard` |Should wildcard and prefix queries be analyzed or
|
|
not. Defaults to `false`.
|
|
|=======================================================================
|
|
|
|
The query may also be sent in the request body:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
GET twitter/tweet/_validate/query
|
|
{
|
|
"query" : {
|
|
"bool" : {
|
|
"must" : {
|
|
"query_string" : {
|
|
"query" : "*:*"
|
|
}
|
|
},
|
|
"filter" : {
|
|
"term" : { "user" : "kimchy" }
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
|
|
NOTE: The query being sent in the body must be nested in a `query` key, same as
|
|
the <<search-search,search api>> works
|
|
|
|
If the query is invalid, `valid` will be `false`. Here the query is
|
|
invalid because Elasticsearch knows the post_date field should be a date
|
|
due to dynamic mapping, and 'foo' does not correctly parse into a date:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
GET twitter/tweet/_validate/query
|
|
{
|
|
"query": {
|
|
"query_string": {
|
|
"query": "post_date:foo",
|
|
"lenient": false
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
{"valid":false,"_shards":{"total":1,"successful":1,"failed":0}}
|
|
--------------------------------------------------
|
|
// TESTRESPONSE
|
|
|
|
An `explain` parameter can be specified to get more detailed information
|
|
about why a query failed:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
GET twitter/tweet/_validate/query?explain=true
|
|
{
|
|
"query": {
|
|
"query_string": {
|
|
"query": "post_date:foo",
|
|
"lenient": false
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
|
|
responds with:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
{
|
|
"valid" : false,
|
|
"_shards" : {
|
|
"total" : 1,
|
|
"successful" : 1,
|
|
"failed" : 0
|
|
},
|
|
"explanations" : [ {
|
|
"index" : "twitter",
|
|
"valid" : false,
|
|
"error" : "twitter/IAEc2nIXSSunQA_suI0MLw] QueryShardException[failed to create query:...failed to parse date field [foo]"
|
|
} ]
|
|
}
|
|
--------------------------------------------------
|
|
// TESTRESPONSE[s/"error" : "[^\"]+"/"error": "$body.explanations.0.error"/]
|
|
|
|
When the query is valid, the explanation defaults to the string
|
|
representation of that query. With `rewrite` set to `true`, the explanation
|
|
is more detailed showing the actual Lucene query that will be executed.
|
|
|
|
For More Like This:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
GET twitter/tweet/_validate/query?rewrite=true
|
|
{
|
|
"query": {
|
|
"more_like_this": {
|
|
"like": {
|
|
"_id": "2"
|
|
},
|
|
"boost_terms": 1
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
// TEST[skip:the output is randomized depending on which shard we hit]
|
|
|
|
Response:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
{
|
|
"valid": true,
|
|
"_shards": {
|
|
"total": 1,
|
|
"successful": 1,
|
|
"failed": 0
|
|
},
|
|
"explanations": [
|
|
{
|
|
"index": "twitter",
|
|
"valid": true,
|
|
"explanation": "((user:terminator^3.71334 plot:future^2.763601 plot:human^2.8415773 plot:sarah^3.4193945 plot:kyle^3.8244398 plot:cyborg^3.9177752 plot:connor^4.040236 plot:reese^4.7133346 ... )~6) -ConstantScore(_uid:tweet#2)) #(ConstantScore(_type:tweet))^0.0"
|
|
}
|
|
]
|
|
}
|
|
--------------------------------------------------
|
|
// TESTRESPONSE
|
|
|
|
By default, the request is executed on a single shard only, which is randomly
|
|
selected. The detailed explanation of the query may depend on which shard is
|
|
being hit, and therefore may vary from one request to another. So, in case of
|
|
query rewrite the `all_shards` parameter should be used to get response from
|
|
all available shards.
|
|
|
|
For Fuzzy Queries:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
GET twitter/tweet/_validate/query?rewrite=true&all_shards=true
|
|
{
|
|
"query": {
|
|
"match": {
|
|
"user": {
|
|
"query": "kimchy",
|
|
"fuzziness": "auto"
|
|
}
|
|
}
|
|
}
|
|
}
|
|
--------------------------------------------------
|
|
// CONSOLE
|
|
|
|
Response:
|
|
|
|
[source,js]
|
|
--------------------------------------------------
|
|
{
|
|
"valid": true,
|
|
"_shards": {
|
|
"total": 5,
|
|
"successful": 5,
|
|
"failed": 0
|
|
},
|
|
"explanations": [
|
|
{
|
|
"index": "twitter",
|
|
"shard": 0,
|
|
"valid": true,
|
|
"explanation": "user:kimchy~2"
|
|
},
|
|
{
|
|
"index": "twitter",
|
|
"shard": 1,
|
|
"valid": true,
|
|
"explanation": "user:kimchy~2"
|
|
},
|
|
{
|
|
"index": "twitter",
|
|
"shard": 2,
|
|
"valid": true,
|
|
"explanation": "user:kimchy~2"
|
|
},
|
|
{
|
|
"index": "twitter",
|
|
"shard": 3,
|
|
"valid": true,
|
|
"explanation": "(user:kimchi)^0.8333333"
|
|
},
|
|
{
|
|
"index": "twitter",
|
|
"shard": 4,
|
|
"valid": true,
|
|
"explanation": "user:kimchy"
|
|
}
|
|
]
|
|
}
|
|
--------------------------------------------------
|
|
// TESTRESPONSE
|