OpenSearch/docs/reference/search/explain.asciidoc

190 lines
5.2 KiB
Plaintext
Raw Normal View History

[[search-explain]]
=== Explain API
Returns information about why a specific document matches (or doesn't match) a
query.
[source,console]
--------------------------------------------------
GET /my-index-000001/_explain/0
{
"query" : {
"match" : { "message" : "elasticsearch" }
}
}
--------------------------------------------------
// TEST[setup:messages]
[[sample-api-request]]
==== {api-request-title}
`GET /<index>/_explain/<id>`
`POST /<index>/_explain/<id>`
[[sample-api-desc]]
==== {api-description-title}
The explain API computes a score explanation for a query and a specific
document. This can give useful feedback whether a document matches or
didn't match a specific query.
[[sample-api-path-params]]
==== {api-path-parms-title}
`<id>`::
(Required, integer) Defines the document ID.
`<index>`::
+
--
(Required, string)
Index names used to limit the request.
Only a single index name can be provided to this parameter.
--
[[sample-api-query-params]]
==== {api-query-parms-title}
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=analyzer]
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=analyze_wildcard]
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=default_operator]
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=df]
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=lenient]
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=preference]
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=search-q]
`stored_fields`::
(Optional, string) A comma-separated list of stored fields to return in the
response.
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=index-routing]
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=source]
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=source_excludes]
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=source_includes]
[[sample-api-request-body]]
==== {api-request-body-title}
include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=query]
[[sample-api-example]]
==== {api-examples-title}
[source,console]
--------------------------------------------------
GET /my-index-000001/_explain/0
{
"query" : {
"match" : { "message" : "elasticsearch" }
}
}
--------------------------------------------------
// TEST[setup:messages]
The API returns the following response:
[source,console-result]
--------------------------------------------------
{
"_index":"my-index-000001",
"_type":"_doc",
"_id":"0",
"matched":true,
"explanation":{
"value":1.6943598,
"description":"weight(message:elasticsearch in 0) [PerFieldSimilarity], result of:",
"details":[
{
"value":1.6943598,
"description":"score(freq=1.0), computed as boost * idf * tf from:",
"details":[
{
"value":2.2,
"description":"boost",
"details":[]
},
{
"value":1.3862944,
"description":"idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:",
"details":[
{
"value":1,
"description":"n, number of documents containing term",
"details":[]
},
{
"value":5,
"description":"N, total number of documents with field",
"details":[]
}
]
},
{
"value":0.5555556,
"description":"tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:",
"details":[
{
"value":1.0,
"description":"freq, occurrences of term within document",
"details":[]
},
{
"value":1.2,
"description":"k1, term saturation parameter",
"details":[]
Nested queries should avoid adding unnecessary filters when possible. (#23079) When nested objects are present in the mappings, many queries get deoptimized due to the need to exclude documents that are not in the right space. For instance, a filter is applied to all queries that prevents them from matching non-root documents (`+*:* -_type:__*`). Moreover, a filter is applied to all child queries of `nested` queries in order to make sure that the child query only matches child documents (`_type:__nested_path`), which is required by `ToParentBlockJoinQuery` (the Lucene query behing Elasticsearch's `nested` queries). These additional filters slow down `nested` queries. In 1.7-, the cost was somehow amortized by the fact that we cached filters very aggressively. However, this has proven to be a significant source of slow downs since 2.0 for users of `nested` mappings and queries, see #20797. This change makes the filtering a bit smarter. For instance if the query is a `match_all` query, then we need to exclude nested docs. However, if the query is `foo: bar` then it may only match root documents since `foo` is a top-level field, so no additional filtering is required. Another improvement is to use a `FILTER` clause on all types rather than a `MUST_NOT` clause on all nested paths when possible since `FILTER` clauses are more efficient. Here are some examples of queries and how they get rewritten: ``` "match_all": {} ``` This query gets rewritten to `ConstantScore(+*:* -_type:__*)` on master and `ConstantScore(_type:AutomatonQuery {\norg.apache.lucene.util.automaton.Automaton@4371da44})` with this change. The automaton is the complement of `_type:__*` so it matches the same documents, but is faster since it is now a positive clause. Simplistic performance testing on a 10M index where each root document has 5 nested documents on average gave a latency of 420ms on master and 90ms with this change applied. ``` "term": { "foo": { "value": "0" } } ``` This query is rewritten to `+foo:0 #(ConstantScore(+*:* -_type:__*))^0.0` on master and `foo:0` with this change: we do not need to filter nested docs out since the query cannot match nested docs. While doing performance testing in the same conditions as above, response times went from 250ms to 50ms. ``` "nested": { "path": "nested", "query": { "term": { "nested.foo": { "value": "0" } } } } ``` This query is rewritten to `+ToParentBlockJoinQuery (+nested.foo:0 #_type:__nested) #(ConstantScore(+*:* -_type:__*))^0.0` on master and `ToParentBlockJoinQuery (nested.foo:0)` with this change. The top-level filter (`-_type:__*`) could be removed since `nested` queries only match documents of the parent space, as well as the child filter (`#_type:__nested`) since the child query may only match nested docs since the `nested` object has both `include_in_parent` and `include_in_root` set to `false`. While doing performance testing in the same conditions as above, response times went from 850ms to 270ms.
2017-02-14 10:05:19 -05:00
},
{
"value":0.75,
"description":"b, length normalization parameter",
"details":[]
Nested queries should avoid adding unnecessary filters when possible. (#23079) When nested objects are present in the mappings, many queries get deoptimized due to the need to exclude documents that are not in the right space. For instance, a filter is applied to all queries that prevents them from matching non-root documents (`+*:* -_type:__*`). Moreover, a filter is applied to all child queries of `nested` queries in order to make sure that the child query only matches child documents (`_type:__nested_path`), which is required by `ToParentBlockJoinQuery` (the Lucene query behing Elasticsearch's `nested` queries). These additional filters slow down `nested` queries. In 1.7-, the cost was somehow amortized by the fact that we cached filters very aggressively. However, this has proven to be a significant source of slow downs since 2.0 for users of `nested` mappings and queries, see #20797. This change makes the filtering a bit smarter. For instance if the query is a `match_all` query, then we need to exclude nested docs. However, if the query is `foo: bar` then it may only match root documents since `foo` is a top-level field, so no additional filtering is required. Another improvement is to use a `FILTER` clause on all types rather than a `MUST_NOT` clause on all nested paths when possible since `FILTER` clauses are more efficient. Here are some examples of queries and how they get rewritten: ``` "match_all": {} ``` This query gets rewritten to `ConstantScore(+*:* -_type:__*)` on master and `ConstantScore(_type:AutomatonQuery {\norg.apache.lucene.util.automaton.Automaton@4371da44})` with this change. The automaton is the complement of `_type:__*` so it matches the same documents, but is faster since it is now a positive clause. Simplistic performance testing on a 10M index where each root document has 5 nested documents on average gave a latency of 420ms on master and 90ms with this change applied. ``` "term": { "foo": { "value": "0" } } ``` This query is rewritten to `+foo:0 #(ConstantScore(+*:* -_type:__*))^0.0` on master and `foo:0` with this change: we do not need to filter nested docs out since the query cannot match nested docs. While doing performance testing in the same conditions as above, response times went from 250ms to 50ms. ``` "nested": { "path": "nested", "query": { "term": { "nested.foo": { "value": "0" } } } } ``` This query is rewritten to `+ToParentBlockJoinQuery (+nested.foo:0 #_type:__nested) #(ConstantScore(+*:* -_type:__*))^0.0` on master and `ToParentBlockJoinQuery (nested.foo:0)` with this change. The top-level filter (`-_type:__*`) could be removed since `nested` queries only match documents of the parent space, as well as the child filter (`#_type:__nested`) since the child query may only match nested docs since the `nested` object has both `include_in_parent` and `include_in_root` set to `false`. While doing performance testing in the same conditions as above, response times went from 850ms to 270ms.
2017-02-14 10:05:19 -05:00
},
{
"value":3.0,
"description":"dl, length of field",
"details":[]
Nested queries should avoid adding unnecessary filters when possible. (#23079) When nested objects are present in the mappings, many queries get deoptimized due to the need to exclude documents that are not in the right space. For instance, a filter is applied to all queries that prevents them from matching non-root documents (`+*:* -_type:__*`). Moreover, a filter is applied to all child queries of `nested` queries in order to make sure that the child query only matches child documents (`_type:__nested_path`), which is required by `ToParentBlockJoinQuery` (the Lucene query behing Elasticsearch's `nested` queries). These additional filters slow down `nested` queries. In 1.7-, the cost was somehow amortized by the fact that we cached filters very aggressively. However, this has proven to be a significant source of slow downs since 2.0 for users of `nested` mappings and queries, see #20797. This change makes the filtering a bit smarter. For instance if the query is a `match_all` query, then we need to exclude nested docs. However, if the query is `foo: bar` then it may only match root documents since `foo` is a top-level field, so no additional filtering is required. Another improvement is to use a `FILTER` clause on all types rather than a `MUST_NOT` clause on all nested paths when possible since `FILTER` clauses are more efficient. Here are some examples of queries and how they get rewritten: ``` "match_all": {} ``` This query gets rewritten to `ConstantScore(+*:* -_type:__*)` on master and `ConstantScore(_type:AutomatonQuery {\norg.apache.lucene.util.automaton.Automaton@4371da44})` with this change. The automaton is the complement of `_type:__*` so it matches the same documents, but is faster since it is now a positive clause. Simplistic performance testing on a 10M index where each root document has 5 nested documents on average gave a latency of 420ms on master and 90ms with this change applied. ``` "term": { "foo": { "value": "0" } } ``` This query is rewritten to `+foo:0 #(ConstantScore(+*:* -_type:__*))^0.0` on master and `foo:0` with this change: we do not need to filter nested docs out since the query cannot match nested docs. While doing performance testing in the same conditions as above, response times went from 250ms to 50ms. ``` "nested": { "path": "nested", "query": { "term": { "nested.foo": { "value": "0" } } } } ``` This query is rewritten to `+ToParentBlockJoinQuery (+nested.foo:0 #_type:__nested) #(ConstantScore(+*:* -_type:__*))^0.0` on master and `ToParentBlockJoinQuery (nested.foo:0)` with this change. The top-level filter (`-_type:__*`) could be removed since `nested` queries only match documents of the parent space, as well as the child filter (`#_type:__nested`) since the child query may only match nested docs since the `nested` object has both `include_in_parent` and `include_in_root` set to `false`. While doing performance testing in the same conditions as above, response times went from 850ms to 270ms.
2017-02-14 10:05:19 -05:00
},
{
"value":5.4,
"description":"avgdl, average length of field",
"details":[]
}
]
}
]
}
]
}
}
--------------------------------------------------
There is also a simpler way of specifying the query via the `q` parameter. The
specified `q` parameter value is then parsed as if the `query_string` query was
used. Example usage of the `q` parameter in the
explain API:
[source,console]
--------------------------------------------------
GET /my-index-000001/_explain/0?q=message:search
--------------------------------------------------
// TEST[setup:messages]
The API returns the same result as the previous request.