2013-08-28 19:24:34 -04:00
|
|
|
[[search-explain]]
|
2019-07-19 14:35:36 -04:00
|
|
|
=== Explain API
|
2013-08-28 19:24:34 -04:00
|
|
|
|
2019-09-20 04:59:11 -04:00
|
|
|
Returns information about why a specific document matches (or doesn't match) a
|
|
|
|
query.
|
|
|
|
|
|
|
|
[source,console]
|
|
|
|
--------------------------------------------------
|
|
|
|
GET /twitter/_explain/0
|
|
|
|
{
|
|
|
|
"query" : {
|
|
|
|
"match" : { "message" : "elasticsearch" }
|
|
|
|
}
|
|
|
|
}
|
|
|
|
--------------------------------------------------
|
|
|
|
// TEST[setup:twitter]
|
|
|
|
|
|
|
|
|
|
|
|
[[sample-api-request]]
|
|
|
|
==== {api-request-title}
|
|
|
|
|
|
|
|
`GET /<index>/_explain/<id>`
|
|
|
|
`POST /<index>/_explain/<id>`
|
|
|
|
|
|
|
|
[[sample-api-desc]]
|
|
|
|
==== {api-description-title}
|
|
|
|
|
|
|
|
The explain API computes a score explanation for a query and a specific
|
2013-08-28 19:24:34 -04:00
|
|
|
document. This can give useful feedback whether a document matches or
|
2013-10-13 10:46:56 -04:00
|
|
|
didn't match a specific query.
|
|
|
|
|
2013-08-28 19:24:34 -04:00
|
|
|
|
2019-09-20 04:59:11 -04:00
|
|
|
[[sample-api-path-params]]
|
|
|
|
==== {api-path-parms-title}
|
|
|
|
|
|
|
|
`<id>`::
|
|
|
|
(Required, integer) Defines the document ID.
|
|
|
|
|
|
|
|
`<index>`::
|
|
|
|
+
|
|
|
|
--
|
|
|
|
(Required, string)
|
|
|
|
Index names used to limit the request.
|
|
|
|
|
|
|
|
Only a single index name can be provided to this parameter.
|
|
|
|
--
|
|
|
|
|
|
|
|
|
|
|
|
[[sample-api-query-params]]
|
|
|
|
==== {api-query-parms-title}
|
|
|
|
|
|
|
|
include::{docdir}/rest-api/common-parms.asciidoc[tag=analyzer]
|
2013-08-28 19:24:34 -04:00
|
|
|
|
2019-09-20 04:59:11 -04:00
|
|
|
include::{docdir}/rest-api/common-parms.asciidoc[tag=analyze_wildcard]
|
|
|
|
|
|
|
|
include::{docdir}/rest-api/common-parms.asciidoc[tag=default_operator]
|
|
|
|
|
|
|
|
include::{docdir}/rest-api/common-parms.asciidoc[tag=df]
|
|
|
|
|
|
|
|
include::{docdir}/rest-api/common-parms.asciidoc[tag=lenient]
|
|
|
|
|
|
|
|
include::{docdir}/rest-api/common-parms.asciidoc[tag=preference]
|
|
|
|
|
|
|
|
include::{docdir}/rest-api/common-parms.asciidoc[tag=search-q]
|
|
|
|
|
|
|
|
`stored_fields`::
|
|
|
|
(Optional, string) A comma-separated list of stored fields to return in the
|
|
|
|
response.
|
|
|
|
|
|
|
|
include::{docdir}/rest-api/common-parms.asciidoc[tag=index-routing]
|
|
|
|
|
|
|
|
include::{docdir}/rest-api/common-parms.asciidoc[tag=source]
|
|
|
|
|
|
|
|
include::{docdir}/rest-api/common-parms.asciidoc[tag=source_excludes]
|
|
|
|
|
|
|
|
include::{docdir}/rest-api/common-parms.asciidoc[tag=source_includes]
|
|
|
|
|
|
|
|
|
|
|
|
[[sample-api-request-body]]
|
|
|
|
==== {api-request-body-title}
|
|
|
|
|
|
|
|
include::{docdir}/rest-api/common-parms.asciidoc[tag=query]
|
|
|
|
|
|
|
|
|
|
|
|
[[sample-api-example]]
|
|
|
|
==== {api-examples-title}
|
2013-08-28 19:24:34 -04:00
|
|
|
|
2019-09-09 12:35:50 -04:00
|
|
|
[source,console]
|
2016-08-01 08:09:54 -04:00
|
|
|
--------------------------------------------------
|
2018-12-10 22:45:13 -05:00
|
|
|
GET /twitter/_explain/0
|
2016-09-14 11:23:25 -04:00
|
|
|
{
|
2016-08-01 08:09:54 -04:00
|
|
|
"query" : {
|
2016-09-14 11:23:25 -04:00
|
|
|
"match" : { "message" : "elasticsearch" }
|
2016-08-01 08:09:54 -04:00
|
|
|
}
|
2016-09-14 11:23:25 -04:00
|
|
|
}
|
2016-08-01 08:09:54 -04:00
|
|
|
--------------------------------------------------
|
2016-09-14 11:23:25 -04:00
|
|
|
// TEST[setup:twitter]
|
2013-08-28 19:24:34 -04:00
|
|
|
|
2019-09-20 04:59:11 -04:00
|
|
|
|
|
|
|
The API returns the following response:
|
2013-08-28 19:24:34 -04:00
|
|
|
|
2019-09-06 09:22:08 -04:00
|
|
|
[source,console-result]
|
2013-08-28 19:24:34 -04:00
|
|
|
--------------------------------------------------
|
|
|
|
{
|
2018-09-06 08:42:06 -04:00
|
|
|
"_index":"twitter",
|
|
|
|
"_type":"_doc",
|
|
|
|
"_id":"0",
|
|
|
|
"matched":true,
|
|
|
|
"explanation":{
|
2019-12-10 12:04:58 -05:00
|
|
|
"value":1.6943598,
|
2018-09-06 08:42:06 -04:00
|
|
|
"description":"weight(message:elasticsearch in 0) [PerFieldSimilarity], result of:",
|
|
|
|
"details":[
|
2016-11-29 12:40:31 -05:00
|
|
|
{
|
2019-12-10 12:04:58 -05:00
|
|
|
"value":1.6943598,
|
|
|
|
"description":"score(freq=1.0), computed as boost * idf * tf from:",
|
2018-09-06 08:42:06 -04:00
|
|
|
"details":[
|
2016-11-29 12:40:31 -05:00
|
|
|
{
|
2018-09-06 08:42:06 -04:00
|
|
|
"value":2.2,
|
2018-12-05 07:43:56 -05:00
|
|
|
"description":"boost",
|
2018-09-06 08:42:06 -04:00
|
|
|
"details":[]
|
|
|
|
},
|
|
|
|
{
|
|
|
|
"value":1.3862944,
|
|
|
|
"description":"idf, computed as log(1 + (N - n + 0.5) / (n + 0.5)) from:",
|
|
|
|
"details":[
|
2016-11-29 12:40:31 -05:00
|
|
|
{
|
2018-09-06 08:42:06 -04:00
|
|
|
"value":1,
|
|
|
|
"description":"n, number of documents containing term",
|
|
|
|
"details":[]
|
2016-11-29 12:40:31 -05:00
|
|
|
},
|
|
|
|
{
|
2018-09-06 08:42:06 -04:00
|
|
|
"value":5,
|
|
|
|
"description":"N, total number of documents with field",
|
|
|
|
"details":[]
|
|
|
|
}
|
|
|
|
]
|
2016-11-29 12:40:31 -05:00
|
|
|
},
|
2018-09-06 08:42:06 -04:00
|
|
|
{
|
2019-12-10 12:04:58 -05:00
|
|
|
"value":0.5555556,
|
2018-09-06 08:42:06 -04:00
|
|
|
"description":"tf, computed as freq / (freq + k1 * (1 - b + b * dl / avgdl)) from:",
|
|
|
|
"details":[
|
2016-11-29 12:40:31 -05:00
|
|
|
{
|
2018-09-06 08:42:06 -04:00
|
|
|
"value":1.0,
|
|
|
|
"description":"freq, occurrences of term within document",
|
|
|
|
"details":[]
|
2016-11-29 12:40:31 -05:00
|
|
|
},
|
|
|
|
{
|
2018-09-06 08:42:06 -04:00
|
|
|
"value":1.2,
|
|
|
|
"description":"k1, term saturation parameter",
|
|
|
|
"details":[]
|
Nested queries should avoid adding unnecessary filters when possible. (#23079)
When nested objects are present in the mappings, many queries get deoptimized
due to the need to exclude documents that are not in the right space. For
instance, a filter is applied to all queries that prevents them from matching
non-root documents (`+*:* -_type:__*`). Moreover, a filter is applied to all
child queries of `nested` queries in order to make sure that the child query
only matches child documents (`_type:__nested_path`), which is required by
`ToParentBlockJoinQuery` (the Lucene query behing Elasticsearch's `nested`
queries).
These additional filters slow down `nested` queries. In 1.7-, the cost was
somehow amortized by the fact that we cached filters very aggressively. However,
this has proven to be a significant source of slow downs since 2.0 for users
of `nested` mappings and queries, see #20797.
This change makes the filtering a bit smarter. For instance if the query is a
`match_all` query, then we need to exclude nested docs. However, if the query
is `foo: bar` then it may only match root documents since `foo` is a top-level
field, so no additional filtering is required.
Another improvement is to use a `FILTER` clause on all types rather than a
`MUST_NOT` clause on all nested paths when possible since `FILTER` clauses
are more efficient.
Here are some examples of queries and how they get rewritten:
```
"match_all": {}
```
This query gets rewritten to `ConstantScore(+*:* -_type:__*)` on master and
`ConstantScore(_type:AutomatonQuery {\norg.apache.lucene.util.automaton.Automaton@4371da44})`
with this change. The automaton is the complement of `_type:__*` so it matches
the same documents, but is faster since it is now a positive clause. Simplistic
performance testing on a 10M index where each root document has 5 nested
documents on average gave a latency of 420ms on master and 90ms with this change
applied.
```
"term": {
"foo": {
"value": "0"
}
}
```
This query is rewritten to `+foo:0 #(ConstantScore(+*:* -_type:__*))^0.0` on
master and `foo:0` with this change: we do not need to filter nested docs out
since the query cannot match nested docs. While doing performance testing in
the same conditions as above, response times went from 250ms to 50ms.
```
"nested": {
"path": "nested",
"query": {
"term": {
"nested.foo": {
"value": "0"
}
}
}
}
```
This query is rewritten to
`+ToParentBlockJoinQuery (+nested.foo:0 #_type:__nested) #(ConstantScore(+*:* -_type:__*))^0.0`
on master and `ToParentBlockJoinQuery (nested.foo:0)` with this change. The
top-level filter (`-_type:__*`) could be removed since `nested` queries only
match documents of the parent space, as well as the child filter
(`#_type:__nested`) since the child query may only match nested docs since the
`nested` object has both `include_in_parent` and `include_in_root` set to
`false`. While doing performance testing in the same conditions as above,
response times went from 850ms to 270ms.
2017-02-14 10:05:19 -05:00
|
|
|
},
|
|
|
|
{
|
2018-09-06 08:42:06 -04:00
|
|
|
"value":0.75,
|
|
|
|
"description":"b, length normalization parameter",
|
|
|
|
"details":[]
|
Nested queries should avoid adding unnecessary filters when possible. (#23079)
When nested objects are present in the mappings, many queries get deoptimized
due to the need to exclude documents that are not in the right space. For
instance, a filter is applied to all queries that prevents them from matching
non-root documents (`+*:* -_type:__*`). Moreover, a filter is applied to all
child queries of `nested` queries in order to make sure that the child query
only matches child documents (`_type:__nested_path`), which is required by
`ToParentBlockJoinQuery` (the Lucene query behing Elasticsearch's `nested`
queries).
These additional filters slow down `nested` queries. In 1.7-, the cost was
somehow amortized by the fact that we cached filters very aggressively. However,
this has proven to be a significant source of slow downs since 2.0 for users
of `nested` mappings and queries, see #20797.
This change makes the filtering a bit smarter. For instance if the query is a
`match_all` query, then we need to exclude nested docs. However, if the query
is `foo: bar` then it may only match root documents since `foo` is a top-level
field, so no additional filtering is required.
Another improvement is to use a `FILTER` clause on all types rather than a
`MUST_NOT` clause on all nested paths when possible since `FILTER` clauses
are more efficient.
Here are some examples of queries and how they get rewritten:
```
"match_all": {}
```
This query gets rewritten to `ConstantScore(+*:* -_type:__*)` on master and
`ConstantScore(_type:AutomatonQuery {\norg.apache.lucene.util.automaton.Automaton@4371da44})`
with this change. The automaton is the complement of `_type:__*` so it matches
the same documents, but is faster since it is now a positive clause. Simplistic
performance testing on a 10M index where each root document has 5 nested
documents on average gave a latency of 420ms on master and 90ms with this change
applied.
```
"term": {
"foo": {
"value": "0"
}
}
```
This query is rewritten to `+foo:0 #(ConstantScore(+*:* -_type:__*))^0.0` on
master and `foo:0` with this change: we do not need to filter nested docs out
since the query cannot match nested docs. While doing performance testing in
the same conditions as above, response times went from 250ms to 50ms.
```
"nested": {
"path": "nested",
"query": {
"term": {
"nested.foo": {
"value": "0"
}
}
}
}
```
This query is rewritten to
`+ToParentBlockJoinQuery (+nested.foo:0 #_type:__nested) #(ConstantScore(+*:* -_type:__*))^0.0`
on master and `ToParentBlockJoinQuery (nested.foo:0)` with this change. The
top-level filter (`-_type:__*`) could be removed since `nested` queries only
match documents of the parent space, as well as the child filter
(`#_type:__nested`) since the child query may only match nested docs since the
`nested` object has both `include_in_parent` and `include_in_root` set to
`false`. While doing performance testing in the same conditions as above,
response times went from 850ms to 270ms.
2017-02-14 10:05:19 -05:00
|
|
|
},
|
|
|
|
{
|
2018-09-06 08:42:06 -04:00
|
|
|
"value":3.0,
|
|
|
|
"description":"dl, length of field",
|
|
|
|
"details":[]
|
Nested queries should avoid adding unnecessary filters when possible. (#23079)
When nested objects are present in the mappings, many queries get deoptimized
due to the need to exclude documents that are not in the right space. For
instance, a filter is applied to all queries that prevents them from matching
non-root documents (`+*:* -_type:__*`). Moreover, a filter is applied to all
child queries of `nested` queries in order to make sure that the child query
only matches child documents (`_type:__nested_path`), which is required by
`ToParentBlockJoinQuery` (the Lucene query behing Elasticsearch's `nested`
queries).
These additional filters slow down `nested` queries. In 1.7-, the cost was
somehow amortized by the fact that we cached filters very aggressively. However,
this has proven to be a significant source of slow downs since 2.0 for users
of `nested` mappings and queries, see #20797.
This change makes the filtering a bit smarter. For instance if the query is a
`match_all` query, then we need to exclude nested docs. However, if the query
is `foo: bar` then it may only match root documents since `foo` is a top-level
field, so no additional filtering is required.
Another improvement is to use a `FILTER` clause on all types rather than a
`MUST_NOT` clause on all nested paths when possible since `FILTER` clauses
are more efficient.
Here are some examples of queries and how they get rewritten:
```
"match_all": {}
```
This query gets rewritten to `ConstantScore(+*:* -_type:__*)` on master and
`ConstantScore(_type:AutomatonQuery {\norg.apache.lucene.util.automaton.Automaton@4371da44})`
with this change. The automaton is the complement of `_type:__*` so it matches
the same documents, but is faster since it is now a positive clause. Simplistic
performance testing on a 10M index where each root document has 5 nested
documents on average gave a latency of 420ms on master and 90ms with this change
applied.
```
"term": {
"foo": {
"value": "0"
}
}
```
This query is rewritten to `+foo:0 #(ConstantScore(+*:* -_type:__*))^0.0` on
master and `foo:0` with this change: we do not need to filter nested docs out
since the query cannot match nested docs. While doing performance testing in
the same conditions as above, response times went from 250ms to 50ms.
```
"nested": {
"path": "nested",
"query": {
"term": {
"nested.foo": {
"value": "0"
}
}
}
}
```
This query is rewritten to
`+ToParentBlockJoinQuery (+nested.foo:0 #_type:__nested) #(ConstantScore(+*:* -_type:__*))^0.0`
on master and `ToParentBlockJoinQuery (nested.foo:0)` with this change. The
top-level filter (`-_type:__*`) could be removed since `nested` queries only
match documents of the parent space, as well as the child filter
(`#_type:__nested`) since the child query may only match nested docs since the
`nested` object has both `include_in_parent` and `include_in_root` set to
`false`. While doing performance testing in the same conditions as above,
response times went from 850ms to 270ms.
2017-02-14 10:05:19 -05:00
|
|
|
},
|
|
|
|
{
|
2018-09-06 08:42:06 -04:00
|
|
|
"value":5.4,
|
|
|
|
"description":"avgdl, average length of field",
|
|
|
|
"details":[]
|
2016-11-29 12:40:31 -05:00
|
|
|
}
|
|
|
|
]
|
|
|
|
}
|
|
|
|
]
|
|
|
|
}
|
|
|
|
]
|
|
|
|
}
|
2016-08-01 08:09:54 -04:00
|
|
|
}
|
2013-08-28 19:24:34 -04:00
|
|
|
--------------------------------------------------
|
|
|
|
|
2019-09-20 04:59:11 -04:00
|
|
|
|
|
|
|
There is also a simpler way of specifying the query via the `q` parameter. The
|
|
|
|
specified `q` parameter value is then parsed as if the `query_string` query was
|
|
|
|
used. Example usage of the `q` parameter in the
|
|
|
|
explain API:
|
2013-08-28 19:24:34 -04:00
|
|
|
|
2019-09-09 12:35:50 -04:00
|
|
|
[source,console]
|
2013-08-28 19:24:34 -04:00
|
|
|
--------------------------------------------------
|
2018-12-10 22:45:13 -05:00
|
|
|
GET /twitter/_explain/0?q=message:search
|
2013-08-28 19:24:34 -04:00
|
|
|
--------------------------------------------------
|
2016-09-14 11:23:25 -04:00
|
|
|
// TEST[setup:twitter]
|
2013-08-28 19:24:34 -04:00
|
|
|
|
|
|
|
|
2019-09-20 04:59:11 -04:00
|
|
|
The API returns the same result as the previous request.
|