450 lines
16 KiB
Plaintext
450 lines
16 KiB
Plaintext
[[search-rank-eval]]
|
||
=== Ranking Evaluation API
|
||
|
||
Allows you to evaluate the quality of ranked search results over a set of
|
||
typical search queries.
|
||
|
||
[[search-rank-eval-api-request]]
|
||
==== {api-request-title}
|
||
|
||
`GET /<index>/_rank_eval`
|
||
|
||
`POST /<index>/_rank_eval`
|
||
|
||
|
||
[[search-rank-eval-api-desc]]
|
||
==== {api-description-title}
|
||
|
||
The ranking evaluation API allows you to evaluate the quality of ranked search
|
||
results over a set of typical search queries. Given this set of queries and a
|
||
list of manually rated documents, the `_rank_eval` endpoint calculates and
|
||
returns typical information retrieval metrics like _mean reciprocal rank_,
|
||
_precision_ or _discounted cumulative gain_.
|
||
|
||
Search quality evaluation starts with looking at the users of your search
|
||
application, and the things that they are searching for. Users have a specific
|
||
_information need_, for example they are looking for gift in a web shop or want
|
||
to book a flight for their next holiday. They usually enter some search terms
|
||
into a search box or some other web form. All of this information, together with
|
||
meta information about the user (for example the browser, location, earlier
|
||
preferences and so on) then gets translated into a query to the underlying
|
||
search system.
|
||
|
||
The challenge for search engineers is to tweak this translation process from
|
||
user entries to a concrete query in such a way, that the search results contain
|
||
the most relevant information with respect to the users information need. This
|
||
can only be done if the search result quality is evaluated constantly across a
|
||
representative test suite of typical user queries, so that improvements in the
|
||
rankings for one particular query doesn't negatively effect the ranking for
|
||
other types of queries.
|
||
|
||
In order to get started with search quality evaluation, three basic things are
|
||
needed:
|
||
|
||
. A collection of documents you want to evaluate your query performance against,
|
||
usually one or more indices.
|
||
. A collection of typical search requests that users enter into your system.
|
||
. A set of document ratings that judge the documents relevance with respect to a
|
||
search request.
|
||
|
||
It is important to note that one set of document ratings is needed per test
|
||
query, and that the relevance judgements are based on the information need of
|
||
the user that entered the query.
|
||
|
||
The ranking evaluation API provides a convenient way to use this information in
|
||
a ranking evaluation request to calculate different search evaluation metrics.
|
||
This gives a first estimation of your overall search quality and give you a
|
||
measurement to optimize against when fine-tuning various aspect of the query
|
||
generation in your application.
|
||
|
||
|
||
[[search-rank-eval-api-path-params]]
|
||
==== {api-path-parms-title}
|
||
|
||
`<index>`::
|
||
(Required, string) Comma-separated list or wildcard expression of index names
|
||
used to limit the request.
|
||
|
||
[[search-rank-eval-api-query-params]]
|
||
==== {api-query-parms-title}
|
||
|
||
include::{docdir}/rest-api/common-parms.asciidoc[tag=allow-no-indices]
|
||
|
||
include::{docdir}/rest-api/common-parms.asciidoc[tag=expand-wildcards]
|
||
+
|
||
--
|
||
Defaults to `open`.
|
||
--
|
||
|
||
include::{docdir}/rest-api/common-parms.asciidoc[tag=index-ignore-unavailable]
|
||
|
||
|
||
[[search-rank-eval-api-example]]
|
||
==== {api-examples-title}
|
||
|
||
In its most basic form, a request to the `_rank_eval` endpoint has two sections:
|
||
|
||
[source,js]
|
||
-----------------------------
|
||
GET /my_index/_rank_eval
|
||
{
|
||
"requests": [ ... ], <1>
|
||
"metric": { <2>
|
||
"mean_reciprocal_rank": { ... } <3>
|
||
}
|
||
}
|
||
-----------------------------
|
||
// NOTCONSOLE
|
||
|
||
<1> a set of typical search requests, together with their provided ratings
|
||
<2> definition of the evaluation metric to calculate
|
||
<3> a specific metric and its parameters
|
||
|
||
The request section contains several search requests typical to your
|
||
application, along with the document ratings for each particular search request.
|
||
|
||
[source,js]
|
||
-----------------------------
|
||
GET /my_index/_rank_eval
|
||
{
|
||
"requests": [
|
||
{
|
||
"id": "amsterdam_query", <1>
|
||
"request": { <2>
|
||
"query": { "match": { "text": "amsterdam" }}
|
||
},
|
||
"ratings": [ <3>
|
||
{ "_index": "my_index", "_id": "doc1", "rating": 0 },
|
||
{ "_index": "my_index", "_id": "doc2", "rating": 3},
|
||
{ "_index": "my_index", "_id": "doc3", "rating": 1 }
|
||
]
|
||
},
|
||
{
|
||
"id": "berlin_query",
|
||
"request": {
|
||
"query": { "match": { "text": "berlin" }}
|
||
},
|
||
"ratings": [
|
||
{ "_index": "my_index", "_id": "doc1", "rating": 1 }
|
||
]
|
||
}
|
||
]
|
||
}
|
||
-----------------------------
|
||
// NOTCONSOLE
|
||
|
||
<1> the search requests id, used to group result details later
|
||
<2> the query that is being evaluated
|
||
<3> a list of document ratings, each entry containing the documents `_index` and
|
||
`_id` together with the rating of the documents relevance with regards to this
|
||
search request
|
||
|
||
A document `rating` can be any integer value that expresses the relevance of the
|
||
document on a user defined scale. For some of the metrics, just giving a binary
|
||
rating (for example `0` for irrelevant and `1` for relevant) will be sufficient,
|
||
other metrics can use a more fine grained scale.
|
||
|
||
|
||
===== Template based ranking evaluation
|
||
|
||
As an alternative to having to provide a single query per test request, it is
|
||
possible to specify query templates in the evaluation request and later refer to
|
||
them. Queries with similar structure that only differ in their parameters don't
|
||
have to be repeated all the time in the `requests` section this way. In typical
|
||
search systems where user inputs usually get filled into a small set of query
|
||
templates, this helps making the evaluation request more succinct.
|
||
|
||
[source,js]
|
||
--------------------------------
|
||
GET /my_index/_rank_eval
|
||
{
|
||
[...]
|
||
"templates": [
|
||
{
|
||
"id": "match_one_field_query", <1>
|
||
"template": { <2>
|
||
"inline": {
|
||
"query": {
|
||
"match": { "{{field}}": { "query": "{{query_string}}" }}
|
||
}
|
||
}
|
||
}
|
||
}
|
||
],
|
||
"requests": [
|
||
{
|
||
"id": "amsterdam_query"
|
||
"ratings": [ ... ],
|
||
"template_id": "match_one_field_query", <3>
|
||
"params": { <4>
|
||
"query_string": "amsterdam",
|
||
"field": "text"
|
||
}
|
||
},
|
||
[...]
|
||
}
|
||
--------------------------------
|
||
// NOTCONSOLE
|
||
|
||
<1> the template id
|
||
<2> the template definition to use
|
||
<3> a reference to a previously defined template
|
||
<4> the parameters to use to fill the template
|
||
|
||
|
||
===== Available evaluation metrics
|
||
|
||
The `metric` section determines which of the available evaluation metrics is
|
||
going to be used. The following metrics are supported:
|
||
|
||
[float]
|
||
[[k-precision]]
|
||
===== Precision at K (P@k)
|
||
|
||
This metric measures the number of relevant results in the top k search results.
|
||
Its a form of the well known
|
||
https://en.wikipedia.org/wiki/Information_retrieval#Precision[Precision] metric
|
||
that only looks at the top k documents. It is the fraction of relevant documents
|
||
in those first k search. A precision at 10 (P@10) value of 0.6 then means six
|
||
out of the 10 top hits are relevant with respect to the users information need.
|
||
|
||
P@k works well as a simple evaluation metric that has the benefit of being easy
|
||
to understand and explain. Documents in the collection need to be rated either
|
||
as relevant or irrelevant with respect to the current query. P@k does not take
|
||
into account where in the top k results the relevant documents occur, so a
|
||
ranking of ten results that contains one relevant result in position 10 is
|
||
equally good as a ranking of ten results that contains one relevant result in
|
||
position 1.
|
||
|
||
[source,console]
|
||
--------------------------------
|
||
GET /twitter/_rank_eval
|
||
{
|
||
"requests": [
|
||
{
|
||
"id": "JFK query",
|
||
"request": { "query": { "match_all": {}}},
|
||
"ratings": []
|
||
}],
|
||
"metric": {
|
||
"precision": {
|
||
"k" : 20,
|
||
"relevant_rating_threshold": 1,
|
||
"ignore_unlabeled": false
|
||
}
|
||
}
|
||
}
|
||
--------------------------------
|
||
// TEST[setup:twitter]
|
||
|
||
The `precision` metric takes the following optional parameters
|
||
|
||
[cols="<,<",options="header",]
|
||
|=======================================================================
|
||
|Parameter |Description
|
||
|`k` |sets the maximum number of documents retrieved per query. This value will act in place of the usual `size` parameter
|
||
in the query. Defaults to 10.
|
||
|`relevant_rating_threshold` |sets the rating threshold above which documents are considered to be
|
||
"relevant". Defaults to `1`.
|
||
|`ignore_unlabeled` |controls how unlabeled documents in the search results are counted.
|
||
If set to 'true', unlabeled documents are ignored and neither count as relevant or irrelevant. Set to 'false' (the default), they are treated as irrelevant.
|
||
|=======================================================================
|
||
|
||
|
||
[float]
|
||
===== Mean reciprocal rank
|
||
|
||
For every query in the test suite, this metric calculates the reciprocal of the
|
||
rank of the first relevant document. For example finding the first relevant
|
||
result in position 3 means the reciprocal rank is 1/3. The reciprocal rank for
|
||
each query is averaged across all queries in the test suite to give the
|
||
https://en.wikipedia.org/wiki/Mean_reciprocal_rank[mean reciprocal rank].
|
||
|
||
[source,console]
|
||
--------------------------------
|
||
GET /twitter/_rank_eval
|
||
{
|
||
"requests": [
|
||
{
|
||
"id": "JFK query",
|
||
"request": { "query": { "match_all": {}}},
|
||
"ratings": []
|
||
}],
|
||
"metric": {
|
||
"mean_reciprocal_rank": {
|
||
"k" : 20,
|
||
"relevant_rating_threshold" : 1
|
||
}
|
||
}
|
||
}
|
||
--------------------------------
|
||
// TEST[setup:twitter]
|
||
|
||
The `mean_reciprocal_rank` metric takes the following optional parameters
|
||
|
||
[cols="<,<",options="header",]
|
||
|=======================================================================
|
||
|Parameter |Description
|
||
|`k` |sets the maximum number of documents retrieved per query. This value will act in place of the usual `size` parameter
|
||
in the query. Defaults to 10.
|
||
|`relevant_rating_threshold` |Sets the rating threshold above which documents are considered to be
|
||
"relevant". Defaults to `1`.
|
||
|=======================================================================
|
||
|
||
|
||
[float]
|
||
===== Discounted cumulative gain (DCG)
|
||
|
||
In contrast to the two metrics above,
|
||
https://en.wikipedia.org/wiki/Discounted_cumulative_gain[discounted cumulative gain]
|
||
takes both, the rank and the rating of the search results, into account.
|
||
|
||
The assumption is that highly relevant documents are more useful for the user
|
||
when appearing at the top of the result list. Therefore, the DCG formula reduces
|
||
the contribution that high ratings for documents on lower search ranks have on
|
||
the overall DCG metric.
|
||
|
||
[source,console]
|
||
--------------------------------
|
||
GET /twitter/_rank_eval
|
||
{
|
||
"requests": [
|
||
{
|
||
"id": "JFK query",
|
||
"request": { "query": { "match_all": {}}},
|
||
"ratings": []
|
||
}],
|
||
"metric": {
|
||
"dcg": {
|
||
"k" : 20,
|
||
"normalize": false
|
||
}
|
||
}
|
||
}
|
||
--------------------------------
|
||
// TEST[setup:twitter]
|
||
|
||
The `dcg` metric takes the following optional parameters:
|
||
|
||
[cols="<,<",options="header",]
|
||
|=======================================================================
|
||
|Parameter |Description
|
||
|`k` |sets the maximum number of documents retrieved per query. This value will act in place of the usual `size` parameter
|
||
in the query. Defaults to 10.
|
||
|`normalize` | If set to `true`, this metric will calculate the https://en.wikipedia.org/wiki/Discounted_cumulative_gain#Normalized_DCG[Normalized DCG].
|
||
|=======================================================================
|
||
|
||
|
||
[float]
|
||
===== Expected Reciprocal Rank (ERR)
|
||
|
||
Expected Reciprocal Rank (ERR) is an extension of the classical reciprocal rank
|
||
for the graded relevance case (Olivier Chapelle, Donald Metzler, Ya Zhang, and
|
||
Pierre Grinspan. 2009.
|
||
http://olivier.chapelle.cc/pub/err.pdf[Expected reciprocal rank for graded relevance].)
|
||
|
||
It is based on the assumption of a cascade model of search, in which a user
|
||
scans through ranked search results in order and stops at the first document
|
||
that satisfies the information need. For this reason, it is a good metric for
|
||
question answering and navigation queries, but less so for survey oriented
|
||
information needs where the user is interested in finding many relevant
|
||
documents in the top k results.
|
||
|
||
The metric models the expectation of the reciprocal of the position at which a
|
||
user stops reading through the result list. This means that relevant document in
|
||
top ranking positions will contribute much to the overall score. However, the
|
||
same document will contribute much less to the score if it appears in a lower
|
||
rank, even more so if there are some relevant (but maybe less relevant)
|
||
documents preceding it. In this way, the ERR metric discounts documents which
|
||
are shown after very relevant documents. This introduces a notion of dependency
|
||
in the ordering of relevant documents that e.g. Precision or DCG don't account
|
||
for.
|
||
|
||
[source,console]
|
||
--------------------------------
|
||
GET /twitter/_rank_eval
|
||
{
|
||
"requests": [
|
||
{
|
||
"id": "JFK query",
|
||
"request": { "query": { "match_all": {}}},
|
||
"ratings": []
|
||
}],
|
||
"metric": {
|
||
"expected_reciprocal_rank": {
|
||
"maximum_relevance" : 3,
|
||
"k" : 20
|
||
}
|
||
}
|
||
}
|
||
--------------------------------
|
||
// TEST[setup:twitter]
|
||
|
||
The `expected_reciprocal_rank` metric takes the following parameters:
|
||
|
||
[cols="<,<",options="header",]
|
||
|=======================================================================
|
||
|Parameter |Description
|
||
| `maximum_relevance` | Mandatory parameter. The highest relevance grade used in the user supplied
|
||
relevance judgments.
|
||
|`k` | sets the maximum number of documents retrieved per query. This value will act in place of the usual `size` parameter
|
||
in the query. Defaults to 10.
|
||
|=======================================================================
|
||
|
||
|
||
===== Response format
|
||
|
||
The response of the `_rank_eval` endpoint contains the overall calculated result
|
||
for the defined quality metric, a `details` section with a breakdown of results
|
||
for each query in the test suite and an optional `failures` section that shows
|
||
potential errors of individual queries. The response has the following format:
|
||
|
||
[source,js]
|
||
--------------------------------
|
||
{
|
||
"rank_eval": {
|
||
"metric_score": 0.4, <1>
|
||
"details": {
|
||
"my_query_id1": { <2>
|
||
"metric_score": 0.6, <3>
|
||
"unrated_docs": [ <4>
|
||
{
|
||
"_index": "my_index",
|
||
"_id": "1960795"
|
||
}, [...]
|
||
],
|
||
"hits": [
|
||
{
|
||
"hit": { <5>
|
||
"_index": "my_index",
|
||
"_type": "page",
|
||
"_id": "1528558",
|
||
"_score": 7.0556192
|
||
},
|
||
"rating": 1
|
||
}, [...]
|
||
],
|
||
"metric_details": { <6>
|
||
"precision" : {
|
||
"relevant_docs_retrieved": 6,
|
||
"docs_retrieved": 10
|
||
}
|
||
}
|
||
},
|
||
"my_query_id2" : { [...] }
|
||
},
|
||
"failures": { [...] }
|
||
}
|
||
}
|
||
--------------------------------
|
||
// NOTCONSOLE
|
||
|
||
<1> the overall evaluation quality calculated by the defined metric
|
||
<2> the `details` section contains one entry for every query in the original `requests` section, keyed by the search request id
|
||
<3> the `metric_score` in the `details` section shows the contribution of this query to the global quality metric score
|
||
<4> the `unrated_docs` section contains an `_index` and `_id` entry for each document in the search result for this
|
||
query that didn't have a ratings value. This can be used to ask the user to supply ratings for these documents
|
||
<5> the `hits` section shows a grouping of the search results with their supplied rating
|
||
<6> the `metric_details` give additional information about the calculated quality metric (e.g. how many of the retrieved
|
||
documents where relevant). The content varies for each metric but allows for better interpretation of the results
|