157 Commits

Author SHA1 Message Date
Isabel Drost-Fromm
d34a117422 Use roundtrip helper for rank eval spec tests. 2016-09-07 13:03:18 +02:00
Isabel Drost-Fromm
ba9956a468 Remove FromXContentBuilder from RatedDocumentKey 2016-09-07 12:52:30 +02:00
Isabel Drost-Fromm
9d8bb720dc Remove leftover FromXContentBuilder reference 2016-09-07 12:50:04 +02:00
Isabel Drost-Fromm
2af0218bdd Rename test class to match renamed implementation
Used to be QuerySpec, is now RatedRequest; changing name
of test accordingly.
2016-09-07 12:12:06 +02:00
Isabel Drost-Fromm
0d25fc4925 Re-use roundtrip helper for QuerySpecTests 2016-09-07 12:11:07 +02:00
Isabel Drost-Fromm
b3a4a89151 Replace magic number in test with random number. 2016-09-07 12:06:39 +02:00
Isabel Drost-Fromm
c8a3b3c32f Move xcontent roundtrip method to helper class. 2016-09-07 12:02:00 +02:00
Isabel Drost-Fromm
efbef20361 Remove call to getClass from hashCode implementations. 2016-09-07 11:53:39 +02:00
Isabel Drost-Fromm
333b769871 Remove usage of FromXContentBuilder 2016-09-07 11:48:50 +02:00
Isabel Drost-Fromm
6bc646ac84 Make constructor enforce non-optional position argument. 2016-09-07 11:21:50 +02:00
Isabel Drost-Fromm
b8652b1224 Test wouldn't compile w/o these annotations. 2016-09-07 11:21:29 +02:00
Isabel Drost-Fromm
c4cfb9d7b5 Merge branch 'feature/rank-eval' into feature/rank-eval-roundtrip-testing 2016-09-07 11:05:56 +02:00
Christoph Büscher
0b92d524a7 Add threshold for document ratings for PrecisionAtN and ReciprocalRank
PrecisionAtN and ReciprocalRank are binary evaluation metrics by default that
only distiguish between relevant/irrelevant search results. So far we assumed
that relevant documents are labaled with 1 (irrelevant docs with 0) in the
evaluation request, but this is cumbersome if the ratings are provided on a
larger integer scale and would need to get mapped to a 0/1 value.

This change introduces a threshold parameter on the PrecisionAtN and
ReciprocalRank metric than can be used to set the threshold from which on a
document is considered "relevant". It defaults to 1, so in case of 0/1 ratings
the threshold doesn't have to be set and only ratings with value 0 are
considered to be irrelevant.
2016-09-05 15:34:47 +02:00
Christoph Büscher
8daec109fc Adding additional json object level to PrecisionAtN rendering 2016-09-02 18:03:58 +02:00
Isabel Drost-Fromm
f18e23eb51 Fix test errors, add roundtrip testing to RankEvalSpec
This adds roundtrip testing to RankEvalSpec, fixes issues
introduced with the previous roundtrip tests, splits
xcontent generation/parsing from actually checking the resulting
objects to deal with e.g. all evaluation metrics needing
some extra treatment.

Renames QuerySpec to RatedRequest, renames newly introduced xcontent
generation helper to conform with naming conventions.

Fixes several lines that were too long, adds missing types where
needed.
2016-08-25 12:09:55 +02:00
Isabel Drost-Fromm
87367be4e7 Add roundtrip testing to RatedDocumentKey 2016-08-24 15:32:44 +02:00
Isabel Drost-Fromm
94497871b5 Add roundtrip testing to QuerySpec 2016-08-24 15:17:58 +02:00
Isabel Drost-Fromm
5979802415 Add comment wrt to changed xcontent generation 2016-08-24 15:17:42 +02:00
Isabel Drost-Fromm
cebb0ba0d8 Add roundtripping to ReciprocalRank 2016-08-24 14:41:58 +02:00
Isabel Drost-Fromm
5c9cc1d453 Add roundtripping to PrecisionAtN 2016-08-24 14:39:12 +02:00
Isabel Drost-Fromm
a2a92b9629 Add roundtrip xcontent test to DiscountedCumulativeGainAt
This factors the roundtripping out of RatedDocumentTests. Makes
RankedListQualityMetric and RatedDocument implement FromXContenBuilder
to be able to do the aforementioned refactoring in a generic way. Adds
a roundtrip test to DiscountedCumulativeGainAt.

Open questions:

DiscountedCumulativeGain didn't have a constructor that accepted all possible
parameters as arguments. Added one. I guess we still want to keep the one
that only requires the position argument?

To make roundtripping work I had to change the NAME parameter when generating
XContent for DiscountedCumulativeGainAt - all remaining unit tests seem to be
passing (haven't checked the REST tests yet) - need to figure out why that was
there to begin with.
2016-08-24 14:21:24 +02:00
Christoph Büscher
2f506bfe04 Add toXContent method to classes used in ranking request 2016-08-18 18:14:21 +02:00
Christoph Büscher
2e892185f0 Adapting to introduction of SearchRequestParers on master 2016-08-17 11:54:07 +02:00
Christoph Büscher
1eedb4c033 Resolve missing imports due to changes in master 2016-08-16 11:48:28 +02:00
Christoph Büscher
b7af7c21d1 Adapt to changes in master 2016-08-12 11:03:55 +02:00
Christoph Büscher
cac4961ef4 Fix test failure because of broken RankEvalResponse serialization
The introduction of RatedDocumentKey accidentally broke the response
serialization because it cannot be written using writeGenericValue().
2016-08-11 15:42:18 +02:00
Christoph Büscher
a2e6dc2750 Adressing review comments and adding test for failed request 2016-08-11 10:08:40 +02:00
Christoph Büscher
8856565b89 Make RankEvalRequestTests work with transport client 2016-08-11 10:07:09 +02:00
Christoph Büscher
795017ddfa Adding listeners for search requests that collect results 2016-08-11 10:06:23 +02:00
Christoph Büscher
e71e29b3a0 Using client instead of TransportSearchAction 2016-08-11 10:05:28 +02:00
Isabel Drost-Fromm
0be997232a Add ConstructingObjectParser to RatedDocumentKey 2016-08-09 12:49:07 +02:00
Isabel Drost-Fromm
c2bd58e13d Switch to ConstructingObjectParser 2016-08-09 12:22:44 +02:00
Isabel Drost-Fromm
978d5366d2 Merge branch 'feature/rank-eval' into feature/rank-eval_index_type_to_id 2016-08-09 10:54:14 +02:00
Christoph Büscher
fa459f88dd Add normalization option
When switched on, compute the normalized ndcg variant.
2016-08-08 16:37:15 +02:00
Christoph Büscher
87e13ca8bb Add Discounted Cumulative Gain metric 2016-08-08 16:37:15 +02:00
Isabel Drost-Fromm
b2fa7c4d96 Fix merge errors. 2016-08-08 14:41:58 +02:00
Isabel Drost-Fromm
a6caabb730 Merge branch 'feature/rank-eval' into feature/rank-eval_index_type_to_id 2016-08-08 14:21:19 +02:00
Isabel Drost-Fromm
cfaa62723d Actually use index/type in addition to docid for comparing hits. 2016-08-08 14:09:27 +02:00
Christoph Büscher
0578a96483 Merge RankEvalResult with Response
The current response object only serves as a wrapper around the result object.
This change merges the two classes into one.
2016-08-08 13:29:44 +02:00
Christoph Büscher
438893c0c7 Adapt to changes to how named writables are registered 2016-08-04 13:40:46 +02:00
Christoph Büscher
d71dc205fa Moving averaging of partial evaluation results to RankedListQualityMetric
For the two current metrics Prec@ and reciprocal rank we currently average the
partial results in the transport action. If other metric later need a different
behaviour or want to parametrize this, this operation should be part of the
metric itself, so this change moves it there. Also removing on of the two test
packages, main code is also in one package only.
2016-08-04 10:34:46 +02:00
Isabel Drost-Fromm
ac3f2421b1 Merge branch 'origin/feature/rank-eval' into feature/rank-eval_index_type_to_id 2016-08-03 13:50:13 +02:00
Isabel Drost-Fromm
0fb7dd9054 Merge pull request #19623 from MaineC/feature/rank-eval
Add parsing of type information
2016-08-02 13:50:16 +02:00
Isabel Drost-Fromm
869e471f9f Add missing file 2016-07-28 11:42:05 +02:00
Isabel Drost-Fromm
34cbc10128 Add index and type information to rated doc
Also add roundtrip testing of the xcontent serialisation of RatedDoc
2016-07-28 11:41:54 +02:00
Christoph Büscher
71c0d59e95 Add option for maximally allowed rank to Reciprocal Rank metric 2016-07-27 16:24:41 +02:00
Isabel Drost-Fromm
ad9f060dc7 Use type information in request
Adds parsing of type and actually using it in TransportRankEvalAction.

Missing: A good idea how to actually test this in isolation...
2016-07-27 15:49:37 +02:00
Isabel Drost-Fromm
2bb5cb83a7 Merge branch 'feature/rank-eval' of github.com:elasticsearch/elasticsearch into feature/rank-eval 2016-07-27 15:46:44 +02:00
Christoph Büscher
ad87bacf91 Add Reciprocal Rank query evaluation metric
This adds a second query evaluation metric alongside precision_at. Reciprocal
Rank is defined as 1/rank, where rank is the position of the first relevant
document in the search result. The results are averaged across all queries
across the sample of queries, according to
https://en.wikipedia.org/wiki/Mean_reciprocal_rank
2016-07-27 15:23:05 +02:00
Christoph Büscher
4162582ee8 Adapt to renaming of RestTestCandidate 2016-07-27 11:39:59 +02:00