Commit Graph

154 Commits

Author SHA1 Message Date
Isabel Drost-Fromm 2af0218bdd Rename test class to match renamed implementation
Used to be QuerySpec, is now RatedRequest; changing name
of test accordingly.
2016-09-07 12:12:06 +02:00
Isabel Drost-Fromm 0d25fc4925 Re-use roundtrip helper for QuerySpecTests 2016-09-07 12:11:07 +02:00
Isabel Drost-Fromm b3a4a89151 Replace magic number in test with random number. 2016-09-07 12:06:39 +02:00
Isabel Drost-Fromm c8a3b3c32f Move xcontent roundtrip method to helper class. 2016-09-07 12:02:00 +02:00
Isabel Drost-Fromm efbef20361 Remove call to getClass from hashCode implementations. 2016-09-07 11:53:39 +02:00
Isabel Drost-Fromm 333b769871 Remove usage of FromXContentBuilder 2016-09-07 11:48:50 +02:00
Isabel Drost-Fromm 6bc646ac84 Make constructor enforce non-optional position argument. 2016-09-07 11:21:50 +02:00
Isabel Drost-Fromm b8652b1224 Test wouldn't compile w/o these annotations. 2016-09-07 11:21:29 +02:00
Isabel Drost-Fromm c4cfb9d7b5 Merge branch 'feature/rank-eval' into feature/rank-eval-roundtrip-testing 2016-09-07 11:05:56 +02:00
Christoph Büscher 0b92d524a7 Add threshold for document ratings for PrecisionAtN and ReciprocalRank
PrecisionAtN and ReciprocalRank are binary evaluation metrics by default that
only distiguish between relevant/irrelevant search results. So far we assumed
that relevant documents are labaled with 1 (irrelevant docs with 0) in the
evaluation request, but this is cumbersome if the ratings are provided on a
larger integer scale and would need to get mapped to a 0/1 value.

This change introduces a threshold parameter on the PrecisionAtN and
ReciprocalRank metric than can be used to set the threshold from which on a
document is considered "relevant". It defaults to 1, so in case of 0/1 ratings
the threshold doesn't have to be set and only ratings with value 0 are
considered to be irrelevant.
2016-09-05 15:34:47 +02:00
Christoph Büscher 8daec109fc Adding additional json object level to PrecisionAtN rendering 2016-09-02 18:03:58 +02:00
Isabel Drost-Fromm f18e23eb51 Fix test errors, add roundtrip testing to RankEvalSpec
This adds roundtrip testing to RankEvalSpec, fixes issues
introduced with the previous roundtrip tests, splits
xcontent generation/parsing from actually checking the resulting
objects to deal with e.g. all evaluation metrics needing
some extra treatment.

Renames QuerySpec to RatedRequest, renames newly introduced xcontent
generation helper to conform with naming conventions.

Fixes several lines that were too long, adds missing types where
needed.
2016-08-25 12:09:55 +02:00
Isabel Drost-Fromm 87367be4e7 Add roundtrip testing to RatedDocumentKey 2016-08-24 15:32:44 +02:00
Isabel Drost-Fromm 94497871b5 Add roundtrip testing to QuerySpec 2016-08-24 15:17:58 +02:00
Isabel Drost-Fromm 5979802415 Add comment wrt to changed xcontent generation 2016-08-24 15:17:42 +02:00
Isabel Drost-Fromm cebb0ba0d8 Add roundtripping to ReciprocalRank 2016-08-24 14:41:58 +02:00
Isabel Drost-Fromm 5c9cc1d453 Add roundtripping to PrecisionAtN 2016-08-24 14:39:12 +02:00
Isabel Drost-Fromm a2a92b9629 Add roundtrip xcontent test to DiscountedCumulativeGainAt
This factors the roundtripping out of RatedDocumentTests. Makes
RankedListQualityMetric and RatedDocument implement FromXContenBuilder
to be able to do the aforementioned refactoring in a generic way. Adds
a roundtrip test to DiscountedCumulativeGainAt.

Open questions:

DiscountedCumulativeGain didn't have a constructor that accepted all possible
parameters as arguments. Added one. I guess we still want to keep the one
that only requires the position argument?

To make roundtripping work I had to change the NAME parameter when generating
XContent for DiscountedCumulativeGainAt - all remaining unit tests seem to be
passing (haven't checked the REST tests yet) - need to figure out why that was
there to begin with.
2016-08-24 14:21:24 +02:00
Christoph Büscher 2f506bfe04 Add toXContent method to classes used in ranking request 2016-08-18 18:14:21 +02:00
Christoph Büscher 2e892185f0 Adapting to introduction of SearchRequestParers on master 2016-08-17 11:54:07 +02:00
Christoph Büscher 1eedb4c033 Resolve missing imports due to changes in master 2016-08-16 11:48:28 +02:00
Christoph Büscher b7af7c21d1 Adapt to changes in master 2016-08-12 11:03:55 +02:00
Christoph Büscher cac4961ef4 Fix test failure because of broken RankEvalResponse serialization
The introduction of RatedDocumentKey accidentally broke the response
serialization because it cannot be written using writeGenericValue().
2016-08-11 15:42:18 +02:00
Christoph Büscher a2e6dc2750 Adressing review comments and adding test for failed request 2016-08-11 10:08:40 +02:00
Christoph Büscher 8856565b89 Make RankEvalRequestTests work with transport client 2016-08-11 10:07:09 +02:00
Christoph Büscher 795017ddfa Adding listeners for search requests that collect results 2016-08-11 10:06:23 +02:00
Christoph Büscher e71e29b3a0 Using client instead of TransportSearchAction 2016-08-11 10:05:28 +02:00
Isabel Drost-Fromm 0be997232a Add ConstructingObjectParser to RatedDocumentKey 2016-08-09 12:49:07 +02:00
Isabel Drost-Fromm c2bd58e13d Switch to ConstructingObjectParser 2016-08-09 12:22:44 +02:00
Isabel Drost-Fromm 978d5366d2 Merge branch 'feature/rank-eval' into feature/rank-eval_index_type_to_id 2016-08-09 10:54:14 +02:00
Christoph Büscher fa459f88dd Add normalization option
When switched on, compute the normalized ndcg variant.
2016-08-08 16:37:15 +02:00
Christoph Büscher 87e13ca8bb Add Discounted Cumulative Gain metric 2016-08-08 16:37:15 +02:00
Isabel Drost-Fromm b2fa7c4d96 Fix merge errors. 2016-08-08 14:41:58 +02:00
Isabel Drost-Fromm a6caabb730 Merge branch 'feature/rank-eval' into feature/rank-eval_index_type_to_id 2016-08-08 14:21:19 +02:00
Isabel Drost-Fromm cfaa62723d Actually use index/type in addition to docid for comparing hits. 2016-08-08 14:09:27 +02:00
Christoph Büscher 0578a96483 Merge RankEvalResult with Response
The current response object only serves as a wrapper around the result object.
This change merges the two classes into one.
2016-08-08 13:29:44 +02:00
Christoph Büscher 438893c0c7 Adapt to changes to how named writables are registered 2016-08-04 13:40:46 +02:00
Christoph Büscher d71dc205fa Moving averaging of partial evaluation results to RankedListQualityMetric
For the two current metrics Prec@ and reciprocal rank we currently average the
partial results in the transport action. If other metric later need a different
behaviour or want to parametrize this, this operation should be part of the
metric itself, so this change moves it there. Also removing on of the two test
packages, main code is also in one package only.
2016-08-04 10:34:46 +02:00
Isabel Drost-Fromm ac3f2421b1 Merge branch 'origin/feature/rank-eval' into feature/rank-eval_index_type_to_id 2016-08-03 13:50:13 +02:00
Isabel Drost-Fromm 0fb7dd9054 Merge pull request #19623 from MaineC/feature/rank-eval
Add parsing of type information
2016-08-02 13:50:16 +02:00
Isabel Drost-Fromm 869e471f9f Add missing file 2016-07-28 11:42:05 +02:00
Isabel Drost-Fromm 34cbc10128 Add index and type information to rated doc
Also add roundtrip testing of the xcontent serialisation of RatedDoc
2016-07-28 11:41:54 +02:00
Christoph Büscher 71c0d59e95 Add option for maximally allowed rank to Reciprocal Rank metric 2016-07-27 16:24:41 +02:00
Isabel Drost-Fromm ad9f060dc7 Use type information in request
Adds parsing of type and actually using it in TransportRankEvalAction.

Missing: A good idea how to actually test this in isolation...
2016-07-27 15:49:37 +02:00
Isabel Drost-Fromm 2bb5cb83a7 Merge branch 'feature/rank-eval' of github.com:elasticsearch/elasticsearch into feature/rank-eval 2016-07-27 15:46:44 +02:00
Christoph Büscher ad87bacf91 Add Reciprocal Rank query evaluation metric
This adds a second query evaluation metric alongside precision_at. Reciprocal
Rank is defined as 1/rank, where rank is the position of the first relevant
document in the search result. The results are averaged across all queries
across the sample of queries, according to
https://en.wikipedia.org/wiki/Mean_reciprocal_rank
2016-07-27 15:23:05 +02:00
Christoph Büscher 4162582ee8 Adapt to renaming of RestTestCandidate 2016-07-27 11:39:59 +02:00
Christoph Büscher ccf275af7b Adapt to introduction of ESClientYamlSuiteTestCase 2016-07-26 16:10:47 +02:00
Isabel Drost-Fromm c8d3098d3e Move github review comments to TODOs 2016-07-21 15:44:47 +02:00
Christoph Büscher b730494bfc Adding rest layer parsing and response rendering
Adding parsers for the rest request and the various components within, also
extending the existing rest test and adding rendering of the response.
2016-07-06 15:30:45 +02:00
Christoph Büscher b38a12ad15 Adapt to new checkstyle constraints 2016-07-05 11:38:19 +02:00
Christoph Büscher 42662fd695 Adapt to changes in master 2016-07-01 15:03:17 +02:00
Christoph Büscher be19d13d14 Adapting to changes in Plugin api 2016-06-30 16:18:05 +02:00
Isabel Drost-Fromm 6d4673fd58 Initial commit for Module to compute metrics on queries
This is an initial squashed commit of the work on a new feature for query metrics
proposed in #18798.
2016-06-30 16:15:08 +02:00