honeymoose
/
OpenSearch
mirror of https://github.com/honeymoose/OpenSearch.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Added support to the percolate query to percolate multiple documents 

The percolator will add a `_percolator_document_slot` field to all percolator
hits to indicate with what document it has matched. This number matches with
the order in which the documents have been specified in the percolate query.

Also improved the support for multiple percolate queries in a search request.
This commit is contained in:
Martijn van Groningen 2017-09-08 14:20:08 +02:00
parent 4e43aac0f8
commit b391425da1
No known key found for this signature in database
GPG Key ID: AB236F4FCF2AF12A
15 changed files with 1044 additions and 153 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
docs/reference/mapping/types/percolator.asciidoc
View File

@ -277,6 +277,9 @@ now returns matches from the new index:
"body": "quick brown fox"
}
}
},
"fields" : {
"_percolator_document_slot" : [0]
}
}
]
@ -472,6 +475,9 @@ This results in a response like this:
}
}
}
},
"fields" : {
"_percolator_document_slot" : [0]
}
}
]

268
docs/reference/query-dsl/percolate-query.asciidoc
View File

@ -103,6 +103,9 @@ The above request will yield the following response:
"message": "bonsai tree"
}
}
},
"fields" : {
"_percolator_document_slot" : [0] <2>
}
}
]
@ -112,6 +115,8 @@ The above request will yield the following response:
// TESTRESPONSE[s/"took": 13,/"took": "$body.took",/]
<1> The query with id `1` matches our document.
<2> The `_percolator_document_slot` field indicates which document has matched with this query.
Useful when percolating multiple document simultaneously.
[float]
==== Parameters
@ -120,7 +125,10 @@ The following parameters are required when percolating a document:
[horizontal]
`field`:: The field of type `percolator` that holds the indexed queries. This is a required parameter.
`name`:: The suffix to be used for the `_percolator_document_slot` field in case multiple `percolate` queries have been specified.
This is an optional parameter.
`document`:: The source of the document being percolated.
`documents`:: Like the `document` parameter, but accepts multiple documents via a json array.
`document_type`:: The type / mapping of the document being percolated. This setting is deprecated and only required for indices created before 6.0
Instead of specifying the source of the document being percolated, the source can also be retrieved from an already
@ -136,6 +144,87 @@ In that case the `document` parameter can be substituted with the following para
`preference`:: Optionally, preference to be used to fetch document to percolate.
`version`:: Optionally, the expected version of the document to be fetched.
[float]
==== Percolating multiple documents
The `percolate` query can match multiple documents simultaneously with the indexed percolator queries.
Percolating multiple documents in a single request can improve performance as queries only need to be parsed and
matched once instead of multiple times.
The `_percolator_document_slot` field that is being returned with each matched percolator query is important when percolating
multiple documents simultaneously. It indicates which documents matched with a particular percolator query. The numbers
correlate with the slot in the `documents` array specified in the `percolate` query.
[source,js]
--------------------------------------------------
GET /my-index/_search
{
"query" : {
"percolate" : {
"field" : "query",
"documents" : [ <1>
{
"message" : "bonsai tree"
},
{
"message" : "new tree"
},
{
"message" : "the office"
},
{
"message" : "office tree"
}
]
}
}
}
--------------------------------------------------
// CONSOLE
// TEST[continued]
<1> The documents array contains 4 documents that are going to be percolated at the same time.
[source,js]
--------------------------------------------------
{
"took": 13,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped" : 0,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 1.5606477,
"hits": [
{
"_index": "my-index",
"_type": "doc",
"_id": "1",
"_score": 1.5606477,
"_source": {
"query": {
"match": {
"message": "bonsai tree"
}
}
},
"fields" : {
"_percolator_document_slot" : [0, 1, 3] <1>
}
}
]
}
}
--------------------------------------------------
// TESTRESPONSE[s/"took": 13,/"took": "$body.took",/]
<1> The `_percolator_document_slot` indicates that the first, second and last documents specified in the `percolate` query
are matching with this query.
[float]
==== Percolating an Existing Document
@ -307,6 +396,9 @@ This will yield the following response.
"message": [
"The quick brown fox jumps over the <em>lazy</em> <em>dog</em>" <1>
]
},
"fields" : {
"_percolator_document_slot" : [0]
}
},
{
@ -325,6 +417,9 @@ This will yield the following response.
"message": [
"The quick <em>brown</em> <em>fox</em> jumps over the lazy dog" <1>
]
},
"fields" : {
"_percolator_document_slot" : [0]
}
}
]
@ -338,6 +433,179 @@ This will yield the following response.
Instead of the query in the search request highlighting the percolator hits, the percolator queries are highlighting
the document defined in the `percolate` query.
When percolating multiple documents at the same time like the request below then the highlight response is different:
[source,js]
--------------------------------------------------
GET /my-index/_search
{
"query" : {
"percolate" : {
"field": "query",
"documents" : [
{
"message" : "bonsai tree"
},
{
"message" : "new tree"
},
{
"message" : "the office"
},
{
"message" : "office tree"
}
]
}
},
"highlight": {
"fields": {
"message": {}
}
}
}
--------------------------------------------------
// CONSOLE
// TEST[continued]
The slightly different response:
[source,js]
--------------------------------------------------
{
"took": 13,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped" : 0,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 1.5606477,
"hits": [
{
"_index": "my-index",
"_type": "doc",
"_id": "1",
"_score": 1.5606477,
"_source": {
"query": {
"match": {
"message": "bonsai tree"
}
}
},
"fields" : {
"_percolator_document_slot" : [0, 1, 3]
},
"highlight" : { <1>
"0_message" : [
"<em>bonsai</em> <em>tree</em>"
],
"3_message" : [
"office <em>tree</em>"
],
"1_message" : [
"new <em>tree</em>"
]
}
}
]
}
}
--------------------------------------------------
// TESTRESPONSE[s/"took": 13,/"took": "$body.took",/]
<1> The highlight fields have been prefixed with the document slot they belong to,
in order to know which highlight field belongs to what document.
[float]
==== Specifying multiple percolate queries
It is possible to specify multiple `percolate` queries in a single search request:
[source,js]
--------------------------------------------------
GET /my-index/_search
{
"query" : {
"bool" : {
"should" : [
{
"percolate" : {
"field" : "query",
"document" : {
"message" : "bonsai tree"
},
"name": "query1" <1>
}
},
{
"percolate" : {
"field" : "query",
"document" : {
"message" : "tulip flower"
},
"name": "query2" <1>
}
}
]
}
}
}
--------------------------------------------------
// CONSOLE
// TEST[continued]
<1> The `name` parameter will be used to identify which percolator document slots belong to what `percolate` query.
The `_percolator_document_slot` field name will be suffixed with what is specified in the `_name` parameter.
If that isn't specified then the `field` parameter will be used, which in this case will result in ambiguity.
The above search request returns a response similar to this:
[source,js]
--------------------------------------------------
{
"took": 13,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped" : 0,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 0.5753642,
"hits": [
{
"_index": "my-index",
"_type": "doc",
"_id": "1",
"_score": 0.5753642,
"_source": {
"query": {
"match": {
"message": "bonsai tree"
}
}
},
"fields" : {
"_percolator_document_slot_query1" : [0] <1>
}
}
]
}
}
--------------------------------------------------
// TESTRESPONSE[s/"took": 13,/"took": "$body.took",/]
<1> The `_percolator_document_slot_query1` percolator slot field indicates that these matched slots are from the `percolate`
query with `_name` parameter set to `query1`.
[float]
==== How it Works Under the Hood

32
modules/percolator/src/main/java/org/elasticsearch/percolator/PercolateQuery.java
View File

@ -38,6 +38,7 @@ import org.elasticsearch.common.bytes.BytesReference;
import org.elasticsearch.common.lucene.Lucene;
import java.io.IOException;
import java.util.List;
import java.util.Objects;
import java.util.Set;
@ -46,15 +47,17 @@ final class PercolateQuery extends Query implements Accountable {
// cost of matching the query against the document, arbitrary as it would be really complex to estimate
private static final float MATCH_COST = 1000;
private final String name;
private final QueryStore queryStore;
private final BytesReference documentSource;
private final List<BytesReference> documents;
private final Query candidateMatchesQuery;
private final Query verifiedMatchesQuery;
private final IndexSearcher percolatorIndexSearcher;
PercolateQuery(QueryStore queryStore, BytesReference documentSource,
PercolateQuery(String name, QueryStore queryStore, List<BytesReference> documents,
Query candidateMatchesQuery, IndexSearcher percolatorIndexSearcher, Query verifiedMatchesQuery) {
this.documentSource = Objects.requireNonNull(documentSource);
this.name = name;
this.documents = Objects.requireNonNull(documents);
this.candidateMatchesQuery = Objects.requireNonNull(candidateMatchesQuery);
this.queryStore = Objects.requireNonNull(queryStore);
this.percolatorIndexSearcher = Objects.requireNonNull(percolatorIndexSearcher);
@ -65,7 +68,7 @@ final class PercolateQuery extends Query implements Accountable {
public Query rewrite(IndexReader reader) throws IOException {
Query rewritten = candidateMatchesQuery.rewrite(reader);
if (rewritten != candidateMatchesQuery) {
return new PercolateQuery(queryStore, documentSource, rewritten, percolatorIndexSearcher, verifiedMatchesQuery);
return new PercolateQuery(name, queryStore, documents, rewritten, percolatorIndexSearcher, verifiedMatchesQuery);
} else {
return this;
}
@ -164,12 +167,16 @@ final class PercolateQuery extends Query implements Accountable {
};
}
String getName() {
return name;
}
IndexSearcher getPercolatorIndexSearcher() {
return percolatorIndexSearcher;
}
BytesReference getDocumentSource() {
return documentSource;
List<BytesReference> getDocuments() {
return documents;
}
QueryStore getQueryStore() {
@ -193,13 +200,22 @@ final class PercolateQuery extends Query implements Accountable {
@Override
public String toString(String s) {
return "PercolateQuery{document_source={" + documentSource.utf8ToString() + "},inner={" +
StringBuilder sources = new StringBuilder();
for (BytesReference document : documents) {
sources.append(document.utf8ToString());
sources.append('\n');
}
return "PercolateQuery{document_sources={" + sources + "},inner={" +
candidateMatchesQuery.toString(s) + "}}";
}
@Override
public long ramBytesUsed() {
return documentSource.ramBytesUsed();
long ramUsed = 0L;
for (BytesReference document : documents) {
ramUsed += document.ramBytesUsed();
}
return ramUsed;
}
@FunctionalInterface

198
modules/percolator/src/main/java/org/elasticsearch/percolator/PercolateQueryBuilder.java
View File

@ -57,12 +57,13 @@ import org.elasticsearch.common.io.stream.StreamOutput;
import org.elasticsearch.common.logging.DeprecationLogger;
import org.elasticsearch.common.logging.Loggers;
import org.elasticsearch.common.lucene.search.Queries;
import org.elasticsearch.common.xcontent.NamedXContentRegistry;
import org.elasticsearch.common.xcontent.XContent;
import org.elasticsearch.common.xcontent.XContentBuilder;
import org.elasticsearch.common.xcontent.XContentFactory;
import org.elasticsearch.common.xcontent.XContentHelper;
import org.elasticsearch.common.xcontent.XContentParser;
import org.elasticsearch.common.xcontent.XContentType;
import org.elasticsearch.index.IndexSettings;
import org.elasticsearch.index.analysis.FieldNameAnalyzer;
import org.elasticsearch.index.fielddata.IndexFieldData;
import org.elasticsearch.index.fielddata.IndexFieldDataCache;
@ -70,6 +71,7 @@ import org.elasticsearch.index.mapper.DocumentMapper;
import org.elasticsearch.index.mapper.DocumentMapperForType;
import org.elasticsearch.index.mapper.MappedFieldType;
import org.elasticsearch.index.mapper.MapperService;
import org.elasticsearch.index.mapper.ParseContext;
import org.elasticsearch.index.mapper.ParsedDocument;
import org.elasticsearch.index.query.AbstractQueryBuilder;
import org.elasticsearch.index.query.QueryBuilder;
@ -82,7 +84,10 @@ import org.elasticsearch.indices.breaker.NoneCircuitBreakerService;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
import java.util.List;
import java.util.Objects;
import java.util.function.Supplier;
@ -95,6 +100,8 @@ public class PercolateQueryBuilder extends AbstractQueryBuilder<PercolateQueryBu
private static final DeprecationLogger DEPRECATION_LOGGER = new DeprecationLogger(Loggers.getLogger(ParseField.class));
static final ParseField DOCUMENT_FIELD = new ParseField("document");
static final ParseField DOCUMENTS_FIELD = new ParseField("documents");
private static final ParseField NAME_FIELD = new ParseField("name");
private static final ParseField QUERY_FIELD = new ParseField("field");
private static final ParseField DOCUMENT_TYPE_FIELD = new ParseField("document_type");
private static final ParseField INDEXED_DOCUMENT_FIELD_INDEX = new ParseField("index");
@ -105,9 +112,10 @@ public class PercolateQueryBuilder extends AbstractQueryBuilder<PercolateQueryBu
private static final ParseField INDEXED_DOCUMENT_FIELD_VERSION = new ParseField("version");
private final String field;
private String name;
@Deprecated
private final String documentType;
private final BytesReference document;
private final List<BytesReference> documents;
private final XContentType documentXContentType;
private final String indexedDocumentIndex;
@ -124,7 +132,7 @@ public class PercolateQueryBuilder extends AbstractQueryBuilder<PercolateQueryBu
*/
@Deprecated
public PercolateQueryBuilder(String field, String documentType, BytesReference document) {
this(field, documentType, document, XContentFactory.xContentType(document));
this(field, documentType, Collections.singletonList(document), XContentFactory.xContentType(document));
}
/**
@ -135,20 +143,31 @@ public class PercolateQueryBuilder extends AbstractQueryBuilder<PercolateQueryBu
* @param documentXContentType The content type of the binary blob containing the document to percolate
*/
public PercolateQueryBuilder(String field, BytesReference document, XContentType documentXContentType) {
this(field, null, document, documentXContentType);
this(field, null, Collections.singletonList(document), documentXContentType);
}
/**
* Creates a percolator query builder instance for percolating a provided document.
*
* @param field The field that contains the percolator query
* @param documents The binary blob containing document to percolate
* @param documentXContentType The content type of the binary blob containing the document to percolate
*/
public PercolateQueryBuilder(String field, List<BytesReference> documents, XContentType documentXContentType) {
this(field, null, documents, documentXContentType);
}
@Deprecated
public PercolateQueryBuilder(String field, String documentType, BytesReference document, XContentType documentXContentType) {
public PercolateQueryBuilder(String field, String documentType, List<BytesReference> documents, XContentType documentXContentType) {
if (field == null) {
throw new IllegalArgumentException("[field] is a required argument");
}
if (document == null) {
if (documents == null) {
throw new IllegalArgumentException("[document] is a required argument");
}
this.field = field;
this.documentType = documentType;
this.document = document;
this.documents = documents;
this.documentXContentType = Objects.requireNonNull(documentXContentType);
indexedDocumentIndex = null;
indexedDocumentType = null;
@ -165,7 +184,7 @@ public class PercolateQueryBuilder extends AbstractQueryBuilder<PercolateQueryBu
}
this.field = field;
this.documentType = documentType;
this.document = null;
this.documents = Collections.emptyList();
this.documentXContentType = null;
this.documentSupplier = documentSupplier;
indexedDocumentIndex = null;
@ -217,7 +236,7 @@ public class PercolateQueryBuilder extends AbstractQueryBuilder<PercolateQueryBu
this.indexedDocumentRouting = indexedDocumentRouting;
this.indexedDocumentPreference = indexedDocumentPreference;
this.indexedDocumentVersion = indexedDocumentVersion;
this.document = null;
this.documents = Collections.emptyList();
this.documentXContentType = null;
this.documentSupplier = null;
}
@ -228,6 +247,9 @@ public class PercolateQueryBuilder extends AbstractQueryBuilder<PercolateQueryBu
PercolateQueryBuilder(StreamInput in) throws IOException {
super(in);
field = in.readString();
if (in.getVersion().onOrAfter(Version.V_6_1_0)) {
name = in.readOptionalString();
}
if (in.getVersion().before(Version.V_6_0_0_beta1)) {
documentType = in.readString();
} else {
@ -243,12 +265,17 @@ public class PercolateQueryBuilder extends AbstractQueryBuilder<PercolateQueryBu
} else {
indexedDocumentVersion = null;
}
document = in.readOptionalBytesReference();
if (document != null) {
if (in.getVersion().onOrAfter(Version.V_6_1_0)) {
documents = in.readList(StreamInput::readBytesReference);
} else {
BytesReference document = in.readOptionalBytesReference();
documents = document != null ? Collections.singletonList(document) : Collections.emptyList();
}
if (documents.isEmpty() == false) {
if (in.getVersion().onOrAfter(Version.V_5_3_0)) {
documentXContentType = XContentType.readFrom(in);
} else {
documentXContentType = XContentFactory.xContentType(document);
documentXContentType = XContentFactory.xContentType(documents.iterator().next());
}
} else {
documentXContentType = null;
@ -256,12 +283,24 @@ public class PercolateQueryBuilder extends AbstractQueryBuilder<PercolateQueryBu
documentSupplier = null;
}
/**
* Sets the name used for identification purposes in <code>_percolator_document_slot</code> response field
* when multiple percolate queries have been specified in the main query.
*/
public PercolateQueryBuilder setName(String name) {
this.name = name;
return this;
}
@Override
protected void doWriteTo(StreamOutput out) throws IOException {
if (documentSupplier != null) {
throw new IllegalStateException("supplier must be null, can't serialize suppliers, missing a rewriteAndFetch?");
}
out.writeString(field);
if (out.getVersion().onOrAfter(Version.V_6_1_0)) {
out.writeOptionalString(name);
}
if (out.getVersion().before(Version.V_6_0_0_beta1)) {
out.writeString(documentType);
} else {
@ -278,8 +317,19 @@ public class PercolateQueryBuilder extends AbstractQueryBuilder<PercolateQueryBu
} else {
out.writeBoolean(false);
}
out.writeOptionalBytesReference(document);
if (document != null && out.getVersion().onOrAfter(Version.V_5_3_0)) {
if (out.getVersion().onOrAfter(Version.V_6_1_0)) {
out.writeVInt(documents.size());
for (BytesReference document : documents) {
out.writeBytesReference(document);
}
} else {
if (documents.size() > 1) {
throw new IllegalArgumentException("Nodes prior to 6.1.0 cannot accept multiple documents");
}
BytesReference doc = documents.isEmpty() ? null : documents.iterator().next();
out.writeOptionalBytesReference(doc);
}
if (documents.isEmpty() == false && out.getVersion().onOrAfter(Version.V_5_3_0)) {
documentXContentType.writeTo(out);
}
}
@ -289,8 +339,18 @@ public class PercolateQueryBuilder extends AbstractQueryBuilder<PercolateQueryBu
builder.startObject(NAME);
builder.field(DOCUMENT_TYPE_FIELD.getPreferredName(), documentType);
builder.field(QUERY_FIELD.getPreferredName(), field);
if (document != null) {
builder.rawField(DOCUMENT_FIELD.getPreferredName(), document);
if (name != null) {
builder.field(NAME_FIELD.getPreferredName(), name);
}
if (documents.isEmpty() == false) {
builder.startArray(DOCUMENTS_FIELD.getPreferredName());
for (BytesReference document : documents) {
try (XContentParser parser = XContentHelper.createParser(NamedXContentRegistry.EMPTY, document)) {
parser.nextToken();
XContentHelper.copyCurrentStructure(builder.generator(), parser);
}
}
builder.endArray();
}
if (indexedDocumentIndex != null || indexedDocumentType != null || indexedDocumentId != null) {
if (indexedDocumentIndex != null) {
@ -320,6 +380,7 @@ public class PercolateQueryBuilder extends AbstractQueryBuilder<PercolateQueryBu
float boost = AbstractQueryBuilder.DEFAULT_BOOST;
String field = null;
String name = null;
String documentType = null;
String indexedDocumentIndex = null;
@ -329,29 +390,62 @@ public class PercolateQueryBuilder extends AbstractQueryBuilder<PercolateQueryBu
String indexedDocumentPreference = null;
Long indexedDocumentVersion = null;
BytesReference source = null;
List<BytesReference> documents = new ArrayList<>();
String queryName = null;
String currentFieldName = null;
boolean documentsSpecified = false;
boolean documentSpecified = false;
XContentParser.Token token;
while ((token = parser.nextToken()) != XContentParser.Token.END_OBJECT) {
if (token == XContentParser.Token.FIELD_NAME) {
currentFieldName = parser.currentName();
} else if (token == XContentParser.Token.START_OBJECT) {
if (DOCUMENT_FIELD.match(currentFieldName)) {
try (XContentBuilder builder = XContentFactory.jsonBuilder()) {
builder.copyCurrentStructure(parser);
builder.flush();
source = builder.bytes();
} else if (token == XContentParser.Token.START_ARRAY) {
if (DOCUMENTS_FIELD.match(currentFieldName)) {
if (documentSpecified) {
throw new IllegalArgumentException("[" + PercolateQueryBuilder.NAME +
"] Either specified [document] or [documents], not both");
}
documentsSpecified = true;
while ((token = parser.nextToken()) != XContentParser.Token.END_ARRAY) {
if (token == XContentParser.Token.START_OBJECT) {
try (XContentBuilder builder = XContentFactory.jsonBuilder()) {
builder.copyCurrentStructure(parser);
builder.flush();
documents.add(builder.bytes());
}
} else {
throw new ParsingException(parser.getTokenLocation(), "[" + PercolateQueryBuilder.NAME +
"] query does not support [" + token + "]");
}
}
} else {
throw new ParsingException(parser.getTokenLocation(), "[" + PercolateQueryBuilder.NAME +
"] query does not support [" + token + "]");
"] query does not field name [" + currentFieldName + "]");
}
} else if (token == XContentParser.Token.START_OBJECT) {
if (DOCUMENT_FIELD.match(currentFieldName)) {
if (documentsSpecified) {
throw new IllegalArgumentException("[" + PercolateQueryBuilder.NAME +
"] Either specified [document] or [documents], not both");
}
documentSpecified = true;
try (XContentBuilder builder = XContentFactory.jsonBuilder()) {
builder.copyCurrentStructure(parser);
builder.flush();
documents.add(builder.bytes());
}
} else {
throw new ParsingException(parser.getTokenLocation(), "[" + PercolateQueryBuilder.NAME +
"] query does not support field name [" + currentFieldName + "]");
}
} else if (token.isValue() || token == XContentParser.Token.VALUE_NULL) {
if (QUERY_FIELD.match(currentFieldName)) {
field = parser.text();
} else if (NAME_FIELD.match(currentFieldName)) {
name = parser.textOrNull();
} else if (DOCUMENT_TYPE_FIELD.match(currentFieldName)) {
documentType = parser.textOrNull();
} else if (INDEXED_DOCUMENT_FIELD_INDEX.match(currentFieldName)) {
@ -381,14 +475,17 @@ public class PercolateQueryBuilder extends AbstractQueryBuilder<PercolateQueryBu
}
PercolateQueryBuilder queryBuilder;
if (source != null) {
queryBuilder = new PercolateQueryBuilder(field, documentType, source, XContentType.JSON);
if (documents.isEmpty() == false) {
queryBuilder = new PercolateQueryBuilder(field, documentType, documents, XContentType.JSON);
} else if (indexedDocumentId != null) {
queryBuilder = new PercolateQueryBuilder(field, documentType, indexedDocumentIndex, indexedDocumentType,
indexedDocumentId, indexedDocumentRouting, indexedDocumentPreference, indexedDocumentVersion);
} else {
throw new IllegalArgumentException("[" + PercolateQueryBuilder.NAME + "] query, nothing to percolate");
}
if (name != null) {
queryBuilder.setName(name);
}
queryBuilder.queryName(queryName);
queryBuilder.boost(boost);
return queryBuilder;
@ -398,7 +495,7 @@ public class PercolateQueryBuilder extends AbstractQueryBuilder<PercolateQueryBu
protected boolean doEquals(PercolateQueryBuilder other) {
return Objects.equals(field, other.field)
&& Objects.equals(documentType, other.documentType)
&& Objects.equals(document, other.document)
&& Objects.equals(documents, other.documents)
&& Objects.equals(indexedDocumentIndex, other.indexedDocumentIndex)
&& Objects.equals(indexedDocumentType, other.indexedDocumentType)
&& Objects.equals(documentSupplier, other.documentSupplier)
@ -408,7 +505,7 @@ public class PercolateQueryBuilder extends AbstractQueryBuilder<PercolateQueryBu
@Override
protected int doHashCode() {
return Objects.hash(field, documentType, document, indexedDocumentIndex, indexedDocumentType, indexedDocumentId, documentSupplier);
return Objects.hash(field, documentType, documents, indexedDocumentIndex, indexedDocumentType, indexedDocumentId, documentSupplier);
}
@Override
@ -418,14 +515,15 @@ public class PercolateQueryBuilder extends AbstractQueryBuilder<PercolateQueryBu
@Override
protected QueryBuilder doRewrite(QueryRewriteContext queryShardContext) {
if (document != null) {
if (documents.isEmpty() == false) {
return this;
} else if (documentSupplier != null) {
final BytesReference source = documentSupplier.get();
if (source == null) {
return this; // not executed yet
} else {
return new PercolateQueryBuilder(field, documentType, source, XContentFactory.xContentType(source));
return new PercolateQueryBuilder(field, documentType, Collections.singletonList(source),
XContentFactory.xContentType(source));
}
}
GetRequest getRequest = new GetRequest(indexedDocumentIndex, indexedDocumentType, indexedDocumentId);
@ -465,7 +563,7 @@ public class PercolateQueryBuilder extends AbstractQueryBuilder<PercolateQueryBu
throw new IllegalStateException("query builder must be rewritten first");
}
if (document == null) {
if (documents.isEmpty()) {
throw new IllegalStateException("no document to percolate");
}
@ -479,7 +577,7 @@ public class PercolateQueryBuilder extends AbstractQueryBuilder<PercolateQueryBu
"] to be of type [percolator], but is of type [" + fieldType.typeName() + "]");
}
final ParsedDocument doc;
final List<ParsedDocument> docs = new ArrayList<>();
final DocumentMapper docMapper;
final MapperService mapperService = context.getMapperService();
if (context.getIndexSettings().isSingleType()) {
@ -496,14 +594,18 @@ public class PercolateQueryBuilder extends AbstractQueryBuilder<PercolateQueryBu
}
}
docMapper = mapperService.documentMapper(type);
doc = docMapper.parse(source(context.index().getName(), type, "_temp_id", document, documentXContentType));
for (BytesReference document : documents) {
docs.add(docMapper.parse(source(context.index().getName(), type, "_temp_id", document, documentXContentType)));
}
} else {
if (documentType == null) {
throw new IllegalArgumentException("[percolate] query is missing required [document_type] parameter");
}
DocumentMapperForType docMapperForType = mapperService.documentMapperWithAutoCreate(documentType);
docMapper = docMapperForType.getDocumentMapper();
doc = docMapper.parse(source(context.index().getName(), documentType, "_temp_id", document, documentXContentType));
for (BytesReference document : documents) {
docs.add(docMapper.parse(source(context.index().getName(), documentType, "_temp_id", document, documentXContentType)));
}
}
FieldNameAnalyzer fieldNameAnalyzer = (FieldNameAnalyzer) docMapper.mappers().indexAnalyzer();
@ -521,11 +623,11 @@ public class PercolateQueryBuilder extends AbstractQueryBuilder<PercolateQueryBu
}
};
final IndexSearcher docSearcher;
if (doc.docs().size() > 1) {
assert docMapper.hasNestedObjects();
docSearcher = createMultiDocumentSearcher(analyzer, doc);
if (docs.size() > 1 || docs.get(0).docs().size() > 1) {
assert docs.size() != 1 || docMapper.hasNestedObjects();
docSearcher = createMultiDocumentSearcher(analyzer, docs);
} else {
MemoryIndex memoryIndex = MemoryIndex.fromDocument(doc.rootDoc(), analyzer, true, false);
MemoryIndex memoryIndex = MemoryIndex.fromDocument(docs.get(0).rootDoc(), analyzer, true, false);
docSearcher = memoryIndex.createSearcher();
docSearcher.setQueryCache(null);
}
@ -534,9 +636,10 @@ public class PercolateQueryBuilder extends AbstractQueryBuilder<PercolateQueryBu
boolean mapUnmappedFieldsAsString = percolatorFieldMapper.isMapUnmappedFieldAsText();
QueryShardContext percolateShardContext = wrap(context);
String name = this.name != null ? this.name : field;
PercolatorFieldMapper.FieldType pft = (PercolatorFieldMapper.FieldType) fieldType;
PercolateQuery.QueryStore queryStore = createStore(pft.queryBuilderField, percolateShardContext, mapUnmappedFieldsAsString);
return pft.percolateQuery(queryStore, document, docSearcher);
return pft.percolateQuery(name, queryStore, documents, docSearcher);
}
public String getField() {
@ -547,8 +650,8 @@ public class PercolateQueryBuilder extends AbstractQueryBuilder<PercolateQueryBu
return documentType;
}
public BytesReference getDocument() {
return document;
public List<BytesReference> getDocuments() {
return documents;
}
//pkg-private for testing
@ -556,12 +659,17 @@ public class PercolateQueryBuilder extends AbstractQueryBuilder<PercolateQueryBu
return documentXContentType;
}
static IndexSearcher createMultiDocumentSearcher(Analyzer analyzer, ParsedDocument doc) {
static IndexSearcher createMultiDocumentSearcher(Analyzer analyzer, Collection<ParsedDocument> docs) {
RAMDirectory ramDirectory = new RAMDirectory();
try (IndexWriter indexWriter = new IndexWriter(ramDirectory, new IndexWriterConfig(analyzer))) {
indexWriter.addDocuments(doc.docs());
indexWriter.commit();
DirectoryReader directoryReader = DirectoryReader.open(ramDirectory);
// Indexing in order here, so that the user provided order matches with the docid sequencing:
Iterable<ParseContext.Document> iterable = () -> docs.stream()
.map(ParsedDocument::docs)
.flatMap(Collection::stream)
.iterator();
indexWriter.addDocuments(iterable);
DirectoryReader directoryReader = DirectoryReader.open(indexWriter);
assert directoryReader.leaves().size() == 1 : "Expected single leaf, but got [" + directoryReader.leaves().size() + "]";
final IndexSearcher slowSearcher = new IndexSearcher(directoryReader) {

8
modules/percolator/src/main/java/org/elasticsearch/percolator/PercolatorFieldMapper.java
View File

@ -41,7 +41,6 @@ import org.apache.lucene.util.BytesRef;
import org.apache.lucene.util.BytesRefBuilder;
import org.elasticsearch.Version;
import org.elasticsearch.action.support.PlainActionFuture;
import org.elasticsearch.common.Booleans;
import org.elasticsearch.common.ParsingException;
import org.elasticsearch.common.bytes.BytesReference;
import org.elasticsearch.common.hash.MurmurHash3;
@ -55,7 +54,6 @@ import org.elasticsearch.common.xcontent.XContentFactory;
import org.elasticsearch.common.xcontent.XContentLocation;
import org.elasticsearch.common.xcontent.XContentParser;
import org.elasticsearch.common.xcontent.XContentType;
import org.elasticsearch.index.IndexSettings;
import org.elasticsearch.index.mapper.BinaryFieldMapper;
import org.elasticsearch.index.mapper.FieldMapper;
import org.elasticsearch.index.mapper.KeywordFieldMapper;
@ -237,7 +235,7 @@ public class PercolatorFieldMapper extends FieldMapper {
throw new QueryShardException(context, "Percolator fields are not searchable directly, use a percolate query instead");
}
Query percolateQuery(PercolateQuery.QueryStore queryStore, BytesReference documentSource,
Query percolateQuery(String name, PercolateQuery.QueryStore queryStore, List<BytesReference> documents,
IndexSearcher searcher) throws IOException {
IndexReader indexReader = searcher.getIndexReader();
Query candidateMatchesQuery = createCandidateQuery(indexReader);
@ -249,9 +247,9 @@ public class PercolatorFieldMapper extends FieldMapper {
if (indexReader.maxDoc() == 1) {
verifiedMatchesQuery = new TermQuery(new Term(extractionResultField.name(), EXTRACTION_COMPLETE));
} else {
verifiedMatchesQuery = new MatchNoDocsQuery("nested docs, so no verified matches");
verifiedMatchesQuery = new MatchNoDocsQuery("multiple/nested docs, so no verified matches");
}
return new PercolateQuery(queryStore, documentSource, candidateMatchesQuery, searcher, verifiedMatchesQuery);
return new PercolateQuery(name, queryStore, documents, candidateMatchesQuery, searcher, verifiedMatchesQuery);
}
Query createCandidateQuery(IndexReader indexReader) throws IOException {

116
modules/percolator/src/main/java/org/elasticsearch/percolator/PercolatorHighlightSubFetchPhase.java
View File

@ -29,12 +29,14 @@ import org.apache.lucene.search.DisjunctionMaxQuery;
import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.search.Query;
import org.elasticsearch.common.bytes.BytesReference;
import org.elasticsearch.common.document.DocumentField;
import org.elasticsearch.common.lucene.search.function.FunctionScoreQuery;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.common.text.Text;
import org.elasticsearch.index.query.ParsedQuery;
import org.elasticsearch.search.SearchHit;
import org.elasticsearch.search.fetch.FetchSubPhase;
import org.elasticsearch.search.fetch.subphase.highlight.HighlightField;
import org.elasticsearch.search.fetch.subphase.highlight.HighlightPhase;
import org.elasticsearch.search.fetch.subphase.highlight.Highlighter;
import org.elasticsearch.search.fetch.subphase.highlight.SearchContextHighlight;
@ -42,6 +44,7 @@ import org.elasticsearch.search.internal.SearchContext;
import org.elasticsearch.search.internal.SubSearchContext;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.Map;
@ -58,68 +61,103 @@ final class PercolatorHighlightSubFetchPhase extends HighlightPhase {
boolean hitsExecutionNeeded(SearchContext context) { // for testing
return context.highlight() != null && locatePercolatorQuery(context.query()) != null;
return context.highlight() != null && locatePercolatorQuery(context.query()).isEmpty() == false;
}
@Override
public void hitsExecute(SearchContext context, SearchHit[] hits) {
public void hitsExecute(SearchContext context, SearchHit[] hits) throws IOException {
if (hitsExecutionNeeded(context) == false) {
return;
}
PercolateQuery percolateQuery = locatePercolatorQuery(context.query());
if (percolateQuery == null) {
List<PercolateQuery> percolateQueries = locatePercolatorQuery(context.query());
if (percolateQueries.isEmpty()) {
// shouldn't happen as we checked for the existence of a percolator query in hitsExecutionNeeded(...)
throw new IllegalStateException("couldn't locate percolator query");
}
List<LeafReaderContext> ctxs = context.searcher().getIndexReader().leaves();
IndexSearcher percolatorIndexSearcher = percolateQuery.getPercolatorIndexSearcher();
PercolateQuery.QueryStore queryStore = percolateQuery.getQueryStore();
boolean singlePercolateQuery = percolateQueries.size() == 1;
for (PercolateQuery percolateQuery : percolateQueries) {
String fieldName = singlePercolateQuery ? PercolatorMatchedSlotSubFetchPhase.FIELD_NAME_PREFIX :
PercolatorMatchedSlotSubFetchPhase.FIELD_NAME_PREFIX + "_" + percolateQuery.getName();
List<LeafReaderContext> ctxs = context.searcher().getIndexReader().leaves();
IndexSearcher percolatorIndexSearcher = percolateQuery.getPercolatorIndexSearcher();
PercolateQuery.QueryStore queryStore = percolateQuery.getQueryStore();
LeafReaderContext percolatorLeafReaderContext = percolatorIndexSearcher.getIndexReader().leaves().get(0);
FetchSubPhase.HitContext hitContext = new FetchSubPhase.HitContext();
SubSearchContext subSearchContext =
createSubSearchContext(context, percolatorLeafReaderContext, percolateQuery.getDocumentSource());
LeafReaderContext percolatorLeafReaderContext = percolatorIndexSearcher.getIndexReader().leaves().get(0);
FetchSubPhase.HitContext hitContext = new FetchSubPhase.HitContext();
for (SearchHit hit : hits) {
final Query query;
try {
for (SearchHit hit : hits) {
LeafReaderContext ctx = ctxs.get(ReaderUtil.subIndex(hit.docId(), ctxs));
int segmentDocId = hit.docId() - ctx.docBase;
query = queryStore.getQueries(ctx).apply(segmentDocId);
} catch (IOException e) {
throw new RuntimeException(e);
}
if (query != null) {
subSearchContext.parsedQuery(new ParsedQuery(query));
hitContext.reset(
new SearchHit(0, "unknown", new Text(hit.getType()), Collections.emptyMap()),
percolatorLeafReaderContext, 0, percolatorIndexSearcher
);
hitContext.cache().clear();
super.hitExecute(subSearchContext, hitContext);
hit.getHighlightFields().putAll(hitContext.hit().getHighlightFields());
final Query query = queryStore.getQueries(ctx).apply(segmentDocId);
if (query != null) {
DocumentField field = hit.field(fieldName);
if (field == null) {
// It possible that a hit did not match with a particular percolate query,
// so then continue highlighting with the next hit.
continue;
}
for (Object matchedSlot : field.getValues()) {
int slot = (int) matchedSlot;
BytesReference document = percolateQuery.getDocuments().get(slot);
SubSearchContext subSearchContext =
createSubSearchContext(context, percolatorLeafReaderContext, document, slot);
subSearchContext.parsedQuery(new ParsedQuery(query));
hitContext.reset(
new SearchHit(slot, "unknown", new Text(hit.getType()), Collections.emptyMap()),
percolatorLeafReaderContext, slot, percolatorIndexSearcher
);
hitContext.cache().clear();
super.hitExecute(subSearchContext, hitContext);
for (Map.Entry<String, HighlightField> entry : hitContext.hit().getHighlightFields().entrySet()) {
if (percolateQuery.getDocuments().size() == 1) {
String hlFieldName;
if (singlePercolateQuery) {
hlFieldName = entry.getKey();
} else {
hlFieldName = percolateQuery.getName() + "_" + entry.getKey();
}
hit.getHighlightFields().put(hlFieldName, new HighlightField(hlFieldName, entry.getValue().fragments()));
} else {
// In case multiple documents are being percolated we need to identify to which document
// a highlight belongs to.
String hlFieldName;
if (singlePercolateQuery) {
hlFieldName = slot + "_" + entry.getKey();
} else {
hlFieldName = percolateQuery.getName() + "_" + slot + "_" + entry.getKey();
}
hit.getHighlightFields().put(hlFieldName, new HighlightField(hlFieldName, entry.getValue().fragments()));
}
}
}
}
}
}
}
static PercolateQuery locatePercolatorQuery(Query query) {
static List<PercolateQuery> locatePercolatorQuery(Query query) {
if (query instanceof PercolateQuery) {
return (PercolateQuery) query;
return Collections.singletonList((PercolateQuery) query);
} else if (query instanceof BooleanQuery) {
List<PercolateQuery> percolateQueries = new ArrayList<>();
for (BooleanClause clause : ((BooleanQuery) query).clauses()) {
PercolateQuery result = locatePercolatorQuery(clause.getQuery());
if (result != null) {
return result;
List<PercolateQuery> result = locatePercolatorQuery(clause.getQuery());
if (result.isEmpty() == false) {
percolateQueries.addAll(result);
}
}
return percolateQueries;
} else if (query instanceof DisjunctionMaxQuery) {
List<PercolateQuery> percolateQueries = new ArrayList<>();
for (Query disjunct : ((DisjunctionMaxQuery) query).getDisjuncts()) {
PercolateQuery result = locatePercolatorQuery(disjunct);
if (result != null) {
return result;
List<PercolateQuery> result = locatePercolatorQuery(disjunct);
if (result.isEmpty() == false) {
percolateQueries.addAll(result);
}
}
return percolateQueries;
} else if (query instanceof ConstantScoreQuery) {
return locatePercolatorQuery(((ConstantScoreQuery) query).getQuery());
} else if (query instanceof BoostQuery) {
@ -127,16 +165,16 @@ final class PercolatorHighlightSubFetchPhase extends HighlightPhase {
} else if (query instanceof FunctionScoreQuery) {
return locatePercolatorQuery(((FunctionScoreQuery) query).getSubQuery());
}
return null;
return Collections.emptyList();
}
private SubSearchContext createSubSearchContext(SearchContext context, LeafReaderContext leafReaderContext, BytesReference source) {
private SubSearchContext createSubSearchContext(SearchContext context, LeafReaderContext leafReaderContext,
BytesReference source, int docId) {
SubSearchContext subSearchContext = new SubSearchContext(context);
subSearchContext.highlight(new SearchContextHighlight(context.highlight().fields()));
// Enforce highlighting by source, because MemoryIndex doesn't support stored fields.
subSearchContext.highlight().globalForceSource(true);
subSearchContext.lookup().source().setSegmentAndDocument(leafReaderContext, 0);
subSearchContext.lookup().source().setSegmentAndDocument(leafReaderContext, docId);
subSearchContext.lookup().source().setSource(source);
return subSearchContext;
}

121
modules/percolator/src/main/java/org/elasticsearch/percolator/PercolatorMatchedSlotSubFetchPhase.java Normal file
View File

@ -0,0 +1,121 @@
/*
* Licensed to Elasticsearch under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
package org.elasticsearch.percolator;
import org.apache.lucene.index.LeafReaderContext;
import org.apache.lucene.index.ReaderUtil;
import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.search.Query;
import org.apache.lucene.search.Scorer;
import org.apache.lucene.search.Sort;
import org.apache.lucene.search.SortField;
import org.apache.lucene.search.TopDocs;
import org.apache.lucene.search.Weight;
import org.apache.lucene.util.BitSet;
import org.apache.lucene.util.BitSetIterator;
import org.elasticsearch.common.document.DocumentField;
import org.elasticsearch.common.lucene.search.Queries;
import org.elasticsearch.search.SearchHit;
import org.elasticsearch.search.fetch.FetchSubPhase;
import org.elasticsearch.search.internal.SearchContext;
import java.io.IOException;
import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;
import java.util.stream.IntStream;
import static org.apache.lucene.search.DocIdSetIterator.NO_MORE_DOCS;
import static org.elasticsearch.percolator.PercolatorHighlightSubFetchPhase.locatePercolatorQuery;
/**
* Adds a special field to the a percolator query hit to indicate which documents matched with the percolator query.
* This is useful when multiple documents are being percolated in a single request.
*/
final class PercolatorMatchedSlotSubFetchPhase implements FetchSubPhase {
static final String FIELD_NAME_PREFIX = "_percolator_document_slot";
@Override
public void hitsExecute(SearchContext context, SearchHit[] hits) throws IOException {
List<PercolateQuery> percolateQueries = locatePercolatorQuery(context.query());
if (percolateQueries.isEmpty()) {
return;
}
boolean singlePercolateQuery = percolateQueries.size() == 1;
for (PercolateQuery percolateQuery : percolateQueries) {
String fieldName = singlePercolateQuery ? FIELD_NAME_PREFIX : FIELD_NAME_PREFIX + "_" + percolateQuery.getName();
IndexSearcher percolatorIndexSearcher = percolateQuery.getPercolatorIndexSearcher();
Weight weight = percolatorIndexSearcher.createNormalizedWeight(Queries.newNonNestedFilter(), false);
Scorer s = weight.scorer(percolatorIndexSearcher.getIndexReader().leaves().get(0));
int memoryIndexMaxDoc = percolatorIndexSearcher.getIndexReader().maxDoc();
BitSet rootDocs = BitSet.of(s.iterator(), memoryIndexMaxDoc);
int[] rootDocsBySlot = null;
boolean hasNestedDocs = rootDocs.cardinality() != percolatorIndexSearcher.getIndexReader().numDocs();
if (hasNestedDocs) {
rootDocsBySlot = buildRootDocsSlots(rootDocs);
}
PercolateQuery.QueryStore queryStore = percolateQuery.getQueryStore();
List<LeafReaderContext> ctxs = context.searcher().getIndexReader().leaves();
for (SearchHit hit : hits) {
LeafReaderContext ctx = ctxs.get(ReaderUtil.subIndex(hit.docId(), ctxs));
int segmentDocId = hit.docId() - ctx.docBase;
Query query = queryStore.getQueries(ctx).apply(segmentDocId);
TopDocs topDocs = percolatorIndexSearcher.search(query, memoryIndexMaxDoc, new Sort(SortField.FIELD_DOC));
if (topDocs.totalHits == 0) {
// This hit didn't match with a percolate query,
// likely to happen when percolating multiple documents
continue;
}
Map<String, DocumentField> fields = hit.fieldsOrNull();
if (fields == null) {
fields = new HashMap<>();
hit.fields(fields);
}
IntStream slots = convertTopDocsToSlots(topDocs, rootDocsBySlot);
fields.put(fieldName, new DocumentField(fieldName, slots.boxed().collect(Collectors.toList())));
}
}
}
static IntStream convertTopDocsToSlots(TopDocs topDocs, int[] rootDocsBySlot) {
IntStream stream = Arrays.stream(topDocs.scoreDocs)
.mapToInt(scoreDoc -> scoreDoc.doc);
if (rootDocsBySlot != null) {
stream = stream.map(docId -> Arrays.binarySearch(rootDocsBySlot, docId));
}
return stream;
}
static int[] buildRootDocsSlots(BitSet rootDocs) {
int slot = 0;
int[] rootDocsBySlot = new int[rootDocs.cardinality()];
BitSetIterator iterator = new BitSetIterator(rootDocs, 0);
for (int rootDocId = iterator.nextDoc(); rootDocId != NO_MORE_DOCS; rootDocId = iterator.nextDoc()) {
rootDocsBySlot[slot++] = rootDocId;
}
return rootDocsBySlot;
}
}

9
modules/percolator/src/main/java/org/elasticsearch/percolator/PercolatorPlugin.java
View File

@ -28,11 +28,11 @@ import org.elasticsearch.plugins.SearchPlugin;
import org.elasticsearch.search.fetch.FetchSubPhase;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
import java.util.Map;
import static java.util.Collections.singletonList;
import static java.util.Collections.singletonMap;
public class PercolatorPlugin extends Plugin implements MapperPlugin, SearchPlugin {
@ -49,7 +49,10 @@ public class PercolatorPlugin extends Plugin implements MapperPlugin, SearchPlug
@Override
public List<FetchSubPhase> getFetchSubPhases(FetchPhaseConstructionContext context) {
return singletonList(new PercolatorHighlightSubFetchPhase(settings, context.getHighlighters()));
return Arrays.asList(
new PercolatorMatchedSlotSubFetchPhase(),
new PercolatorHighlightSubFetchPhase(settings, context.getHighlighters())
);
}
@Override
@ -60,7 +63,7 @@ public class PercolatorPlugin extends Plugin implements MapperPlugin, SearchPlug
@Override
public Map<String, Mapper.TypeParser> getMappers() {
return Collections.singletonMap(PercolatorFieldMapper.CONTENT_TYPE, new PercolatorFieldMapper.TypeParser());
return singletonMap(PercolatorFieldMapper.CONTENT_TYPE, new PercolatorFieldMapper.TypeParser());
}
}

18
modules/percolator/src/test/java/org/elasticsearch/percolator/CandidateQueryTests.java
View File

@ -309,7 +309,7 @@ public class CandidateQueryTests extends ESSingleNodeTestCase {
MemoryIndex memoryIndex = MemoryIndex.fromDocument(Collections.singleton(new IntPoint("int_field", 3)), new WhitespaceAnalyzer());
IndexSearcher percolateSearcher = memoryIndex.createSearcher();
Query query = fieldType.percolateQuery(queryStore, new BytesArray("{}"), percolateSearcher);
Query query = fieldType.percolateQuery("_name", queryStore, Collections.singletonList(new BytesArray("{}")), percolateSearcher);
TopDocs topDocs = shardSearcher.search(query, 1);
assertEquals(1L, topDocs.totalHits);
assertEquals(1, topDocs.scoreDocs.length);
@ -317,7 +317,7 @@ public class CandidateQueryTests extends ESSingleNodeTestCase {
memoryIndex = MemoryIndex.fromDocument(Collections.singleton(new LongPoint("long_field", 7L)), new WhitespaceAnalyzer());
percolateSearcher = memoryIndex.createSearcher();
query = fieldType.percolateQuery(queryStore, new BytesArray("{}"), percolateSearcher);
query = fieldType.percolateQuery("_name", queryStore, Collections.singletonList(new BytesArray("{}")), percolateSearcher);
topDocs = shardSearcher.search(query, 1);
assertEquals(1L, topDocs.totalHits);
assertEquals(1, topDocs.scoreDocs.length);
@ -326,7 +326,7 @@ public class CandidateQueryTests extends ESSingleNodeTestCase {
memoryIndex = MemoryIndex.fromDocument(Collections.singleton(new HalfFloatPoint("half_float_field", 12)),
new WhitespaceAnalyzer());
percolateSearcher = memoryIndex.createSearcher();
query = fieldType.percolateQuery(queryStore, new BytesArray("{}"), percolateSearcher);
query = fieldType.percolateQuery("_name", queryStore, Collections.singletonList(new BytesArray("{}")), percolateSearcher);
topDocs = shardSearcher.search(query, 1);
assertEquals(1L, topDocs.totalHits);
assertEquals(1, topDocs.scoreDocs.length);
@ -334,7 +334,7 @@ public class CandidateQueryTests extends ESSingleNodeTestCase {
memoryIndex = MemoryIndex.fromDocument(Collections.singleton(new FloatPoint("float_field", 17)), new WhitespaceAnalyzer());
percolateSearcher = memoryIndex.createSearcher();
query = fieldType.percolateQuery(queryStore, new BytesArray("{}"), percolateSearcher);
query = fieldType.percolateQuery("_name", queryStore, Collections.singletonList(new BytesArray("{}")), percolateSearcher);
topDocs = shardSearcher.search(query, 1);
assertEquals(1, topDocs.totalHits);
assertEquals(1, topDocs.scoreDocs.length);
@ -342,7 +342,7 @@ public class CandidateQueryTests extends ESSingleNodeTestCase {
memoryIndex = MemoryIndex.fromDocument(Collections.singleton(new DoublePoint("double_field", 21)), new WhitespaceAnalyzer());
percolateSearcher = memoryIndex.createSearcher();
query = fieldType.percolateQuery(queryStore, new BytesArray("{}"), percolateSearcher);
query = fieldType.percolateQuery("_name", queryStore, Collections.singletonList(new BytesArray("{}")), percolateSearcher);
topDocs = shardSearcher.search(query, 1);
assertEquals(1, topDocs.totalHits);
assertEquals(1, topDocs.scoreDocs.length);
@ -351,7 +351,7 @@ public class CandidateQueryTests extends ESSingleNodeTestCase {
memoryIndex = MemoryIndex.fromDocument(Collections.singleton(new InetAddressPoint("ip_field",
forString("192.168.0.4"))), new WhitespaceAnalyzer());
percolateSearcher = memoryIndex.createSearcher();
query = fieldType.percolateQuery(queryStore, new BytesArray("{}"), percolateSearcher);
query = fieldType.percolateQuery("_name", queryStore, Collections.singletonList(new BytesArray("{}")), percolateSearcher);
topDocs = shardSearcher.search(query, 1);
assertEquals(1, topDocs.totalHits);
assertEquals(1, topDocs.scoreDocs.length);
@ -464,7 +464,8 @@ public class CandidateQueryTests extends ESSingleNodeTestCase {
private void duelRun(PercolateQuery.QueryStore queryStore, MemoryIndex memoryIndex, IndexSearcher shardSearcher) throws IOException {
boolean requireScore = randomBoolean();
IndexSearcher percolateSearcher = memoryIndex.createSearcher();
Query percolateQuery = fieldType.percolateQuery(queryStore, new BytesArray("{}"), percolateSearcher);
Query percolateQuery = fieldType.percolateQuery("_name", queryStore,
Collections.singletonList(new BytesArray("{}")), percolateSearcher);
Query query = requireScore ? percolateQuery : new ConstantScoreQuery(percolateQuery);
TopDocs topDocs = shardSearcher.search(query, 10);
@ -497,7 +498,8 @@ public class CandidateQueryTests extends ESSingleNodeTestCase {
MemoryIndex memoryIndex,
IndexSearcher shardSearcher) throws IOException {
IndexSearcher percolateSearcher = memoryIndex.createSearcher();
Query percolateQuery = fieldType.percolateQuery(queryStore, new BytesArray("{}"), percolateSearcher);
Query percolateQuery = fieldType.percolateQuery("_name", queryStore,
Collections.singletonList(new BytesArray("{}")), percolateSearcher);
return shardSearcher.search(percolateQuery, 10);
}

120
modules/percolator/src/test/java/org/elasticsearch/percolator/PercolateQueryBuilderTests.java
View File

@ -35,6 +35,7 @@ import org.elasticsearch.common.bytes.BytesArray;
import org.elasticsearch.common.bytes.BytesReference;
import org.elasticsearch.common.compress.CompressedXContent;
import org.elasticsearch.common.io.stream.BytesStreamOutput;
import org.elasticsearch.common.io.stream.StreamInput;
import org.elasticsearch.common.xcontent.XContentBuilder;
import org.elasticsearch.common.xcontent.XContentFactory;
import org.elasticsearch.common.xcontent.XContentType;
@ -53,9 +54,13 @@ import org.hamcrest.Matchers;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Base64;
import java.util.Collection;
import java.util.Collections;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
import static org.hamcrest.Matchers.equalTo;
@ -63,7 +68,10 @@ import static org.hamcrest.Matchers.sameInstance;
public class PercolateQueryBuilderTests extends AbstractQueryTestCase<PercolateQueryBuilder> {
private static final String[] SHUFFLE_PROTECTED_FIELDS = new String[] { PercolateQueryBuilder.DOCUMENT_FIELD.getPreferredName()};
private static final String[] SHUFFLE_PROTECTED_FIELDS = new String[] {
PercolateQueryBuilder.DOCUMENT_FIELD.getPreferredName(),
PercolateQueryBuilder.DOCUMENTS_FIELD.getPreferredName()
};
private static String queryField;
private static String docType;
@ -74,7 +82,7 @@ public class PercolateQueryBuilderTests extends AbstractQueryTestCase<PercolateQ
private String indexedDocumentRouting;
private String indexedDocumentPreference;
private Long indexedDocumentVersion;
private BytesReference documentSource;
private List<BytesReference> documentSource;
private boolean indexedDocumentExists = true;
@ -104,7 +112,18 @@ public class PercolateQueryBuilderTests extends AbstractQueryTestCase<PercolateQ
}
private PercolateQueryBuilder doCreateTestQueryBuilder(boolean indexedDocument) {
documentSource = randomSource();
if (indexedDocument) {
documentSource = Collections.singletonList(randomSource(new HashSet<>()));
} else {
int numDocs = randomIntBetween(1, 8);
documentSource = new ArrayList<>(numDocs);
Set<String> usedFields = new HashSet<>();
for (int i = 0; i < numDocs; i++) {
documentSource.add(randomSource(usedFields));
}
}
PercolateQueryBuilder queryBuilder;
if (indexedDocument) {
indexedDocumentIndex = randomAlphaOfLength(4);
indexedDocumentType = "doc";
@ -112,11 +131,15 @@ public class PercolateQueryBuilderTests extends AbstractQueryTestCase<PercolateQ
indexedDocumentRouting = randomAlphaOfLength(4);
indexedDocumentPreference = randomAlphaOfLength(4);
indexedDocumentVersion = (long) randomIntBetween(0, Integer.MAX_VALUE);
return new PercolateQueryBuilder(queryField, docType, indexedDocumentIndex, indexedDocumentType, indexedDocumentId,
queryBuilder = new PercolateQueryBuilder(queryField, docType, indexedDocumentIndex, indexedDocumentType, indexedDocumentId,
indexedDocumentRouting, indexedDocumentPreference, indexedDocumentVersion);
} else {
return new PercolateQueryBuilder(queryField, docType, documentSource, XContentType.JSON);
queryBuilder = new PercolateQueryBuilder(queryField, docType, documentSource, XContentType.JSON);
}
if (randomBoolean()) {
queryBuilder.setName(randomAlphaOfLength(4));
}
return queryBuilder;
}
/**
@ -139,8 +162,8 @@ public class PercolateQueryBuilderTests extends AbstractQueryTestCase<PercolateQ
assertThat(getRequest.version(), Matchers.equalTo(indexedDocumentVersion));
if (indexedDocumentExists) {
return new GetResponse(
new GetResult(indexedDocumentIndex, indexedDocumentType, indexedDocumentId, 0L, true, documentSource,
Collections.emptyMap())
new GetResult(indexedDocumentIndex, indexedDocumentType, indexedDocumentId, 0L, true,
documentSource.iterator().next(), Collections.emptyMap())
);
} else {
return new GetResponse(
@ -154,7 +177,7 @@ public class PercolateQueryBuilderTests extends AbstractQueryTestCase<PercolateQ
assertThat(query, Matchers.instanceOf(PercolateQuery.class));
PercolateQuery percolateQuery = (PercolateQuery) query;
assertThat(docType, Matchers.equalTo(queryBuilder.getDocumentType()));
assertThat(percolateQuery.getDocumentSource(), Matchers.equalTo(documentSource));
assertThat(percolateQuery.getDocuments(), Matchers.equalTo(documentSource));
}
@Override
@ -181,12 +204,13 @@ public class PercolateQueryBuilderTests extends AbstractQueryTestCase<PercolateQ
@Override
protected Set<String> getObjectsHoldingArbitraryContent() {
//document contains arbitrary content, no error expected when an object is added to it
return Collections.singleton(PercolateQueryBuilder.DOCUMENT_FIELD.getPreferredName());
return new HashSet<>(Arrays.asList(PercolateQueryBuilder.DOCUMENT_FIELD.getPreferredName(),
PercolateQueryBuilder.DOCUMENTS_FIELD.getPreferredName()));
}
public void testRequiredParameters() {
IllegalArgumentException e = expectThrows(IllegalArgumentException.class, () -> {
new PercolateQueryBuilder(null, null, new BytesArray("{}"), XContentType.JSON);
new PercolateQueryBuilder(null, new BytesArray("{}"), XContentType.JSON);
});
assertThat(e.getMessage(), equalTo("[field] is a required argument"));
@ -227,16 +251,42 @@ public class PercolateQueryBuilderTests extends AbstractQueryTestCase<PercolateQ
}
}
public void testCreateMultiDocumentSearcher() throws Exception {
int numDocs = randomIntBetween(2, 8);
List<ParseContext.Document> docs = new ArrayList<>(numDocs);
for (int i = 0; i < numDocs; i++) {
public void testBothDocumentAndDocumentsSpecified() throws IOException {
expectThrows(IllegalArgumentException.class,
() -> parseQuery("{\"percolate\" : { \"document\": {}, \"documents\": [{}, {}], \"field\":\"" + queryField + "\"}}"));
}
public void testCreateNestedDocumentSearcher() throws Exception {
int numNestedDocs = randomIntBetween(2, 8);
List<ParseContext.Document> docs = new ArrayList<>(numNestedDocs);
for (int i = 0; i < numNestedDocs; i++) {
docs.add(new ParseContext.Document());
}
Collection<ParsedDocument> parsedDocument = Collections.singleton(
new ParsedDocument(null, null, "_id", "_type", null, docs, null, null, null));
Analyzer analyzer = new WhitespaceAnalyzer();
ParsedDocument parsedDocument = new ParsedDocument(null, null, "_id", "_type", null, docs, null, null, null);
IndexSearcher indexSearcher = PercolateQueryBuilder.createMultiDocumentSearcher(analyzer, parsedDocument);
assertThat(indexSearcher.getIndexReader().numDocs(), equalTo(numNestedDocs));
// ensure that any query get modified so that the nested docs are never included as hits:
Query query = new MatchAllDocsQuery();
BooleanQuery result = (BooleanQuery) indexSearcher.createNormalizedWeight(query, true).getQuery();
assertThat(result.clauses().size(), equalTo(2));
assertThat(result.clauses().get(0).getQuery(), sameInstance(query));
assertThat(result.clauses().get(0).getOccur(), equalTo(BooleanClause.Occur.MUST));
assertThat(result.clauses().get(1).getOccur(), equalTo(BooleanClause.Occur.MUST_NOT));
}
public void testCreateMultiDocumentSearcher() throws Exception {
int numDocs = randomIntBetween(2, 8);
List<ParsedDocument> docs = new ArrayList<>();
for (int i = 0; i < numDocs; i++) {
docs.add(new ParsedDocument(null, null, "_id", "_type", null,
Collections.singletonList(new ParseContext.Document()), null, null, null));
}
Analyzer analyzer = new WhitespaceAnalyzer();
IndexSearcher indexSearcher = PercolateQueryBuilder.createMultiDocumentSearcher(analyzer, docs);
assertThat(indexSearcher.getIndexReader().numDocs(), equalTo(numDocs));
// ensure that any query get modified so that the nested docs are never included as hits:
@ -248,10 +298,46 @@ public class PercolateQueryBuilderTests extends AbstractQueryTestCase<PercolateQ
assertThat(result.clauses().get(1).getOccur(), equalTo(BooleanClause.Occur.MUST_NOT));
}
private static BytesReference randomSource() {
public void testSerializationBwc() throws IOException {
final byte[] data = Base64.getDecoder().decode("P4AAAAAFZmllbGQEdHlwZQAAAAAAAA57ImZvbyI6ImJhciJ9AAAAAA==");
final Version version = randomFrom(Version.V_5_0_0, Version.V_5_0_1, Version.V_5_0_2,
Version.V_5_1_1, Version.V_5_1_2, Version.V_5_2_0);
try (StreamInput in = StreamInput.wrap(data)) {
in.setVersion(version);
PercolateQueryBuilder queryBuilder = new PercolateQueryBuilder(in);
assertEquals("type", queryBuilder.getDocumentType());
assertEquals("field", queryBuilder.getField());
assertEquals("{\"foo\":\"bar\"}", queryBuilder.getDocuments().iterator().next().utf8ToString());
assertEquals(XContentType.JSON, queryBuilder.getXContentType());
try (BytesStreamOutput out = new BytesStreamOutput()) {
out.setVersion(version);
queryBuilder.writeTo(out);
assertArrayEquals(data, out.bytes().toBytesRef().bytes);
}
}
}
private static BytesReference randomSource(Set<String> usedFields) {
try {
// If we create two source that have the same field, but these fields have different kind of values (str vs. lng) then
// when these source get indexed, indexing can fail. To solve this test issue, we should generate source that
// always have unique fields:
Map<String, ?> source;
boolean duplicateField;
do {
duplicateField = false;
source = RandomDocumentPicks.randomSource(random());
for (String field : source.keySet()) {
if (usedFields.add(field) == false) {
duplicateField = true;
break;
}
}
} while (duplicateField);
XContentBuilder xContent = XContentFactory.jsonBuilder();
xContent.map(RandomDocumentPicks.randomSource(random()));
xContent.map(source);
return xContent.bytes();
} catch (IOException e) {
throw new RuntimeException(e);

10
modules/percolator/src/test/java/org/elasticsearch/percolator/PercolateQueryTests.java
View File

@ -116,7 +116,7 @@ public class PercolateQueryTests extends ESTestCase {
memoryIndex.addField("field", "the quick brown fox jumps over the lazy dog", new WhitespaceAnalyzer());
IndexSearcher percolateSearcher = memoryIndex.createSearcher();
// no scoring, wrapping it in a constant score query:
Query query = new ConstantScoreQuery(new PercolateQuery(queryStore, new BytesArray("a"),
Query query = new ConstantScoreQuery(new PercolateQuery("_name", queryStore, Collections.singletonList(new BytesArray("a")),
new TermQuery(new Term("select", "a")), percolateSearcher, new MatchNoDocsQuery("")));
TopDocs topDocs = shardSearcher.search(query, 10);
assertThat(topDocs.totalHits, equalTo(1L));
@ -126,7 +126,7 @@ public class PercolateQueryTests extends ESTestCase {
assertThat(explanation.isMatch(), is(true));
assertThat(explanation.getValue(), equalTo(topDocs.scoreDocs[0].score));
query = new ConstantScoreQuery(new PercolateQuery(queryStore, new BytesArray("b"),
query = new ConstantScoreQuery(new PercolateQuery("_name", queryStore, Collections.singletonList(new BytesArray("b")),
new TermQuery(new Term("select", "b")), percolateSearcher, new MatchNoDocsQuery("")));
topDocs = shardSearcher.search(query, 10);
assertThat(topDocs.totalHits, equalTo(3L));
@ -146,13 +146,13 @@ public class PercolateQueryTests extends ESTestCase {
assertThat(explanation.isMatch(), is(true));
assertThat(explanation.getValue(), equalTo(topDocs.scoreDocs[2].score));
query = new ConstantScoreQuery(new PercolateQuery(queryStore, new BytesArray("c"),
query = new ConstantScoreQuery(new PercolateQuery("_name", queryStore, Collections.singletonList(new BytesArray("c")),
new MatchAllDocsQuery(), percolateSearcher, new MatchAllDocsQuery()));
topDocs = shardSearcher.search(query, 10);
assertThat(topDocs.totalHits, equalTo(4L));
query = new PercolateQuery(queryStore, new BytesArray("{}"), new TermQuery(new Term("select", "b")),
percolateSearcher, new MatchNoDocsQuery(""));
query = new PercolateQuery("_name", queryStore, Collections.singletonList(new BytesArray("{}")),
new TermQuery(new Term("select", "b")), percolateSearcher, new MatchNoDocsQuery(""));
topDocs = shardSearcher.search(query, 10);
assertThat(topDocs.totalHits, equalTo(3L));
assertThat(topDocs.scoreDocs.length, equalTo(3));

52
modules/percolator/src/test/java/org/elasticsearch/percolator/PercolatorHighlightSubFetchPhaseTests.java
View File

@ -38,16 +38,15 @@ import java.util.Arrays;
import java.util.Collections;
import static java.util.Collections.emptyMap;
import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.is;
import static org.hamcrest.Matchers.nullValue;
import static org.hamcrest.Matchers.sameInstance;
public class PercolatorHighlightSubFetchPhaseTests extends ESTestCase {
public void testHitsExecutionNeeded() {
PercolateQuery percolateQuery = new PercolateQuery(
ctx -> null, new BytesArray("{}"), new MatchAllDocsQuery(), Mockito.mock(IndexSearcher.class), new MatchAllDocsQuery()
);
PercolateQuery percolateQuery = new PercolateQuery("_name", ctx -> null, Collections.singletonList(new BytesArray("{}")),
new MatchAllDocsQuery(), Mockito.mock(IndexSearcher.class), new MatchAllDocsQuery());
PercolatorHighlightSubFetchPhase subFetchPhase = new PercolatorHighlightSubFetchPhase(Settings.EMPTY,
emptyMap());
SearchContext searchContext = Mockito.mock(SearchContext.class);
@ -60,35 +59,50 @@ public class PercolatorHighlightSubFetchPhaseTests extends ESTestCase {
}
public void testLocatePercolatorQuery() {
PercolateQuery percolateQuery = new PercolateQuery(
ctx -> null, new BytesArray("{}"), new MatchAllDocsQuery(), Mockito.mock(IndexSearcher.class), new MatchAllDocsQuery()
);
assertThat(PercolatorHighlightSubFetchPhase.locatePercolatorQuery(new MatchAllDocsQuery()), nullValue());
PercolateQuery percolateQuery = new PercolateQuery("_name", ctx -> null, Collections.singletonList(new BytesArray("{}")),
new MatchAllDocsQuery(), Mockito.mock(IndexSearcher.class), new MatchAllDocsQuery());
assertThat(PercolatorHighlightSubFetchPhase.locatePercolatorQuery(new MatchAllDocsQuery()).size(), equalTo(0));
BooleanQuery.Builder bq = new BooleanQuery.Builder();
bq.add(new MatchAllDocsQuery(), BooleanClause.Occur.FILTER);
assertThat(PercolatorHighlightSubFetchPhase.locatePercolatorQuery(bq.build()), nullValue());
assertThat(PercolatorHighlightSubFetchPhase.locatePercolatorQuery(bq.build()).size(), equalTo(0));
bq.add(percolateQuery, BooleanClause.Occur.FILTER);
assertThat(PercolatorHighlightSubFetchPhase.locatePercolatorQuery(bq.build()), sameInstance(percolateQuery));
assertThat(PercolatorHighlightSubFetchPhase.locatePercolatorQuery(bq.build()).size(), equalTo(1));
assertThat(PercolatorHighlightSubFetchPhase.locatePercolatorQuery(bq.build()).get(0), sameInstance(percolateQuery));
ConstantScoreQuery constantScoreQuery = new ConstantScoreQuery(new MatchAllDocsQuery());
assertThat(PercolatorHighlightSubFetchPhase.locatePercolatorQuery(constantScoreQuery), nullValue());
assertThat(PercolatorHighlightSubFetchPhase.locatePercolatorQuery(constantScoreQuery).size(), equalTo(0));
constantScoreQuery = new ConstantScoreQuery(percolateQuery);
assertThat(PercolatorHighlightSubFetchPhase.locatePercolatorQuery(constantScoreQuery), sameInstance(percolateQuery));
assertThat(PercolatorHighlightSubFetchPhase.locatePercolatorQuery(constantScoreQuery).size(), equalTo(1));
assertThat(PercolatorHighlightSubFetchPhase.locatePercolatorQuery(constantScoreQuery).get(0), sameInstance(percolateQuery));
BoostQuery boostQuery = new BoostQuery(new MatchAllDocsQuery(), 1f);
assertThat(PercolatorHighlightSubFetchPhase.locatePercolatorQuery(boostQuery), nullValue());
assertThat(PercolatorHighlightSubFetchPhase.locatePercolatorQuery(boostQuery).size(), equalTo(0));
boostQuery = new BoostQuery(percolateQuery, 1f);
assertThat(PercolatorHighlightSubFetchPhase.locatePercolatorQuery(boostQuery), sameInstance(percolateQuery));
assertThat(PercolatorHighlightSubFetchPhase.locatePercolatorQuery(boostQuery).size(), equalTo(1));
assertThat(PercolatorHighlightSubFetchPhase.locatePercolatorQuery(boostQuery).get(0), sameInstance(percolateQuery));
FunctionScoreQuery functionScoreQuery = new FunctionScoreQuery(new MatchAllDocsQuery(), new RandomScoreFunction(0, 0, null));
assertThat(PercolatorHighlightSubFetchPhase.locatePercolatorQuery(functionScoreQuery), nullValue());
assertThat(PercolatorHighlightSubFetchPhase.locatePercolatorQuery(functionScoreQuery).size(), equalTo(0));
functionScoreQuery = new FunctionScoreQuery(percolateQuery, new RandomScoreFunction(0, 0, null));
assertThat(PercolatorHighlightSubFetchPhase.locatePercolatorQuery(functionScoreQuery), sameInstance(percolateQuery));
assertThat(PercolatorHighlightSubFetchPhase.locatePercolatorQuery(functionScoreQuery).size(), equalTo(1));
assertThat(PercolatorHighlightSubFetchPhase.locatePercolatorQuery(functionScoreQuery).get(0), sameInstance(percolateQuery));
DisjunctionMaxQuery disjunctionMaxQuery = new DisjunctionMaxQuery(Arrays.asList(new MatchAllDocsQuery()), 1f);
assertThat(PercolatorHighlightSubFetchPhase.locatePercolatorQuery(disjunctionMaxQuery), nullValue());
DisjunctionMaxQuery disjunctionMaxQuery = new DisjunctionMaxQuery(Collections.singleton(new MatchAllDocsQuery()), 1f);
assertThat(PercolatorHighlightSubFetchPhase.locatePercolatorQuery(disjunctionMaxQuery).size(), equalTo(0));
disjunctionMaxQuery = new DisjunctionMaxQuery(Arrays.asList(percolateQuery, new MatchAllDocsQuery()), 1f);
assertThat(PercolatorHighlightSubFetchPhase.locatePercolatorQuery(disjunctionMaxQuery), sameInstance(percolateQuery));
assertThat(PercolatorHighlightSubFetchPhase.locatePercolatorQuery(disjunctionMaxQuery).size(), equalTo(1));
assertThat(PercolatorHighlightSubFetchPhase.locatePercolatorQuery(disjunctionMaxQuery).get(0), sameInstance(percolateQuery));
PercolateQuery percolateQuery2 = new PercolateQuery("_name", ctx -> null, Collections.singletonList(new BytesArray("{}")),
new MatchAllDocsQuery(), Mockito.mock(IndexSearcher.class), new MatchAllDocsQuery());
bq = new BooleanQuery.Builder();
bq.add(new MatchAllDocsQuery(), BooleanClause.Occur.FILTER);
assertThat(PercolatorHighlightSubFetchPhase.locatePercolatorQuery(bq.build()).size(), equalTo(0));
bq.add(percolateQuery, BooleanClause.Occur.FILTER);
bq.add(percolateQuery2, BooleanClause.Occur.FILTER);
assertThat(PercolatorHighlightSubFetchPhase.locatePercolatorQuery(bq.build()).size(), equalTo(2));
assertThat(PercolatorHighlightSubFetchPhase.locatePercolatorQuery(bq.build()).get(0), sameInstance(percolateQuery));
assertThat(PercolatorHighlightSubFetchPhase.locatePercolatorQuery(bq.build()).get(1), sameInstance(percolateQuery2));
}
}

72
modules/percolator/src/test/java/org/elasticsearch/percolator/PercolatorMatchedSlotSubFetchPhaseTests.java Normal file
View File

@ -0,0 +1,72 @@
/*
* Licensed to Elasticsearch under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
package org.elasticsearch.percolator;
import org.apache.lucene.search.ScoreDoc;
import org.apache.lucene.search.TopDocs;
import org.apache.lucene.util.FixedBitSet;
import org.elasticsearch.test.ESTestCase;
import java.util.stream.IntStream;
public class PercolatorMatchedSlotSubFetchPhaseTests extends ESTestCase {
public void testConvertTopDocsToSlots() {
ScoreDoc[] scoreDocs = new ScoreDoc[randomInt(128)];
for (int i = 0; i < scoreDocs.length; i++) {
scoreDocs[i] = new ScoreDoc(i, 1f);
}
TopDocs topDocs = new TopDocs(scoreDocs.length, scoreDocs, 1f);
IntStream stream = PercolatorMatchedSlotSubFetchPhase.convertTopDocsToSlots(topDocs, null);
int[] result = stream.toArray();
assertEquals(scoreDocs.length, result.length);
for (int i = 0; i < scoreDocs.length; i++) {
assertEquals(scoreDocs[i].doc, result[i]);
}
}
public void testConvertTopDocsToSlots_nestedDocs() {
ScoreDoc[] scoreDocs = new ScoreDoc[5];
scoreDocs[0] = new ScoreDoc(2, 1f);
scoreDocs[1] = new ScoreDoc(5, 1f);
scoreDocs[2] = new ScoreDoc(8, 1f);
scoreDocs[3] = new ScoreDoc(11, 1f);
scoreDocs[4] = new ScoreDoc(14, 1f);
TopDocs topDocs = new TopDocs(scoreDocs.length, scoreDocs, 1f);
FixedBitSet bitSet = new FixedBitSet(15);
bitSet.set(2);
bitSet.set(5);
bitSet.set(8);
bitSet.set(11);
bitSet.set(14);
int[] rootDocsBySlot = PercolatorMatchedSlotSubFetchPhase.buildRootDocsSlots(bitSet);
int[] result = PercolatorMatchedSlotSubFetchPhase.convertTopDocsToSlots(topDocs, rootDocsBySlot).toArray();
assertEquals(scoreDocs.length, result.length);
assertEquals(0, result[0]);
assertEquals(1, result[1]);
assertEquals(2, result[2]);
assertEquals(3, result[3]);
assertEquals(4, result[4]);
}
}

160
modules/percolator/src/test/java/org/elasticsearch/percolator/PercolatorQuerySearchIT.java
View File

@ -38,6 +38,7 @@ import org.elasticsearch.search.sort.SortOrder;
import org.elasticsearch.test.ESIntegTestCase;
import java.util.Arrays;
import java.util.Collections;
import static org.elasticsearch.common.xcontent.XContentFactory.jsonBuilder;
import static org.elasticsearch.common.xcontent.XContentFactory.smileBuilder;
@ -100,7 +101,9 @@ public class PercolatorQuerySearchIT extends ESIntegTestCase {
.get();
assertHitCount(response, 2);
assertThat(response.getHits().getAt(0).getId(), equalTo("1"));
assertThat(response.getHits().getAt(0).getFields().get("_percolator_document_slot").getValue(), equalTo(0));
assertThat(response.getHits().getAt(1).getId(), equalTo("2"));
assertThat(response.getHits().getAt(1).getFields().get("_percolator_document_slot").getValue(), equalTo(0));
source = jsonBuilder().startObject().field("field1", "value").field("field2", "value").endObject().bytes();
logger.info("percolating doc with 2 fields");
@ -110,8 +113,27 @@ public class PercolatorQuerySearchIT extends ESIntegTestCase {
.get();
assertHitCount(response, 3);
assertThat(response.getHits().getAt(0).getId(), equalTo("1"));
assertThat(response.getHits().getAt(0).getFields().get("_percolator_document_slot").getValue(), equalTo(0));
assertThat(response.getHits().getAt(1).getId(), equalTo("2"));
assertThat(response.getHits().getAt(1).getFields().get("_percolator_document_slot").getValue(), equalTo(0));
assertThat(response.getHits().getAt(2).getId(), equalTo("3"));
assertThat(response.getHits().getAt(2).getFields().get("_percolator_document_slot").getValue(), equalTo(0));
logger.info("percolating doc with 2 fields");
response = client().prepareSearch()
.setQuery(new PercolateQueryBuilder("query", Arrays.asList(
jsonBuilder().startObject().field("field1", "value").endObject().bytes(),
jsonBuilder().startObject().field("field1", "value").field("field2", "value").endObject().bytes()
), XContentType.JSON))
.addSort("_uid", SortOrder.ASC)
.get();
assertHitCount(response, 3);
assertThat(response.getHits().getAt(0).getId(), equalTo("1"));
assertThat(response.getHits().getAt(0).getFields().get("_percolator_document_slot").getValues(), equalTo(Arrays.asList(0, 1)));
assertThat(response.getHits().getAt(1).getId(), equalTo("2"));
assertThat(response.getHits().getAt(1).getFields().get("_percolator_document_slot").getValues(), equalTo(Arrays.asList(0, 1)));
assertThat(response.getHits().getAt(2).getId(), equalTo("3"));
assertThat(response.getHits().getAt(2).getFields().get("_percolator_document_slot").getValues(), equalTo(Arrays.asList(1)));
}
public void testPercolatorRangeQueries() throws Exception {
@ -446,6 +468,119 @@ public class PercolatorQuerySearchIT extends ESIntegTestCase {
equalTo("The quick brown fox jumps over the lazy <em>dog</em>"));
assertThat(searchResponse.getHits().getAt(4).getHighlightFields().get("field1").fragments()[0].string(),
equalTo("The quick brown <em>fox</em> jumps over the lazy dog"));
BytesReference document1 = jsonBuilder().startObject()
.field("field1", "The quick brown fox jumps")
.endObject().bytes();
BytesReference document2 = jsonBuilder().startObject()
.field("field1", "over the lazy dog")
.endObject().bytes();
searchResponse = client().prepareSearch()
.setQuery(boolQuery()
.should(new PercolateQueryBuilder("query", document1, XContentType.JSON).setName("query1"))
.should(new PercolateQueryBuilder("query", document2, XContentType.JSON).setName("query2"))
)
.highlighter(new HighlightBuilder().field("field1"))
.addSort("_uid", SortOrder.ASC)
.get();
logger.info("searchResponse={}", searchResponse);
assertHitCount(searchResponse, 5);
assertThat(searchResponse.getHits().getAt(0).getHighlightFields().get("query1_field1").fragments()[0].string(),
equalTo("The quick <em>brown</em> <em>fox</em> jumps"));
assertThat(searchResponse.getHits().getAt(1).getHighlightFields().get("query2_field1").fragments()[0].string(),
equalTo("over the <em>lazy</em> <em>dog</em>"));
assertThat(searchResponse.getHits().getAt(2).getHighlightFields().get("query1_field1").fragments()[0].string(),
equalTo("The quick brown fox <em>jumps</em>"));
assertThat(searchResponse.getHits().getAt(3).getHighlightFields().get("query2_field1").fragments()[0].string(),
equalTo("over the lazy <em>dog</em>"));
assertThat(searchResponse.getHits().getAt(4).getHighlightFields().get("query1_field1").fragments()[0].string(),
equalTo("The quick brown <em>fox</em> jumps"));
searchResponse = client().prepareSearch()
.setQuery(new PercolateQueryBuilder("query", Arrays.asList(
jsonBuilder().startObject().field("field1", "dog").endObject().bytes(),
jsonBuilder().startObject().field("field1", "fox").endObject().bytes(),
jsonBuilder().startObject().field("field1", "jumps").endObject().bytes(),
jsonBuilder().startObject().field("field1", "brown fox").endObject().bytes()
), XContentType.JSON))
.highlighter(new HighlightBuilder().field("field1"))
.addSort("_uid", SortOrder.ASC)
.get();
assertHitCount(searchResponse, 5);
assertThat(searchResponse.getHits().getAt(0).getFields().get("_percolator_document_slot").getValues(),
equalTo(Arrays.asList(1, 3)));
assertThat(searchResponse.getHits().getAt(0).getHighlightFields().get("1_field1").fragments()[0].string(),
equalTo("<em>fox</em>"));
assertThat(searchResponse.getHits().getAt(0).getHighlightFields().get("3_field1").fragments()[0].string(),
equalTo("<em>brown</em> <em>fox</em>"));
assertThat(searchResponse.getHits().getAt(1).getFields().get("_percolator_document_slot").getValues(),
equalTo(Collections.singletonList(0)));
assertThat(searchResponse.getHits().getAt(1).getHighlightFields().get("0_field1").fragments()[0].string(),
equalTo("<em>dog</em>"));
assertThat(searchResponse.getHits().getAt(2).getFields().get("_percolator_document_slot").getValues(),
equalTo(Collections.singletonList(2)));
assertThat(searchResponse.getHits().getAt(2).getHighlightFields().get("2_field1").fragments()[0].string(),
equalTo("<em>jumps</em>"));
assertThat(searchResponse.getHits().getAt(3).getFields().get("_percolator_document_slot").getValues(),
equalTo(Collections.singletonList(0)));
assertThat(searchResponse.getHits().getAt(3).getHighlightFields().get("0_field1").fragments()[0].string(),
equalTo("<em>dog</em>"));
assertThat(searchResponse.getHits().getAt(4).getFields().get("_percolator_document_slot").getValues(),
equalTo(Arrays.asList(1, 3)));
assertThat(searchResponse.getHits().getAt(4).getHighlightFields().get("1_field1").fragments()[0].string(),
equalTo("<em>fox</em>"));
assertThat(searchResponse.getHits().getAt(4).getHighlightFields().get("3_field1").fragments()[0].string(),
equalTo("brown <em>fox</em>"));
searchResponse = client().prepareSearch()
.setQuery(boolQuery()
.should(new PercolateQueryBuilder("query", Arrays.asList(
jsonBuilder().startObject().field("field1", "dog").endObject().bytes(),
jsonBuilder().startObject().field("field1", "fox").endObject().bytes()
), XContentType.JSON).setName("query1"))
.should(new PercolateQueryBuilder("query", Arrays.asList(
jsonBuilder().startObject().field("field1", "jumps").endObject().bytes(),
jsonBuilder().startObject().field("field1", "brown fox").endObject().bytes()
), XContentType.JSON).setName("query2"))
)
.highlighter(new HighlightBuilder().field("field1"))
.addSort("_uid", SortOrder.ASC)
.get();
logger.info("searchResponse={}", searchResponse);
assertHitCount(searchResponse, 5);
assertThat(searchResponse.getHits().getAt(0).getFields().get("_percolator_document_slot_query1").getValues(),
equalTo(Collections.singletonList(1)));
assertThat(searchResponse.getHits().getAt(0).getFields().get("_percolator_document_slot_query2").getValues(),
equalTo(Collections.singletonList(1)));
assertThat(searchResponse.getHits().getAt(0).getHighlightFields().get("query1_1_field1").fragments()[0].string(),
equalTo("<em>fox</em>"));
assertThat(searchResponse.getHits().getAt(0).getHighlightFields().get("query2_1_field1").fragments()[0].string(),
equalTo("<em>brown</em> <em>fox</em>"));
assertThat(searchResponse.getHits().getAt(1).getFields().get("_percolator_document_slot_query1").getValues(),
equalTo(Collections.singletonList(0)));
assertThat(searchResponse.getHits().getAt(1).getHighlightFields().get("query1_0_field1").fragments()[0].string(),
equalTo("<em>dog</em>"));
assertThat(searchResponse.getHits().getAt(2).getFields().get("_percolator_document_slot_query2").getValues(),
equalTo(Collections.singletonList(0)));
assertThat(searchResponse.getHits().getAt(2).getHighlightFields().get("query2_0_field1").fragments()[0].string(),
equalTo("<em>jumps</em>"));
assertThat(searchResponse.getHits().getAt(3).getFields().get("_percolator_document_slot_query1").getValues(),
equalTo(Collections.singletonList(0)));
assertThat(searchResponse.getHits().getAt(3).getHighlightFields().get("query1_0_field1").fragments()[0].string(),
equalTo("<em>dog</em>"));
assertThat(searchResponse.getHits().getAt(4).getFields().get("_percolator_document_slot_query1").getValues(),
equalTo(Collections.singletonList(1)));
assertThat(searchResponse.getHits().getAt(4).getFields().get("_percolator_document_slot_query2").getValues(),
equalTo(Collections.singletonList(1)));
assertThat(searchResponse.getHits().getAt(4).getHighlightFields().get("query1_1_field1").fragments()[0].string(),
equalTo("<em>fox</em>"));
assertThat(searchResponse.getHits().getAt(4).getHighlightFields().get("query2_1_field1").fragments()[0].string(),
equalTo("brown <em>fox</em>"));
}
public void testTakePositionOffsetGapIntoAccount() throws Exception {
@ -463,7 +598,7 @@ public class PercolatorQuerySearchIT extends ESIntegTestCase {
client().admin().indices().prepareRefresh().get();
SearchResponse response = client().prepareSearch().setQuery(
new PercolateQueryBuilder("query", null, new BytesArray("{\"field\" : [\"brown\", \"fox\"]}"), XContentType.JSON)
new PercolateQueryBuilder("query", new BytesArray("{\"field\" : [\"brown\", \"fox\"]}"), XContentType.JSON)
).get();
assertHitCount(response, 1);
assertThat(response.getHits().getAt(0).getId(), equalTo("2"));
@ -614,6 +749,29 @@ public class PercolatorQuerySearchIT extends ESIntegTestCase {
.addSort("_doc", SortOrder.ASC)
.get();
assertHitCount(response, 0);
response = client().prepareSearch()
.setQuery(new PercolateQueryBuilder("query", Arrays.asList(
XContentFactory.jsonBuilder()
.startObject().field("companyname", "stark")
.startArray("employee")
.startObject().field("name", "virginia potts").endObject()
.startObject().field("name", "tony stark").endObject()
.endArray()
.endObject().bytes(),
XContentFactory.jsonBuilder()
.startObject().field("companyname", "stark")
.startArray("employee")
.startObject().field("name", "peter parker").endObject()
.startObject().field("name", "virginia potts").endObject()
.endArray()
.endObject().bytes()
), XContentType.JSON))
.addSort("_doc", SortOrder.ASC)
.get();
assertHitCount(response, 1);
assertThat(response.getHits().getAt(0).getId(), equalTo("q1"));
assertThat(response.getHits().getAt(0).getFields().get("_percolator_document_slot").getValues(), equalTo(Arrays.asList(0, 1)));
}
public void testPercolatorQueryViaMultiSearch() throws Exception {

7
test/framework/src/main/java/org/elasticsearch/test/AbstractQueryTestCase.java
View File

@ -20,7 +20,6 @@
package org.elasticsearch.test;
import com.fasterxml.jackson.core.io.JsonStringEncoder;
import org.apache.lucene.search.BoostQuery;
import org.apache.lucene.search.Query;
import org.apache.lucene.search.TermQuery;
@ -413,7 +412,9 @@ public abstract class AbstractQueryTestCase<QB extends AbstractQueryBuilder<QB>>
// Parse the valid query and inserts a new object level called "newField"
XContentParser.Token token;
while ((token = parser.nextToken()) != null) {
if (token == XContentParser.Token.START_OBJECT) {
if (token == XContentParser.Token.START_ARRAY) {
levels.addLast(parser.currentName());
} else if (token == XContentParser.Token.START_OBJECT) {
objectIndex++;
levels.addLast(parser.currentName());
@ -438,7 +439,7 @@ public abstract class AbstractQueryTestCase<QB extends AbstractQueryBuilder<QB>>
// Jump to next token
continue;
}
} else if (token == XContentParser.Token.END_OBJECT) {
} else if (token == XContentParser.Token.END_OBJECT || token == XContentParser.Token.END_ARRAY) {
levels.removeLast();
}