Our own fork of the lucene PostingsHighlighter is not easy to maintain and doesn't give us any added value at this point. In particular, it was introduced to support the require_field_match option and discrete per value highlighting, used in case one wants to highlight the whole content of a field, but get back one snippet per value. These two features won't
make it into lucene as they slow things down and shouldn't have been supported from day one on our end probably.
One other customization we had was support for a wider range of queries via custom rewrite etc. (yet another way to slow
things down), which got added to lucene and works much much better than what we used to do (instead of or rewrite, term
s are pulled out of the automata for multi term queries).
Removing our fork means the following in terms of features:
- dropped support for require_field_match: the postings highlighter will only highlight fields that were queried
- some custom es queries won't be supported anymore, meaning they won't be highlighted. The only one I found up until now is the phrase_prefix. Postings highlighter rewrites against an empty reader to avoid slow operations (like the ones that we were performing with the fork that we are removing here), thus the prefix will not be expanded to any term. What the postings highlighter does instead is pulling the automata out of multi term queries, but this is not supported at the moment with our MultiPhrasePrefixQuery.
Closes#10625Closes#11077
The underlying automaton-backed implementation throws an error if there are too many states.
This fix changes to using an implementation based on Set lookups for lists of excluded terms.
If the global-ordinals execution mode is in effect this implementation also addresses the slowness identified in issue 11181 which is caused by traversing the TermsEnum - instead the excluded terms’ global ordinals are looked up individually and unset the bits of acceptable terms. This is significantly faster.
Closes#11176
When scrolling, SCAN previously collected documents until it reached where it
had stopped on the previous iteration. This makes pagination slower and slower
as you request deep pages. With this change, SCAN now directly jumps to the
doc ID where is had previously stopped.
Most aggregations (terms, histogram, stats, percentiles, geohash-grid) now
support a new `missing` option which defines the value to consider when a
field does not have a value. This can be handy if you eg. want a terms
aggregation to handle the same way documents that have "N/A" or no value
for a `tag` field.
This works in a very similar way to the `missing` option on the `sort`
element.
One known issue is that this option sometimes cannot make the right decision
in the unmapped case: it needs to replace all values with the `missing` value
but might not know what kind of values source should be produced (numerics,
strings, geo points?). For this reason, we might want to add an `unmapped_type`
option in the future like we did for sorting.
Related to #5324
When specifying relative paths on startup, handling plugin
paths failed due to recently added security fix. This fix
ensures normalization of the plugin path as well.
In addition a new matcher has been added to easily check for a
status code of an HTTP response likes this
assertThat(response, hasStatus(OK));
Closes#10958
When an index setting is invalid and fails to be set, a WARN statement
is logged but it doesn't contain the index name, making tracking down
and fixing the problem more difficult. This commit adds the index name
to the log statement.
Previously, collate feature would be executed on all shards of an index using the client,
this leads to a deadlock when concurrent collate requests are run from the _search API,
due to the fact that both the external request and internal collate requests use the
same search threadpool.
As phrase suggestions are generated from the terms of the local shard, in most cases the
generated suggestion, which does not yield a hit for the collate query on the local shard
would not yield a hit for collate query on non-local shards.
Instead of using the client for collating suggestions, collate query is executed against
the ContextIndexSearcher. This PR removes the ability to specify a preference for a collate
query, as the collate query is only run on the local shard.
closes#9377
This adds back the ability to disable _source, as well as set includes
and excludes. However, it also restricts these settings to not be
updateable. enabled was actually already not modifiable, but no
conflict was previously given if an attempt was made to change it.
This also adds a check that can be made on the source mapper to
know if the the source is "complete" and can be used for
purposes other than returning in search or get requests. There is
one example use here in highlighting, but more need to be added
in a follow up issue (eg in the update API).
closes#11116
Add methods to operate on multi-valued fields in the expressions language.
Note that users will still not be able to access individual values
within a multi-valued field.
The following methods will be included:
* min
* max
* avg
* median
* count
* sum
Additionally, changes have been made to MultiValueMode to support the
new median method.
closes#11105
Today we barf if repositories are unregistered with a `*` pattern. This
happens on almost every test and adds weird log messages. I dont' think
we should barf in that case.
Closes#11113
We have some builders, specifically query builders, `SearchSourceBuilder`, `QuerySourceBuilder` and `SuggestBuilder`, that implement `ToXContent` and also allow to build their content as bytes by simply creating a `BytesReference` that holds their json (or yaml etc.) content (`buildAsBytes` methods). They can also print out their content through `toString`. Made sure that those common methods are in one single place and reused where needed.
Also, merged `QueryBuilder` and `BaseQueryBuilder` and made `QueryBuilder` an abstract class instead of an interface.
Closes#11063
We parse the rewrite field in FuzzyQueryParser but we don't allow to set it via FuzzyQueryBuilder for our java api users. Added missing field and setter.
Closes#11130Closes#11139
The esoteric classifier contains in particular maps that take bytes or doubles
as keys. In the byte case, we can just use integer, and in the double case we
can use their long bits instead.
These clauses filter the document space without affecting scoring and map to
Lucene's BooleanClause.Occur.FILTER. The `filtered` query is now deprecated and
```json
{
"filtered": {
"query": { //query },
"filter": { //filter }
}
}
```
should be replaced with
```json
{
"bool": {
"must": { //query },
"filter": { //filter }
}
}
```
In case FieldNameAnalyzer does not find an explicit analyzer for a given
field name, it returns the default analyzer. This behaviour can hide bugs
where the analyzer fails to be propagated to FieldNameAnalyzer or an
analyzer is requested for a field which is not mapped.
We are using a a VLong to serialize the PercolateResponse#tookInMillis. This
can due to several `System.currentTimeMillis()` implemenation details be negative.
We should prevent the negavite value for being serialized as a VLong and make sure
we use a valid value for this in the first place
Closes#11138
javanna
markharwood
Adrien Grand
Clinton Gormley
Jun Ohtani
Martijn van Groningen
Alexander Reelsen
Nicholas Canzoneri
Ryan Ernst
Areek Zillur
Lee Hinman
Jack Conradson
Robert Muir
Robert Muir
Simon Willnauer
Dawid Weiss
Shay Banon