* Throw a better error message for empty field names
When a document is parsed with a `""` for a field name, we currently throw a
confusing error about `.` being present in the field. This changes the error
message to be clearer about what's causing the problem.
Resolves#23348
* Fix exception message in test
To protect against poisonous situations, ES will only try to allocate a shard 5 times (by default). After 5 consecutive failures, ES will stop assigning the shard and wait for an operator to fix the problem. Once the problem is fixed, the operator is expected to call `_reroute` with a `retry_failed` flag to force retrying of those shards. Currently that retry flag is only used for a single allocation run. However, if not all shards can be allocated at once (due to throttling) the operator has to keep on calling the API until all shards are assigned which is cumbersome. This PR changes the behavior of the flag to reset the failed allocations counter and this allowing shards to be assigned again.
This test should not rely on strict ordering for same score suggestions.
The Lucene completion suggester uses the doc id in case of a tie and documents are indexed randomly.
This commit removes a norelease from the codebase now that there is a CI
job that fails on the norelease pattern being present. Instead, a new
issue has been opened to track this one.
Relates #26544
The completion suggester has a `shard_size` option that sets the size of the suggestions to retrieve per shard but it is ignored
by the builder. This commit restores the handling of this option and fixes a test that can randomly fail without it.
This change exposes the duplicate removal option added in Lucene for the completion suggester
with a new option called `skip_duplicates` (defaults to false).
This commit also adapts the custom suggest collector to handle deduplication when multiple contexts match the input.
Closes#23364
This change fixes a regression introduced in 6 that removes the skipping of the rescore phase
when a sort other than _score is used.
We now fail the request when a sort is provided in conjunction with rescore instead of just skipping the rescore phase
This commit also adds an assert that checks if the topdocs are sorted by _score after the rescoring.
This is the responsibility of the rescorer to make sure that topdocs are sorted after rescore so we
just check that this condition is met in the rescore phase.
The three SortBuilders that can have inner NestedSortBuilders currently don't
rewrite any of the filters contained in them. This change adds a rewrite method
to NestedSortBuilder and changes rewriting in FieldSortBuilder,
ScriptSortBuilder and GeoDistanceSortBuilder to make sure inner nested sorts get
rewritten if they need to.
Improve testing around the ScriptSortBuilder#build method, adding checks for
correct transfers of the sort mode and nested sorts.
Also changing the behaviour around the nested_path, nested_filter vs. nested
parameter in a similar way as in #26490 and deprecating the setters/getters for
the old syntax.
Closes#17286
Security manager policy files contains grants for specific codebases,
where a codebase is a jar file. We use a system property containing the
name of the jar file to resolve the jar file location when parsing the
policy file. However, this means the version of the jars must be
modified when versions of dependencies change. This is particularly
messy for elasticsearch, where we now have a dependency on the rest
client, and need to support both a snapshot version for testing and non
snapshot for release.
This commit adds an alias for the elasticsearch rest client without a
version to be used in policy files. That allows the policy files to not care whether
the rest client is a snapshot or release.
Resoves #26332 where too many tasks occurred while adjustment was happening, the
measurements were reset to 0, and then an assert failed due to tasks executing
in 0 nanoseconds
When a cache entry expires, it remains in the cache (both the segment
that it belongs to, and the LRU list) until an eviction occurs. The
problem here is that the compute if absent implementation relies on
there not being an association to a key that we are trying to put
because it internally uses put if absent on the underlying segment. If
we try to put an association for a key that has expired but not been
evicted, then compute if absent will return as if there is nothing in
the cache for the given key, yet no call to compute if absent will
succeed in putting a new association for the key. To remedy this, we
modify the internal get method for the cache to let the caller take
action if the entry they are retrieving is expired. This allows the
compute if absent method to take the action of evicting the entry from
the cache, thus allowing the put if absent method used by compute if
absent to succeed for one of the callers trying to compute if absent a
new association.
Relates #26516
This change adds a dynamic cluster setting named `search.max_keep_alive`.
It is used as an upper limit for scroll expiry time in scroll queries and defaults to 1 hour.
This change also ensures that the existing setting `search.default_keep_alive` is always smaller than `search.max_keep_alive`.
Relates #11511
* check style
* add skip for bwc
* iter
* Add a maxium throttle wait time of 1h for reindex
* review
* remove empty line
The `_type` and `types` version of the current `type` parameter have been
deprecated since 5.0. We can remove support for them in 7.0 and also in 6.x and
6.0.
We currently have a weird relationship between Transport,
TransportService, and TransportServiceAdaptor. At some point I think
that we would like to collapse these all into one concept as we only
support TCP transports.
This commit moves in that direction by eliminating the adaptor and just
passing the transport service to the transport.
Improve testing around the GeoDistanceSortBuilder#build method, adding checks for correct
transfers of the sort order, mode, nested sorts and points validation and coercion.
Also changing the behaviour around the nested_path, nested_filter vs. nested parameter in
a similar way as in #26490 and deprecating the setters/getters for the old syntax.
Relates to #17286
Currently we allow both "old" and "new" way of setting nested sorts on the
FieldSortBuilder at the same time. This should throw an error, instead the user
should choose one of the two possible options.
Also adding testing for the now deprecated nestedPath/nestedFilter parameters,
inlcuding checks that they emmit warnings on parsing and that the new
NestetedSortBuilder overwrites the deprecated parameters when building the
SortField.
Relates to #17286
Adding a check to QueryStringQueryBuilderTests that checks the override
behaviour of `quote_analyzer`, also adding documentation explaining the use of
this parameter in `query_string` query.
Closes#25417
We are getting the default Object#toString implementation here, we need
more than this. This commit instead formats the primary response to JSON
so we can see into its soul.
The new NestedSortBuilder currently is only tested via its use in the other
SortBuilder implementations it can be used in. This adds its own simple unit
test class that at first checks our usual fromXContent parsing, serialization
and hashCode/equals checks. It also adds tests for cases where NestedSortBuilder
is nested in itself and reuses the code for creating randomized instances in the
other SortBuilder tests.
In addition to the tests, this changes the `path` parameter in NestedSortBuilder
to be mandatory and removes the `read` method since it is not really needed.
The current script service has a script compilation limit for a one
minute window. This is set to a small default value of 15. Instead of
increasing that default value, this commit introduces a new setting
that allows to configure a rate per time unit, so that the script service can deal with bursts better.
The new setting is named `script.max_compilations_rate`,
requires a nonnegative number and a positive time value.
The default is `75/5m`, which is equivalent to the existing 15 per minute.
In some cases a request can already be aborted and retried. This means
the condition that aborting a request should only happen when an item
has not been processed yet is too strict. This commit allows for a
double abort. If we attempt to abort an operation that was previously
processed but not aborted, we treat that as a hard failure.
Relates #26434
Adds support for bulk items to be aborted before they are processed by the TransportShardBulkAction.
This can be used by an ActionFilter to reject a subset of the items in a bulk action without rejecting the whole action (or all the items for a shard).
Currently we don't have much unit testing about the SortField that is created then
calling the SortBuilders `build` method. Most of this is covered by integration tests
somewhere but it would be good to have some basic checks in FieldSortBuilderTest
as well.
This adds testing for the sort order, mode, missing values and checks that `nested`
gets set in the XFieldComparatorSource when `nestedPath` and `nestedFilter` are
set on the builder.
Relates to #17286
* Implement adaptive replica selection
This implements the selection algorithm described in the C3 paper for
determining which copy of the data a query should be routed to.
By using the service time EWMA, response time EWMA, and queue size EWMA we
calculate the score of a node by piggybacking these metrics with each search
request.
Since Elasticsearch lacks the "broadcast to every copy" behavior that Cassandra
has (as mentioned in the C3 paper) to update metrics after a node has been
highly weighted, this implementation adjusts a node's response stats using the
average of the its own and the "best" node's metrics. This is so that a long GC
or other activity that may cause a node's rank to increase dramatically does not
permanently keep a node from having requests routed to it, instead it will
eventually lower its score back to the realm where it is a potential candidate
for new queries.
This feature is off by default and can be turned on with the dynamic setting
`cluster.routing.use_adaptive_replica_selection`.
Relates to #24915, however instead of `b=3` I used `b=4` (after benchmarking)
* Randomly use adaptive replica selection for internal test cluster
* Use an action name *prefix* for retrieving pending requests
* Add unit test for replica selection
* don't use adaptive replica selection in SearchPreferenceIT
* Track client connections in a SearchTransportService instead of TransportService
* Bind `entry` pieces in local variables
* Add javadoc link to C3 paper and javadocs for stat adjustments
* Bind entry's key and value to local variables
* Remove unneeded actionNamePrefix parameter
* Use conns.longValue() instead of cached Long
* Add comments about removing entries from the map
* Pull out bindings for `entry` in IndexShardRoutingTable
* Use .compareTo instead of manually comparing
* add assert for connections not being null and gte to 1
* Copy map for pending search connections instead of "live" map
* Increase the number of pending search requests used for calculating rank when chosen
When a node gets chosen, this increases the number of search counts for the
winning node so that it will not be as likely to be chosen again for
non-concurrent search requests.
* Remove unused HashMap import
* Rename rank -> rankShardsAndUpdateStats
* Rename rankedActiveInitializingShardsIt -> activeInitializingShardsRankedIt
* Instead of precalculating winning node, use "winning" shard from ranked list
* Sort null ranked nodes before nodes that have a rank
Multi-level Nested Sort with Filters
Allow multiple levels of nested sorting where each level can have it's own filter.
Backward compatible with previous single-level nested sort.
* Moves deferring code into its own subclass
This change moves the code that deals with deferring collection to a subclass of BucketAggregator called DeferringBucketAggregator. This means that the code in AggregatorBase is simplified and also means that the code for deferring colleciton is in one place and easier to maintain.
* Makes SIngleBucketAggregator an interface
This is so aggregators that extend BucketsAggregator directly and those that extend DeferringBucketAggregator can be a single bucket aggregator
* review comments
* More review comments
When creating the keystore explicitly (from executing
elasticsearch-keystore create) or implicitly (for plugins that require
the keystore to be created on install) on an Elasticsearch package
installation, we are running as the root user. This leaves
/etc/elasticsearch/elasticsearch.keystore having the wrong ownership
(root:root) so that the elasticsearch user can not read the keystore on
startup. This commit adds setgid to /etc/elasticsearch on package
installation so that when executing this directory (as we would when
creating the keystore), we will end up with the correct ownership
(root:elasticsearch). Additionally, we set the permissions on the
keystore to be 660 so that the elasticsearch user via its group can read
this file on startup.
Relates #26412
* Remove the _all metadata field
This change removes the `_all` metadata field. This field is deprecated in 6
and cannot be activated for indices created in 6 so it can be safely removed in
the next major version (e.g. 7).
Lee Hinman
Antonio Matarrese
Jim Ferenczi
Jason Tedor
Jim Ferenczi
Christoph Büscher
Ryan Ernst
Lee Hinman
Tim Brooks
Martijn van Groningen
Boaz Leskes
Alexander Reelsen
Tal Levy
Tim Vernum
Adrien Grand
Matt Weber
Colin Goodheart-Smithe
Sergey Galkin
Stuart Neivandt