ShardSearchRequest was previously taking in the whole ShardRouting as a constructor argument while it only needs the ShardsId, changed that to carry over only the needed bits.
* Transport client: Fix remove address to actually work
The removeTransportAddress method of TransportClient removes the address
from the list of nodes that the client pings to sniff for nodes.
However, it does not remove it from the list of existing connected
nodes. This means removing a node is not possible, as long as that node
is still up.
This change removes the node from the connected nodes list before
triggering sampling (ie sniffing). While the fix is simple, testing was
not because there were no existing tests for sniffing. This change also
modifies the mocks used by transport client unit tests in order to allow
mocking sniffing.
* Scripting: Remove groovy scripting language
Groovy was deprecated in 5.0. This change removes it, along with the
legacy default language infrastructure in scripting.
The `error_trace` parameter turns on the `stack_trace` field
in errors which returns stack traces.
Removes documentation for `camelCase` because it hasn't worked
in a while....
Documents the internal parameters used to render stack traces as
internal only.
Closes#21708
This commit refactors the handling of bind permissions, which is in need
of a little cleanup. For example, in its current state, the code for
handling permissions for transport profiles is split across two
methods. This commit refactors this code hopefully making it easier to
work with in future changes. This change is mostly mechanical, no
functionality is changed.
Relates #21742
The test UnicastZenPing#testResolveTimeout chooses a random resolve
timeout between 1ms and 100ms. Close to the lower bound, this is far too
short and the test races against the concurrent resolves executing
before the timeout elapses. This commit increases the timeout to
something that is far less likely to race, yet will not slow the test
down since we are not doing resolves against a real DNS service anyway.
Note that we still want a short resolve timeout since we are testing
whether or not timeouts really work here (by latching one of the
resolves to respond slowly).
Add indices and filter information to search shards api output
The search shards api returns info about which shards are going to be hit by executing a search with provided parameters: indices, routing, preference. Indices can also be aliases, which can also hold filters. The output includes an array of shards and a summary of all the nodes the shards are allocated on. This commit adds a new indices section to the search shards output that includes one entry per index, where each index can be associated with an optional filter in case the index was hit through a filtered alias.
This is relevant since we have moved parsing of alias filters to the coordinating node.
Relates to #20916
Today there is no way to get notified if a node is disconnected. Client code
must poll the TransportClient constantly to detect that a node is not connected
anymore in order to react and add new nodes or notify altering etc. For instance
if a hostname gets resolved to an IP but that host is disconnected clients want
to reconnect by resolving the hostname again which is a common situation in cloud
environments.
Closes#21424
Today we eagerly resolve unicast hosts. This means that if DNS changes,
we will never find the host at the new address. Moreover, a single host
failng to resolve causes startup to abort. This commit introduces lazy
resolution of unicast hosts. If a DNS entry changes, there is an
opportunity for the host to be discovered. Note that under the Java
security manager, there is a default positive cache of infinity for
resolved hosts; this means that if a user does want to operate in an
environment where DNS can change, they must adjust
networkaddress.cache.ttl in their security policy. And if a host fails
to resolve, we warn log the hostname but continue pinging other
configured hosts.
When doing DNS resolutions for unicast hostnames, we wait until the DNS
lookups timeout. This appears to be forty-five seconds on modern JVMs,
and it is not configurable. If we do these serially, the cluster can be
blocked during ping for a lengthy period of time. This commit introduces
doing the DNS lookups in parallel, and adds a user-configurable timeout
for these lookups.
Relates #21630
PR #19416 added a safety mechanism to shard state fetching to only access the store when the shard lock can be acquired. This can lead to the following situation however where a shard has not fully shut down yet while the shard fetching is going on, resulting in a ShardLockObtainFailedException. PrimaryShardAllocator that decides where to allocate primary shards sees this exception and treats the shard as unusable. If this is the only shard copy in the cluster, the cluster stays red and a new shard fetching cycle will not be triggered as shard state fetching treats exceptions while opening the store as permanent failures.
This commit makes it so that PrimaryShardAllocator treats the locked shard as a possible allocation target (although with the least priority).
This commit clarifies the contract of Cache#computeIfAbsent so that an exception that occurs during the execution
of the loader is thrown to all callers. Prior to this commit, the first caller would get the ExecutionException
and other callers that called during the load execution would get null, which is confusing.
The `type` parameter has always been accepted by the search_shards api, probably to make the api and its urls the same as search. Truth is that the type never had any effect, it's been ignored from day one while accepting it may make users think that we actually do something with it.
This commit removes support for the type parameter from the REST layer and the Java API. Backwards compatibility is maintained on the transport layer though.
The new added serialization test also uncovered a bug in the java API where the `ClusterSearchShardsRequest` could be created with no arguments, but the indices were required to be not null otherwise the request couldn't be serialized as `writeTo` would throw NPE. Fixed by setting a default value (empty array) for indices.
When a fatal error tragically closes an index writer, such an error
never makes its way to the uncaught exception handler. This prevents the
node from being torn down if an out of memory error or other fatal error
is thrown in the Lucene layer. This commit ensures that such events
bubble their way up to the uncaught exception handler.
Relates #21721
The PR #21694 was initially planned to go into v6.0.0 and v5.1.0. Due to another PR relying on this one though for backport to v5.0.2, #21694 must go to v5.0.2
as well. As such, the initial backward compatibility rules established by the PR must be changed to include v5.0.2 and above.
As part of #20925 and #21341 we added an "all-fields" mode to the
`query_string` and `simple_query_string`. This would expand the query to
all fields and automatically set `lenient` to true.
However, we should still allow a user to override the `lenient` flag to
whichever value they desire, should they add it in the request. This
commit does that.
When Elasticsearch starts, we go through some initialization before we
install a security manager. Yet, the JVM makes internal policy decisions
on the basis of whether or not a security manager is present. This
commit installs a security manager immediately on startup so that the
JVM always thinks a security manager is present when making such policy
decisions.
Relates #21716
Today we read a vint from the stream to allocate the size of an array up-front
before we start reading the values. This can be dangerous if for instance we read
from a corrupted stream or if some manipulated bytes are send for instance from
an attacker or a fuzzer. In most of the cases we can apply some best effort and
validate the array size to be _sane_ by ensuring we can at read at least N bytes
where N is the expected size of the array.
* Add support for merging custom meta data in tribe node
Currently, when any underlying cluster has custom metadata
(via plugin), tribe node does not store custom meta data in its
cluster state. This is because the tribe node has no idea how to
select the appropriate custom metadata from one or many custom
metadata (corresponding to the number of underlying clusters).
This change adds an interface that custom metadata implementations
can extend to add support for merging mulitple custom metadata of
the same type for storing in the tribe state.
Relates to #20544
Supersedes #20791
* Simplify updating tribe state
* Add tests for merging multiple custom metadata types in tribe node
* cleanup merging custom md logic in tribe service
Today it's not possible to add exceptions to the serialization layer
without breaking BWC. This commit adds the ability to specify the Version
an exception was added that allows to fall back not NotSerializableExceptionWrapper
if the exception is not present in the streams version.
Relates to #21656
TransportSearchAction optimizes the search_type in certain cases, when for instance we are searching against a single shard, or when there is only a suggest section in the request. That optimization is wrapped in a try catch, and when an exception happens we log it and ignore it. This may be a leftover from the past though, as no exception is expected to be thrown in that code block, hence if there is any exception we are probably better off bubbling it up rather than ignoring it.
The `lookupPrototype` method is not used anywhere. Seems like we rather use its `lookupProrotypeSafe` variant (which also throws exception if the prototype is not found) is always. This commit makes the safer variant the default one, by renaming it to "lookupPrototype" and removes the previous "unsafe" variant.
Today we call `writeByte` up to 3x per character in each string written via
`StreamOutput#writeString` this can have quite some overhead when strings
are long or many strings are written. This change adds a local buffer to
convert chars to bytes into the local buffer. Converted bytes are then
written via `writeBytes` instead reducing the overhead of this opertion.
Closes#21660
The overflows were happening in two places, the parsing of the template that
implicitly truncates the `order` when its value does not fall into the `integer`
range, and the comparator that sorts templates in ascending order, since it
returns `order2-order1`, which might overflow.
Closes#21622
* Fix highlighting on a stored keyword field
The highlighter converts stored keyword fields using toString().
Since the keyword fields are stored as utf8 bytes the conversion is broken.
This change uses BytesRef.utf8toString() to convert the field value in a valid string.
Fixes#21636
* Replace BytesRef#utf8ToString with MappedFieldType#valueForDisplay
If the node name is explicitly set it's not derived from the node ID
meaning that it doesn't immediately appear in the logs. While it can be
tracked down in other places, it would be easier for info purposes if it
just showed up explicitly. This commit adds the node ID to the logs,
whether or not the node name is set.
Relates #21673
Plugins are closed if they implement java.io.Closeable but this is not
clear from the plugin interface. This commit clarifies this by declaring
that Plugins implement java.io.Closeable and adding an empty
implementation to the base Plugin class.
Relates #21669
AbstractScopedSettings has the ability to only apply updates/deletes
to dynamic settings. The flag is currently not respected when a setting
is reset/deleted which causes static node settings to be reset if a non-dynamic
key is reset via `null` value.
Closes#21593
This commit moves several allocation decider related inner classes
into their own top-level class, in order to use more easily in
the allocation explain API. This commit also renames some of those
decision related classes to more suitable names.
This is simply a cosmetic change - no functionality changes with this
commit whatsoever.
To summarize the changes:
1. ShardAllocationDecision renamed to AllocateUnassignedDecision
2. RelocationDecision moved to a top-level class
3. MoveDecision moved to a top-level class
4. RebalanceDecision moved to a top-level class
5. ShardAllocationDecisionTests renamed to AllocateUnassignedDecisionTests
6. NodeRebalanceResult moved to a top-level class
7. ShardAllocationDecision#WeightedDecision moved to a top-level class and renamed to NodeAllocationResult.
The "test" task can complete its execution with a timeout exception before the "block-task" actually starts executing. The test thus has to wait for both to be
completed before checking that the updateTasksPerExecutor map has been properly cleaned up.
According to the docs and our own tests we accept an ids query without specified
types and default to all types in the index mapping in this case. This changes
the builder to reflect this by making the types no longer a required constructor
argument and changes the parser to reflect that.
A first step moving away from the current parsing to use the generalized
Objectparser and ConstructingObjectParser. This PR start by making use of it in
MatchAllQueryBuilder and IdsQueryBuilder.
* Fix compilation in Eclipse
I'm not sure what the bug is, but ecj doesn't like this expression
unless the type is set explicitly.
* Add comment explaining why no diamond operator
Luca Cavanna
Ryan Ernst
Igor Motov
Nik Everett
Jason Tedor
umeshdangat
Simon Willnauer
Clinton Gormley
Yannick Welsch
Jay Modi
Lee Hinman
Areek Zillur
Tanguy Leroux
Adrien Grand
Jim Ferenczi
David Roberts
javanna
Ali Beyad
Christoph Büscher