When a setting is deprecated, if that setting is used repeatedly we
currently emit a deprecation warning every time the setting is used. In
cases like hitting settings endpoints over and over against a node with
a lot of deprecated settings, this can lead to excessive deprecation
warnings which can crush a node. This commit ensures that a given
setting only sees deprecation logging at most once.
Relates #25457
In #24477, a less verbose option was added to retrieve snapshot info via
GET /_snapshot/{repo}/{snapshots}. The point of adding this less
verbose option was so that if the repository is a cloud based one, and
there are many snapshots for which the snapshot info needed to be
retrieved, then each snapshot would require reading a separate snapshot
metadata file to pull out the necessary information. This can be costly
(performance and cost) on cloud based repositories, so a less verbose
option was added that only retrieves very basic information about each
snapshot that is all available in the index-N blob - requiring only one
read!
In order to display this less verbose snapshot info appropriately, logic
was added to not display those fields which could not be populated.
However, this broke integrators (e.g. ECE) that required these fields to
be present, even if empty. This commit is to return these fields in the
response, even if empty, if the verbose option is set.
This commit fixes a race condition in the node supplier used by the RemoteClusterConnection. The
node supplier stores an iterator over a set backed by a ConcurrentHashMap, but the get operation
of the supplier uses multiple methods of the iterator and is suceptible to a race between the
calls to hasNext() and next(). The test in this commit fails under the old implementation with a
NoSuchElementException. This commit adds a wrapper object over a set and a iterator, with all methods
being synchronized to avoid races. Modifications to the set result in the iterator being set to null
and the next retrieval creates a new iterator.
In SimpleNioTransportTests we assert that an IOException has a certain
message. This message appears that it is not dependible (and might
change based on platform).
Our other transport tests (mock and netty) do not make this assertion.
Instead they only assert on our application exception message. This
commit removes the IOException message assertion. And retains the
ConnectTransportException message assertion.
This commit changes how we determine if there were any remote indices that a search should have
been executed against. Previously, we used the list of remote shard iterators but if the remote
index pattern resolved to no indices there would be no remote shard iterators even though the
request specified remote indices. The map of remote cluster names to the original indices is used
instead so that we can determine if there were remote indices even when there are no remote shard
iterators.
Closes#25426
Expand `/_cat/nodes` with already present information about available disk space `diskAvail` (alias: `d`, `disk`) by:
* `diskTotal` (alias `dt`): total disk space
* `diskUsed` (alias `du`): used disk space (`diskTotal - diskAvail`)
* `diskUsedPercent` (alias `dup`): used disk space percentage
Note: The available disk space is the number of bytes available to the node's Java virtual machine. The size might be smaller than the real one. That means the used disk space (percentage) is larger.
Closes#21679
This commit introduces a nio based tcp transport into framework for
testing.
Currently Elasticsearch uses a simple blocking tcp transport for
testing purposes (MockTcpTransport). This diverges from production
where our current transport (netty) is non-blocking.
The point of this commit is to introduce a testing variant that more
closely matches the behavior of production instances.
This catches `AlreadyClosedException` during `stats` calls to avoid failing a `_nodes/stats` request because of the ignorable, concurrent index closure.
When relocating a shard before changing the state to relocated, we
verify that a relocation is a still taking place. Yet, this can throw an
exception if the relocation is in fact no longer valid. Sadly, we were
swallowing the exception in this situation. This commit allows such an
exception to bubble up after safely releasing resources.
We previously tried to maintain (while not formally supporting) 32-bit
support, although we never tested this anywhere in CI. Since we do not
formally support this, and 32-bit usage is very low, we have elected to
no longer maintain 32-bit support. This commit removes any implication
of 32-bit support.
Relates #25435
Changed `rescore`s to `rescore` requests as an backtick followed by the s character appears to be interpreted as an apostrophe which then leads to an unbalanced backtick for the next code span in the remainder of the paragraph
Closes#25443
The primary shard uses the GlobalCheckPointTracker to track local checkpoint information of recovering and started replicas in order to calculate the global checkpoint. As the tracker is updated through recoveries as well, it is easier to reason about the tracker if we can ensure that there are no concurrent recovery attempts for the same target shard (which can happen in case of network disconnects).
When a replica shard increases its primary term under the mandate of a new primary, it should also update its global checkpoint; this gives us the guarantee that its global checkpoint is at least as high as the new primary and gives a starting point for the primary/replica resync.
Relates to #25355, #10708
We already have these tests in InternalAggregationTestCase to check random insertions into the response xContent so that we don't fail on future changes in the response format. This change adds the same to AggregationsTests and runs on a whole aggregations tree. Unfortunately we need to exclude many places in the xContent from random insertion, but I added a long comment trying to explaine those.
This commit marks a failing test as awaits fix. The test is failing due
to a primary shard not knowing its own local checkpoint in the global
checkpoint tracker after recovery. If such a shard becomes primary after
promotion, and is then subsequently relocated, it can lead to a
violation of an assertion that when the primary context is transferred
the knowledge of all in-sync local checkpoints is consistent with the
global checkpoint on the relocation target.
Relates #25415
This commit removes the default path settings for data and logs. With
this change, we now ship the packages with these settings set in the
elasticsearch.yml configuration file rather than going through the
default.path.data and default.path.logs dance that we went through in
the past.
Relates #25408
Today we load plugins reflectively, looking for constructors that
conform to specific signatures. This commit tightens the reflective
operations here, not allowing plugins to have ambiguous constructors.
Relates #25405
This commit removes path.conf as a valid setting and replaces it with a
command-line flag for specifying a non-default path for configuration.
Relates #25392
This commit removes an abstraction that was introduced when introducing
the primary context. As this abstraction is used in exactly one place,
we simply make that abstraction local to its usage so that we do not
accumulate yet another general abstraction with exactly one usage.
Relates #25402
This commit updates some assertions in the primary context sealing test
after the restriction on updating allocation IDs from master and
updating global checkpoint on replica while sealed were removed.
* Introduce primary context
The target of a primary relocation is not aware of the state of the
replication group. In particular, it is not tracking in-sync and
initializing shards and their checkpoints. This means that after the
target shard is started, its knowledge of the replication group could
differ from that of the relocation source. In particular, this differing
view can lead to it computing a global checkpoint that moves backwards
after it becomes aware of the state of the entire replication
group. This commit addresses this issue by transferring a primary
context during relocation handoff.
* Fix test
* Add assertion messages
* Javadocs
* Barrier between marking a shard in sync and relocating
* Fix misplaced call
* Paranoia
* Better latch countdown
* Catch any exception
* Fix comment
* Fix wait for cluster state relocation test
* Update knowledge via upate local checkpoint API
* toString
* Visibility
* Refactor permit
* Push down
* Imports
* Docs
* Fix compilation
* Remove assertion
* Fix compilation
* Remove context wrapper
* Move PrimaryContext to new package
* Piping for cluster state version
This commit adds piping for the cluster state version to the global
checkpoint tracker. We do not use it yet.
* Remove unused import
* Implement versioning in tracker
* Fix test
* Unneeded public
* Imports
* Promote on our own
* Add tests
* Import
* Newline
* Update comment
* Serialization
* Assertion message
* Update stale comment
* Remove newline
* Less verbose
* Remove redundant assertion
* Tracking -> in-sync
* Assertions
* Just say no
Friends do not let friends block the cluster state update thread on
network operations.
* Extra newline
* Add allocation ID to assertion
* Rename method
* Another rename
* Introduce sealing
* Sealing tests
* One more assertion
* Fix imports
* Safer sealing
* Remove check
* Remove another sealed check
Christoph Büscher
Martijn van Groningen
olcbean
Jason Tedor
DeDe Morton
Ali Beyad
Jay Modi
Tim Brooks
Andreas Gebhardt
Chris Earle
Yannick Welsch
Robin Clarke
Marcus Wittig
Daniel Mitterdorfer
Russ Cam
Tal Levy
Lisa Cawley
Deb Adair
Ryan Ernst
Alexander Kazakov