When a node shuts down, `TransportService` moves to stopped state and
then closes connections. If a request is done in between, an exception
was thrown that was not retried in replication actions. Now throw a
wrapped `NodeClosedException` exception instead, which is correctly
handled in replication action. Fixed other usages too.
Relates #42612
* Move periodic job to ES repo
This change kickstarts the process of moving CI job definitions to this
repo.
* Added a minimal readme to provide pointers to the documentation
* Update .ci/README.md
Co-Authored-By: Rory Hunter <pugnascotia@users.noreply.github.com>
* Update .ci/README.md
Co-Authored-By: Rory Hunter <pugnascotia@users.noreply.github.com>
* point to main repo
* PR review
* Add link to JJBB
The team sometimes get questions around the use of `!foo` vs. `foo == false` in
PRs and reviews (e.g. #48615). This change adds a bullet point to CONTRIBUTING.md
to make expectations here clearer and gives us something to point to in case of
discussion.
This commit fixes an off-by-one bug in the AutoFollowIT test that causes
failures because the leaderIndices counter is incremented during the evaluation
of the leaderIndices.incrementAndGet() < 20 condition but the 20th index is
not created, making the final assertion not verified.
It also gives a bit more time for cluster state updates to be processed on the
follower cluster.
Closes#48982
Ensures that we always use the primary term established by the primary to index docs on the
replica. Makes the logic around replication less brittle by always using the operation primary
term on the replica that is coming from the primary.
Our documentation regarding FIPS 140 claimed that when using SAML
in a JVM that is configured in FIPS approved only mode, one could
not use encrypted assertions. This stemmed from a wrong
understanding regarding the compliance of RSA-OAEP which is used
as the key wrapping algorithm for encrypting the key with which the
SAML Assertion is encrypted.
However, as stated for instance in
https://downloads.bouncycastle.org/fips-java/BC-FJA-SecurityPolicy-1.0.0.pdf
RSA-OAEP is approved for key transport, so this limitation is not
effective.
This change removes the limitation from our FIPS 140 related
documentation.
This PR makes the following two fixes around updating flattened fields:
* Make sure that the new value for ignore_above is immediately taken into
affect. Previously we recorded the new value but did not use it when parsing
documents.
* Allow depth_limit to be updated dynamically. It seems plausible that a user
might want to tweak this setting as they encounter more data.
When using the move-to-step API, we should reread the phase JSON from
the latest version of the ILM policy. This allows a user to move to the
same step while re-reading the policy's latest version. For example,
when changing rollover criteria.
While manually messing around with some other things I discovered that
we only reread the policy when using the retry API, not the move-to-step
API. This commit changes the move-to-step API to always read the latest
version of the policy.
Reverts #48947 and fixes the issue orginally addressed by removing the assertion.
It turns out we can't simply pass empty shard generations to the snapshot finalization in the
BwC case as that results in no indices being added to the meta for the given snapshot since
we take the indices from the shard generations (even in the BwC case the `null` generations work
fine for this).
Closes#48983
This commit changes the ESMockAPIBasedRepositoryIntegTestCase so
that HttpHandler are now wrapped in order to log any exceptions that
could be thrown when executing the server side logic in repository
integration tests.
Backport of #48908
The enrich project doesn't have much history as all the other gradle projects,
so it makes sense to enable spotless for this gradle project.
Today in 6.x it is possible to add an index tombstone to the graveyard without
deleting the corresponding index metadata, because the deletion is slightly
deferred. If you shut down the node and upgrade to 7.x when in this state then
the node will fail to apply any cluster states, reporting
java.lang.IllegalStateException: Cannot delete index [...], it is still part of the cluster state.
This commit addresses this situation by skipping over any index metadata with a
corresponding tombstone, allowing this metadata to be cleaned up by the 7.x
node.
The previous approach did not work because the system property is passed
to Gradle but not to the tests JVM.
We shouldn't really pass this to the tests as we wouldn't want to have
differences.
This timeout being different might not be bad, but having a way to
differentiate could lead to others and it's best avoided.
The `HttpExportBulk` exporter is using a lot more memory than it needs to
by allocating buffers for serialization and IO:
* Remove copying of all bytes when flushing, instead use the stream wrapper
* Remove copying step turning the BAOS into a `byte[]`
* This also avoids the allocation of a single huge `byte[]` and instead makes use of the internal paging logic of the `BytesStreamOutput`
* Don't allocate a new BAOS for every document, just keep appending to a single BAOS
This commits sends the cluster name and discovery naode in the transport
level handshake response. This will allow us to stop sending the
transport service level handshake request in the 8.0-8.x release cycle.
It is necessary to start sending this in 7.x so that 8.0 is guaranteed
to be communicating with a version that sends the required information.
Currently the BulkProcessor class uses a single scheduler to schedule
flushes and retries. Functionally these are very different concerns but
can result in a dead lock. Specifically, the single shared scheduler
can kick off a flush task, which only finishes it's task when the bulk
that is being flushed finishes. If (for what ever reason), any items in
that bulk fails it will (by default) schedule a retry. However, that retry
will never run it's task, since the flush task is consuming the 1 and
only thread available from the shared scheduler.
Since the BulkProcessor is mostly client based code, the client can
provide their own scheduler. As-is the scheduler would require
at minimum 2 worker threads to avoid the potential deadlock. Since the
number of threads is a configuration option in the scheduler, the code
can not enforce this 2 worker rule until runtime. For this reason this
commit splits the single task scheduler into 2 schedulers. This eliminates
the potential for the flush task to block the retry task and removes this
deadlock scenario.
This commit also deprecates the Java APIs that presume a single scheduler,
and updates any internal code to no longer use those APIs.
Fixes#47599
Note - #41451 fixed the general case where a bulk fails and is retried
that can result in a deadlock. This fix should address that case as well as
the case when a bulk failure *from the flush* needs to be retried.
This commit fixes the names of the UBI-based Docker build contexts to
lift the ubi component of the name into the archive base name, instead
of the classifier.
* [ML] Add new geo_results.(actual_point|typical_point) fields for `lat_long` results (#47050)
[ML] Add new geo_results.(actual_point|typical_point) fields for `lat_long` results (#47050)
Related PR: https://github.com/elastic/ml-cpp/pull/809
* adjusting bwc version
We were tripping the assertion that the makes sure we only have empty `ShardGenerations` in `RepositoryData` in the BwC case because shard generations were passed to the `Repository` in the BwC case. Fixed by only generating empty shard gen for BwC snapshots in `SnapshotsService`.
The timeout was increased to 60s to allow this test more time to reach a
yellow state. However, the test will still on occasion fail even with the
60s timeout.
Related: #48381
Related: #48434
Related: #47950
Related: #40178
Backport of #48450.
Make a number of changes so that code in the `server` directory is more
resilient to automatic formatting. This covers:
* Reformatting multiline JSON to embed whitespace in the strings
* Move some comments around to they aren't auto-formatted to a strange
place. This also required moving some `&&` and `||` operators from the
end-of-line to start-of-line`.
* Add helper method `reformatJson()`, to strip whitespace from a JSON
document using XContent methods. This is sometimes necessary where
a test is comparing some machine-generated JSON with an expected
value.
Also, `HyperLogLogPlusPlus.java` is now excluded from formatting because it
contains large data tables that don't reformat well with the current settings,
and changing the settings would be worse for the rest of the codebase.
Backport of #48898.
We no longer configure distributions for prior versions for Docker. This
is because doing so prompts Gradle to try and resolve the Docker
dependencies, which doesn't work as they can't be downloaded via Ivy
(configured in DistributionDownloadPlugin). Since we need these for the
BATS upgrade tests, and those tests only cover .rpm and .deb, it's OK to
omit creating such distributions in the first place. We may need to
revisit this in the future, to allow upgrade testing using Docker
containers.
* Add links to infra-stats for scans generated in CI
It turns out we already gather system logs in infra-stats, and we have
system metrics too there.
This PR adds a links to the logs we gather for the host the build is
runnig on.
And a link to the host overview in the infrastructure app tuned to 5
minutes from before gradle started to 5 minutes after the scan was
generated.
* add buildFinished
Przemysław Witek
Henning Andersen
Alpar Torok
Tanguy Leroux
James Rodewig
István Zoltán Szabó
Christoph Büscher
Alan Woodward
Ioannis Kakavas
Yannick Welsch
Julie Tibshirani
Lee Hinman
Igor Motov
Armin Braun
David Roberts
Michael Basnight
Orhan Toy
Martijn van Groningen
David Turner
jimczi
Tim Brooks
Jake Landis
Jason Tedor
Benjamin Trent
Mark Vieira
Mark Tozzi
Patrick Maynard
Rory Hunter
Rafael Acevedo
Arne Welzel