This commit updates the IndexAuditUpgradeIT test after backporting. The
backport fixes a bug that caused the node name field to be null, which
affected the expected number of nodes in the tests.
Original commit: elastic/x-pack-elasticsearch@9df99d8800
This commit is related to #27260. Currently there is a weird
relationship between channel contexts and nio channels. The selectors
use the context for read and writing. But the selector operates directly
on the nio channel for registering, closing, and connecting.
This commit works on improving this relationship. The selector operates
directly on the context which wraps the low level java.nio.channels. The
NioChannel class is simply an API that is used to interact with the
channel (sending messages from outside the selector event loop,
scheduling a close, adding listeners, etc). The context is only used
internally by the channel to implement these apis and by the selector to
perform these operations.
With this commit we reduce the maximum amount of memory that the javac
compiler can use from 1g to 512mb. While the build would succeed even
with 256mb, it influences compile time slightly negatively.
We have measured that the runtime overhead stays tolerable by running
the following command five times under repeatable conditions (i.e. we
execute `./gradlew clean`, then drop the caches and TRIM the disk):
```
./gradlew compileGroovy compileJava compileJava9Java\
compileTestGroovy compileTestJava compileGroovy
```
The results in seconds (as reported by Gradle) are:
* 1gb: avg: 253s, min: 252s, max: 256s
* 512mb: avg: 257s, min: 256s, max: 259s
Similarly to what has been done for s3 and azure, this commit removes
the repository settings `application_name` and `connect/read_timeout`
in favor of client settings. It introduce a GoogleCloudStorageClientSettings
class (similar to S3ClientSettings) and a bunch of unit tests for that,
it aligns the documentation to be more coherent with the S3 one, it
documents the connect/read timeouts that were not documented at all and
also adds a new client setting that allows to define a custom endpoint.
This commit makes AcknowledgedResponse implement ToXContentObject, so that the response knows how to print its own content out to XContent, which allows us to remove AcknowledgedRestListener.
This adds a new setting, `xpack.monitoring.collection.enabled`, and
disables it by default (`false`).
Original commit: elastic/x-pack-elasticsearch@4b3a5a1161
This commit moves the distribution specific tasks into the respective
archives and packages builds. The collocation of common and distribution
specific tasks make it much easier to reason about what is expected in a
particular distribution.
Although not frequently used in production, we make heavy use of the
FileRolesStore within integration tests. This change adds a little bit
more logging at INFO level when the roles.yml file is (re)loaded.
Original commit: elastic/x-pack-elasticsearch@bbacd46e28
These tests need to be skipped. They cause plugins to be loaded which
causes a child classloader to be opened. We do not want to add the
permissions to be able to close a classloader solely for these tests,
and the JARs created in the test can not be deleted on Windows until the
classloader is closed. Since this will not happen before test teardown,
the test will fail on Windows. So, we skip these tests.
This allows us to save a bit of code, but also adds more coverage as it tests serialization which was missing in some of the existing tests. Also it requires implementing equals/hashcode and we get the corresponding tests for them for free from the base test class.
This is related to #28662. It wraps the azure repository inputstream in
an inputstream that ensures `read` calls have socket permissions. This
is because the azure inputstream internally makes service calls.
* Pass InputStream when creating XContent parser
Rather than passing the raw `BytesReference` in when creating the xcontent
parser, this passes the StreamInput (which is an InputStream), this allows us to
decouple XContent from BytesReference.
This also removes the use of `commons.Booleans` so it doesn't require more
external commons classes.
Related to #28504
* Undo boolean removal
* Enhance deprecation javadoc
There is a bug in the for statement where we execute the JVM options
parser. The bug manfiests in the handling of paths with ) in the
name. The problem is this: we use a for statement to capture the output
of the JVM options parser. A for statement that executes a command
defers execution to cmd. There is this gem from the help:
1. If all of the following conditions are met, then quote characters
on the command line are preserved:
- no /S switch
- exactly two quote characters
- no special characters between the two quote characters,
where special is one of: &<>()@^|
- there are one or more whitespace characters between the
two quote characters
- the string between the two quote characters is the name
of an executable file.
2. Otherwise, old behavior is to see if the first character is
a quote character and if so, strip the leading character and
remove the last quote character on the command line, preserving
any text after the last quote character.
This means that the ) causes the quotes to be stripped which ruins
everything. This commit fixes this by delaying expansion of the paths.
Relates #28753
This pull request extracts in a dedicated class the request/response
logic that "emulates" a Google Cloud Storage service in our
repository-gcs tests.
The idea behind this is to make the logic more reusable. The class
MockHttpTransport has been renamed to MockStorage which now
only takes care of instantiating a Storage client and does the low-level
request/response plumbing needed by this client.
The "Google Cloud Storage" logic has been extracted from
MockHttpTransport and put in a new GoogleCloudStorageTestServer
that is now independent from the google client testing framework.
In order to check for the REST tests if triggering of watches with
security enabled works as expected, we have to add a watch and wait for
its background execution. In the REST tests the only wait is to wait for
this with a timeout. If the timeout is reached but the watch has not
been executed yet, the test will fail.
This commit replaces the YAML with a java based REST test, so that
helper methods like assertBusy() can be used and waiting for a watch to
be executed now works as expected.
relates elastic/x-pack-elasticsearch#3753
Original commit: elastic/x-pack-elasticsearch@fc39636ef7
The current toXContent serialization of a failed hipchat message writes
the same field called status twice and thus cannot be stored in the
watch history.
This commit ensures the field gets only written once.
relates elastic/x-pack-elasticsearch#3919
Original commit: elastic/x-pack-elasticsearch@fb499e8055
Add support version and version_type in ingest pipelines
Add support for setting document version and version type in set
processor of an ingest pipeline.
Nodes are reusing task ids after restart. So in some rare circumstances
the same task id might be assigned to the reindexing task stored by the
old cluster and the new task that is trying to retrieve the task
results. As a result, the get task request can timeout waiting on
itself. Since we already waited for the task to finish before restarting
the cluster, waiting for the task here doesn't make any sense to start
with.
Fixes#28732
This change makes the IndexAuditTrail update mappings for the current
index even when indexing locally. This is needed in a couple of
scenarios; the first being upgrades from 5x to 6x where the document
type was changed and the second is when upgrading minor versions and
a new field has been introduced. A new field was introduced in 6.2.
Additionally, writing tests for the rolling upgrade revealed a bug
where the actual node name was never being set as it was using the name
key instead of the node.name key.
relates elastic/x-pack-elasticsearch#3901
relates elastic/x-pack-elasticsearch#3718
Original commit: elastic/x-pack-elasticsearch@7374a62ff1
This commit removes some outdated logic in the SecurityTestUtils class
for writing files and creating directories. A long, long time ago there
was a global cluster for tests that was used across suites and because
of this there were calls to delete files if they already existed. The
global cluster has been removed, so we no longer need the code that
deletes the content of a directory if it already exists.
Additionally, the file writing used in SecurityTestUtils did not use
atomic moves when possible and this commit changes the code such that a
temp file is written and we try to atomically move it to the correct
path; if atomic moves are not supported a regular move is performed.
relates elastic/x-pack-elasticsearch#3912
Original commit: elastic/x-pack-elasticsearch@973fcfe2e1
* Remove log4j dependency from elasticsearch-core
This removes the log4j dependency from our elasticsearch-core project. It was
originally necessary only for our jar classpath checking. It is now replaced by
a `Consumer<String>` so that the es-core dependency doesn't have external
dependencies.
The parts of #28191 which were moved in conjunction (like `ESLoggerFactory` and
`Loggers`) have been moved back where appropriate, since they are not required
in the core jar.
This is tangentially related to #28504
* Add javadocs for `output` parameter
* Change @code to @link
Pruning tombstones is best effort and should not block if a key is currently
locked. This can cause a deadlock in rare situations if we switch of append
only optimization while heavily updating the same key in the engine
while the LiveVersionMap is locked. This is very rare since this code
patch only executed every 15 seconds by default since that is the interval
we try to prune the deletes in the version map.
Closes#28714
This commit fixes an issue with setting plugin.mandatory to include a
meta-plugin. The issue here is that the names that we collect are the
underlying plugins, not the meta-plugin. We should not use the
underlying plugins instead using the names of non-meta plugins and the
names of meta-plugins. This commit addresses this. The strategy here is
that when we look at the installed plugins on the filesystem, we keep
track of which ones are meta-plugins and carry this information up to
where check which plugins are installed against the mandatory plugins.
Relates #28710
We previously specified the -server flag to force the JVM to use the
server JVM. This is the default on all the systems that we support when
using a 64-bit JVM (and we no longer support 32-bit JVMs). There was
some trouble with this flag for the Windows service since procrun did
not understand what to do with it; as such, we had to filter this flag
out in the service. When we migrated to parsing JVM options in Java (via
the JVM options parser) we simplified this situation and removed
specifying the -server flag. This commit removes a leftover statement
that we are forcing the server JVM.
Relates #28738
Chris Earle
jaymode
Tim Brooks
Daniel Mitterdorfer
javanna
Tanguy Leroux
Jason Tedor
David Kyle
Luca Cavanna
Aaron Bull Schaefer
Ryan Ernst
Tim Vernum
Deb Adair
Lee Hinman
Lisa Cawley
Martijn van Groningen
Lokesh Mohanty
Alexander Reelsen
Yu
Igor Motov
Michael Basnight
Jay Modi
olcbean
Mikal
Jim Ferenczi
Simon Willnauer
Nhat Nguyen