Moves the test infrastructure away from using node.max_local_storage_nodes, allowing us in a
follow-up PR to deprecate this setting in 7.x and to remove it in 8.0.
This also changes the behavior of InternalTestCluster so that starting up nodes will not automatically
reuse data folders of previously stopped nodes. If this behavior is desired, it needs to be explicitly
done by passing the data path from the stopped node to the new node that is started.
This commit removes the jdk11 download in vagrant provisioning and
converts it to using the jdk downloader for the system jdk, and sets up
a separate jdk for use by the test (which will be converted to running
gradle in a followup).
This commit adds a guard around reading the spooled LoggedExec output.
It is possible the exec command did not output anything, and failed,
which would trigger a failure to read the output file.
This commit fixes the logging in LoggedExec which uses an in memory
buffer to read from a local reference, instead of with
getStandardOutput() of the Exec task. This is due to gradle internally
wrapping with a TeeOutputStream, breaking our cast.
Previously we used LoggedExec for running the internal bwc builds.
However, this had bad performance implications as all the output was
buffered into memory, thus we changed back to normal Exec. This commit
adds a `spoolOutput` setting to LoggedExec which can be used for
commands with large amounts of output, and switches the bwc builds to
use this flag.
* Fix slow sync test clustres artifacts task
The task was mistakenly adding a combinational explosion of task
actions all doing the same thing.
With this PR this is fixed and each version - distribution pair is only
extracted once.
I appologieze for the SSD wear.
* Look for configurations on the root project
* Add dependency on configurations
* This should be a `copy` so we don't blow away all the other distros
* Don't copy example plugin build directory in integration tests
This commit disables rhel 8 from being tested in vagrant packaging
tests. The vagrant image we use is beta release, but RHEL 8 was just
released, which has caused the package mirrors for the beta to stop
working.
This commit reworks the tests for jdk download to test the old and new
url pattern from oracle. Additionally it limits to one repository
created per version, based on the old or new pattern, and restricts
other repositories from trying to resolve jdks.
closes#41998
We currently download 3 variants of the same version of the jdk for
bundling into the distributions. Additionally, the vagrant images do
their own downloading. This commit moves the jdk downloading into a
utility gradle plugin. This will be used in a future PR by the packaging
tests.
The new plugin exposes a "jdks" project extension which allows creating
named jdks. Once the jdk version and platform are set for a named jdk,
the jdk object may be used as a lazy String for the jdk home path, or a
file collection for copying.
testclusters detect from settings that security is enabled
if a user is not specified using the DSL introduced in this PR, a default one is created
the appropriate wait conditions are used authenticating with the first user defined in the DSL ( or the default user ).
an example DSL to create a user is user username:"test_user" password:"x-pack-test-password" role: "superuser" all keys are optional and default to the values shown in this example
The run task is supposed to run elasticsearch with the given plugin or
module. However, for modules, this is most realistic if using the full
distribution. This commit changes the run setup to use the default or
oss as appropriate.
* Revert "Revert "Clean up clusters between tests (#41187)""
This reverts commit 9efc853aa668e285ede733d37b6fc7a0f4b02041.
* Remove the jdk directory to save space on bwc tests
This PR adresses the same concern as #41187 in a different way.
It removes only the JDK from the distribution once the cluster stops,
so we keep the same disk space requirements as before adding the JDK.
This is still a temporary measure, testclusters already deals with this
by doing the equivalent of `cp -l` instead of an actual copy.
Today we allow adding entries from a file or from a string, yet we
internally maintain this distinction such that if you try to add a value
from a file for a setting that expects a string or add a value from a
string for a setting that expects a file, you will have a bad time. This
causes a pain for operators such that for each setting they need to know
this difference. Yet, we do not need to maintain this distinction
internally as they are bytes after all. This commit removes that
distinction and includes logic to upgrade legacy keystores.
This commit bumps the bundled JDK to version 12.0.1. Note that we had to
add a new pattern here as Oracle has changed the source of the
builds. This commit will be backported to 6.7 in a different form to
bump the bundled JDK in the Docker images too.
hamcrest has some improvements in newer versions, like FileMatchers
that make assertions regarding file exists cleaner. This commit upgrades
to the latest version of hamcrest so we can start using new and improved
matchers.
To reduce configuration time, we fork some threads to compute the Java
version for the various configured Javas. However, as the number of
JAVA${N}_HOME variable increases, the current implementation creates as
many threads as there are such variables, which could be more than the
number of physical cores on the machine. It is not likely that we would
see benefits to trying to execute all of these once beyond the number of
physical cores (maybe simultaneous multi-threading helps though, who
knows. Therefore, this commit limits the parallelization here to the
number number of physical cores.
If no Java versions are set then when we size the executor thread pool
we end up with zero threads, which is illegal. This commit avoids that
problem by only starting the executor when needed.
ClusterFormationTasks auto configured these properties for clusters.
This PR adds FIPS specific configuration across all test clusters from
the main build script to prevent coupling betwwen testclusters and the
build plugin.
Closes#40904
This will help with reproduction lines and running tests form IDEs and
other operations that are quick and executed often enough for the
configuration time to matter.
Running Gradle with a FIPS JVM is not supproted, so if the runtime JVM
is the same one, no need to spend time checking for fips support.
Verification of the JAVA<version>_HOME env vars is now async and
parallel so it doesn't block configuration.
This PR adds additional cleanup when stopping the node.
The data dir is excepted because it gets reused in some tests.
Without this cleanup the number of working dir copies could grew to
exhaust all available disk space.
This is related to #36652. We intend to deprecate a number of transport
settings in 7.x and remove them in 8.0. This commit removes the string
usages of these settings.
With the 7.0.0 release, we switched to download the packages instead of
using locally built ones.
This PR fixes the artifact names to include the architecture as
introduced in the 7.0.0 release.
* Replace usages RandomizedTestingTask with built-in Gradle Test (#40978)
This commit replaces the existing RandomizedTestingTask and supporting code with Gradle's built-in JUnit support via the Test task type. Additionally, the previous workaround to disable all tasks named "test" and create new unit testing tasks named "unitTest" has been removed such that the "test" task now runs unit tests as per the normal Gradle Java plugin conventions.
(cherry picked from commit 323f312bbc829a63056a79ebe45adced5099f6e6)
* Fix forking JVM runner
* Don't bump shadow plugin version
This commit sets the version to ensure that we use the bundled Java when
running integration tests for all eligible versions. In particular,
since we started bundling Java with 7.0.0, this commits sets said
version to 7.0.0.
This change updates our version of httpclient to version 4.5.8, which
contains the fix for HTTPCLIENT-1968, which is a bug where the client
started re-writing paths that contained encoded reserved characters
with their unreserved form.
Many gradle projects specifically use the -try exclude flag, because
there are many cases where auto-closeable resource ignore is never
referenced in body of corresponding try statement. Suppressing this
warning specifically in each case that it happens using
`@SuppressWarnings("try")` would be very verbose.
This change removes `-try` from any gradle project and adds it to the
build plugin. Also this change removes exclude flags from gradle projects
that is already specified in build plugin (for example -deprecation).
Relates to #40366
By default, in integ tests we wait for the standalone cluster to start
by using the ant Get task to retrieve the cluster health endpoint.
However the ant task has no facilities for customising the trusted
CAs for a https resource, so if the integ test cluster has TLS enabled
on the http interface (using a custom CA) we need a separate utility
for that purpose.
Backport of: #40573
Jim Ferenczi
Jim Ferenczi
Jake Landis
Alpar Torok
Yannick Welsch
Alexander Reelsen
Ryan Ernst
Mark Vieira
Alpar Torok
Jason Tedor
Toby McLaughlin
Armin Braun
David Kyle
Tim Brooks
Jay Modi
Martijn van Groningen
Tim Vernum