Commit Graph

1341 Commits

Author SHA1 Message Date
Alpar Torok 2f8bbce85d Clean up build tool dependencies (#41336)
We are no longer using these dependencies.

Relates to #41061 since the class that seems to be leaking is both part
of Gradle and the logging jar.
2019-04-19 09:22:38 +03:00
Alpar Torok 1f91759bc7 Improove the configuration time if the build (#41251)
This will help with reproduction lines and running tests form IDEs and
other operations that are quick and executed often enough for the
configuration time to matter.

Running Gradle with a FIPS JVM is not supproted, so if the runtime JVM
is the same one, no need to spend time checking for fips support.

Verification of the JAVA<version>_HOME env vars is now async and
parallel so it doesn't block configuration.
2019-04-19 08:58:43 +03:00
Mark Vieira 0227ac5690
Fix issue with subproject test task dependencies (#41321) (#41351) 2019-04-18 11:15:34 -07:00
Alpar Torok 44a0c468cf Clean up clusters between tests (#41187)
This PR adds additional cleanup when stopping the node.
The data dir is excepted because it gets reused in some tests.
Without this cleanup the number of working dir copies could grew to
exhaust all available disk space.
2019-04-18 08:58:47 +03:00
Tim Brooks 56c00eecbc
Remove string usages of old transport settings (#41207)
This is related to #36652. We intend to deprecate a number of transport
settings in 7.x and remove them in 8.0. This commit removes the string
usages of these settings.
2019-04-15 16:54:24 -06:00
Alpar Torok 636b9338a6 Fix the rpm and deb names for version starting with 7.0.0 (#41198)
With the 7.0.0 release, we switched to download the packages instead of
using locally built ones.
This PR fixes the artifact names to include the architecture as
introduced in the 7.0.0 release.
2019-04-15 18:52:19 +03:00
Alpar Torok ff64314b2b Simplify testclusters, don't allow cross project clusters (#40972)
* Simplify testclusters, don't allow cross project clusters
2019-04-12 12:41:52 +03:00
Mark Vieira 1287c7d91f
[Backport] Replace usages RandomizedTestingTask with built-in Gradle Test (#40978) (#40993)
* Replace usages RandomizedTestingTask with built-in Gradle Test (#40978)

This commit replaces the existing RandomizedTestingTask and supporting code with Gradle's built-in JUnit support via the Test task type. Additionally, the previous workaround to disable all tasks named "test" and create new unit testing tasks named "unitTest" has been removed such that the "test" task now runs unit tests as per the normal Gradle Java plugin conventions.

(cherry picked from commit 323f312bbc829a63056a79ebe45adced5099f6e6)

* Fix forking JVM runner

* Don't bump shadow plugin version
2019-04-09 11:52:50 -07:00
Jason Tedor 3e078b2026
Use bundled Java for all eligible versions in tests (#40928)
This commit sets the version to ensure that we use the bundled Java when
running integration tests for all eligible versions. In particular,
since we started bundling Java with 7.0.0, this commits sets said
version to 7.0.0.
2019-04-08 08:15:59 -04:00
Jay Modi f34663282c
Update apache httpclient to version 4.5.8 (#40875)
This change updates our version of httpclient to version 4.5.8, which
contains the fix for HTTPCLIENT-1968, which is a bug where the client
started re-writing paths that contained encoded reserved characters
with their unreserved form.
2019-04-05 13:48:10 -06:00
Martijn van Groningen 809a5f13a4
Make -try xlint warning disabled by default. (#40833)
Many gradle projects specifically use the -try exclude flag, because
there are many cases where auto-closeable resource ignore is never
referenced in body of corresponding try statement. Suppressing this
warning specifically in each case that it happens using
`@SuppressWarnings("try")` would be very verbose.

This change removes `-try` from any gradle project and adds it to the
build plugin. Also this change removes exclude flags from gradle projects
that is already specified in build plugin (for example -deprecation).

Relates to #40366
2019-04-05 08:02:26 +02:00
Tim Vernum 9d34164ce6
Add build utility to check cluster health over ssl (#40713)
By default, in integ tests we wait for the standalone cluster to start
by using the ant Get task to retrieve the cluster health endpoint.
However the ant task has no facilities for customising the trusted
CAs for a https resource, so if the integ test cluster has TLS enabled
on the http interface (using a custom CA) we need a separate utility
for that purpose.

Backport of: #40573
2019-04-04 21:44:03 +11:00
Alpar Torok 25944c4317 convert modules to use testclusters (#40804)
* convert modules to use testclusters
* Eliminate PluginPropertiesTask and move logic in plugin where it belongs
2019-04-04 11:45:40 +03:00
Ryan Ernst 9d785e2b69 Don't replace forbidden pattern failures when found (#40710)
This commit fixes a bug in forbidden patterns where the failures for a
file replace the failures from the previous files instead of extending
them.
2019-04-03 16:55:25 -07:00
Alpar Torok 95395938c8 Testclusters support for multi node clusters (#40699)
* Testclusters: introduce support for a multi node cluster
2019-04-03 19:35:36 +03:00
Jason Tedor 60000400db
Drop support for Ubuntu 14.04 (#40709)
With the 7.0.0 release, we are dropping support for Ubuntu 14.04. This
commit removes this OS from our testing infrastructure.
2019-04-02 08:51:40 -04:00
Ricky(道离) 74bf4b8292 If not set JAVA_HOME then use the JDK that Gradle was run with instead (for master) (#40622)
Extending the IDE behavior to the cli
2019-04-01 09:03:44 +03:00
Alpar Torok e8c0b53796 Add ability to mute and mute flaky fixture (#40630) 2019-03-29 12:10:04 +02:00
Alpar Torok 49f18811c3 Run the build integ test in parallel (#39788)
* Run the build integ test in parallel

Because the randomized runner lives in buildSrc, we run these tests with
the Gradle runner, and had no parallelism configured so far.

* Handle Windows and "auto" better
2019-03-29 08:11:20 +02:00
Alpar Torok d791e08932 Test fixtures krb5 (#40297)
Replaces the vagrant based kerberos fixtures with docker based test fixtures plugin.
The configuration is now entirely static on the docker side and no longer driven by Gradle,
also two different services are being configured since there are two different consumers of the fixture that can run in parallel and require different configurations.
2019-03-28 17:26:58 +02:00
Alpar Torok 524e0273ae Testclusters: convert plugin repository-s3 (#40399)
* Add support for setting and keystore settings
* system properties and env var config
* use testclusters for repository-s3
* Some cleanup of the build.gradle file for plugin-s3
* add runner {} to rest integ test task
2019-03-27 08:40:16 +02:00
Ryan Ernst f82f9423b7 Remove platform test command (#40280)
The platformTest gradle task was a packaging test meant to ensure unit
tests run on the various supported operating systems without relying on
CI to maintain a full matrix of platforms. Howevever, it never really
worked out as intended and is now additional code in our vagrant setup
to maintain. This commit removes the platformTest task.
2019-03-26 19:47:33 -07:00
Ryan Ernst e49ba6a9d9 Remove unused variable (#40468)
This was leftover from previous iterations of test clusters refactorings
2019-03-26 11:26:31 -07:00
Alpar Torok b5a8de9a7f Revert tmpdir (#40312)
* Revert "Configure TMP for test nodes on Windows (#39959)"
This reverts commit 97562a874fcb1f29fb05272ab860a0307e97d1aa.

* Configure a tmp dir without spaces
* Pass on TMP instead of changing it
2019-03-26 13:31:18 +02:00
Alpar Torok d50305bbee Upgrade to Gradle 5.3! (#40346)
Here are the highlights of this release:
 - Feature variants AKA "optional dependencies"
 - Type-safe accessors in Kotlin precompiled script plugins
 - Gradle Module Metadata 1.0

For more details see https://docs.gradle.org/5.3/release-notes.html
2019-03-26 13:23:40 +02:00
Alpar Torok cd4eac5fa4 Testclusters links (#39831)
* Create hardlinks instead of sync to save on disk space
2019-03-26 13:10:13 +02:00
Jason Tedor 3212557da4
Require JDK 12 for compilation (#40231)
This commit switches from requiring JDK 11 for compilation to requiring
JDK 12 for compilation.
2019-03-22 08:33:23 -04:00
Jason Tedor c2a566a64e
Upgrade bundled JDK and Docker images to JDK 12 (#40229)
This commit bumps the JDK used the bundled JDK and the Docker images to
JDK 12.
2019-03-19 23:45:07 -04:00
Alpar Torok cb29d3d8d0 Fix build-tools unit test on windows (#39986) 2019-03-19 08:07:33 +02:00
Ryan Ernst 8f09c77777 Add no-jdk distributions (#39882)
This commit adds a variant for every official distribution that omits
the bundled jdk. The "no-jdk" naming is conveyed through the package
classifier, alongside the platform. Package tests are also added for
each new distribution.
2019-03-15 00:55:57 -07:00
Jason Tedor d02bca1314
Upgrade the bouncycastle dependency to 1.61 (#40017)
This commit upgrades the bouncycastle dependency from 1.59 to 1.61.
2019-03-14 08:54:47 -04:00
Alpar Torok 9b8b47114e Configure TMP for test nodes on Windows (#39959)
This breaks on windows where TMP dir default to C:\Windows and startup
fails with a permission error.
I tried to create a tmp dir and pass in `TMP` env, but it lead to a
class not found error, and since testclusers is already independent of
the calling environment I stopped there.
2019-03-13 19:21:13 +02:00
Jim Ferenczi 7a7658707a
Upgrade to Lucene release 8.0.0 (#39998)
This commit upgrades to the GA release of Lucene 8

Closes #39640
2019-03-13 18:11:50 +01:00
Daniel Mitterdorfer 34006105ae
Allow IntegTestClusters to run with FIPS 140 JVMs (#39917) (#39923)
The changes in #39732 mean that nodes in the IntegTest clusters will
now run with whichever java version is defined as `runtime.java` and
not JAVA_HOME anymore.
This means that these nodes will also run in JVM with fips approved
mode enabled and as such, need to have access to the password for the
BCFKS keystore that is used as the default keystore/truststore.

This change sets the two necessary system properties.

Resolves #39855
2019-03-12 06:46:51 +01:00
Ryan Ernst 35dd52461f Make integ tests always use runtime java (#39732)
This commit ensures cluster formation always uses runtime java, never
attempting to use the bundled jdk, since the integ test zip does not
contain it.
2019-03-08 11:08:26 -08:00
Ryan Ernst 465343f12a
Bundle java in distributions (#38013)
* Bundle java in distributions

Setting up a jdk is currently a required external step when installing
elasticsearch. This is particularly problematic for the rpm/deb packages
as installing a jdk in the same package installation command does not
guarantee any order, so must be done in separate steps. Additionally,
JAVA_HOME must be set and often causes problems in selecting a correct
jdk when, for example, the system java is an older unsupported version.

This commit bundles platform specific openjdks into each distribution.
In addition to eliminating the issues above, it also presents future
possible improvements like using jlink to build jdk images only
containing modules that elasticsearch uses.

closes #31845
2019-03-08 11:04:18 -08:00
Jason Tedor 0250d554b6
Introduce forget follower API (#39718)
This commit introduces the forget follower API. This API is needed in cases that
unfollowing a following index fails to remove the shard history retention leases
on the leader index. This can happen explicitly through user action, or
implicitly through an index managed by ILM. When this occurs, history will be
retained longer than necessary. While the retention lease will eventually
expire, it can be expensive to allow history to persist for that long, and also
prevent ILM from performing actions like shrink on the leader index. As such, we
introduce an API to allow for manual removal of the shard history retention
leases in this case.
2019-03-07 11:08:45 -05:00
Alpar Torok 6c75a2f2b0 Testclusters: start using it for testing some plugins (#39693)
* enable testclusters for some plugins
2019-03-07 17:52:50 +02:00
Alpar Torok 7dcc191aa8 Fix verify versions (#39624)
closes #38708
2019-03-07 17:40:25 +02:00
Alpar Torok 0f89427eb6
Back port build changes from same version bwc tests (#39744)
* Back port build changes from #39102

This back-ports how versions are determined and bwc test are set up from
 #39102 without enabling the bwc from current version tests so it's
 easier/possible to backmerge future buld changes.
It's expected that the tets are lacking many of the required fixes in
this version to enable them.
2019-03-07 17:25:09 +02:00
Akshesh Doshi 4df26a2a90 Implement ConcatFilesTask from Groovy to Java issues#34459 (#37497)
relates  #344597
2019-03-07 15:30:54 +02:00
Alpar Torok 34ea84948c
Fix bwc tests failure to extract (#39619)
* Set correct packaging for older versions

Continue using zip packages for pre 7
Some other bwc fixes that hid behind this one.
Closes #39441  #39751
2019-03-07 09:07:11 +02:00
Alpar Torok 0f85182abe Testclusters: implement support to install plugins (#39116)
* methods to run bin script
* Add support for specifying and installing plugins
* Add OS specific distirbution support
* Add test to verify plugin installed
* Remove use of Gradle internal OperatingSystem
2019-03-04 09:15:23 +02:00
Alpar Torok 813351fe26 Un-mute and fix BuildExamplePluginsIT (#38899)
* Un-mute and fix BuildExamplePluginsIT

There doesn't seem to be anything wrong with the test iteself.
I think the failure were CI performance related, but while it was muted,
some failures managed to sneak in.

Closes #38784

* PR review
2019-03-04 08:50:55 +02:00
James Baiera 2a20e8e3fe
Remove missing variable from error message (#39321)
When test clusters are stood up, one of the steps in the wait task is to wait for 
ports files to appear. An exception throw was added if this were to time out
instead of failing with no information, but the exception text uses a missing 
variable which further obfuscates the problem.

Backports #39321
2019-03-01 13:27:57 -05:00
Jason Tedor 6d72d45e33
Use https to obtain Lucene snapshots (#39458)
This commit changes the protocol used to download Lucene snapshots.
2019-02-27 11:41:06 -05:00
Alan Woodward 71b8494181
Upgrade to lucene 8.0.0-snapshot-ff9509a8df (#39444)
Backport of #39350

Contains the following:

* LUCENE-8635: Move terms dictionary off-heap for non-primary-key fields in `MMapDirectory`
* LUCENE-8292: `TermsEnum` is fully abstract
* LUCENE-8679: Return WITHIN in `EdgeTree#relateTriangle` only when polygon and triangle share one edge
* LUCENE-8676: Nori tokenizer deals correctly with large buffers
* LUCENE-8697: `GraphTokenStreamFiniteStrings` better handles side paths with gaps
* LUCENE-8664: Add `equals` and `hashCode` to `TotalHits`
* LUCENE-8660: `TopDocsCollector` returns accurate hit counts if the total equals the threshold
* LUCENE-8654: `Polygon2D#relateTriangle` fix for when the polygon is inside the triangle
* LUCENE-8645: `Intervals#fixField` can merge intervals from different fields
* LUCENE-8585: Create jump-tables for DocValues at index time
2019-02-27 14:36:08 +00:00
Alpar Torok d955375e9c Don't do BWC packaing tests if disabled (#39397)
Packaging tests did not honor bwc tests being off.
This was also the reason for which we were building  the BWC versinons
even if the tests are off, so this closes #35347.
2019-02-27 09:25:29 +02:00
James Baiera e6a124c118
[Backport 7.x] Fix the OS sensing code in ClusterFormationTasks (#38457)
This fixes a bug in the sensing of the current OS family in the test cluster
formation code. Previously all builds would assume every environment 
was windows and would jump to using the windows zip build. This fixes 
the OS sensing code as well as updates some tests to account for 
different build flavors.

Backport of #38457
2019-02-25 14:39:34 -05:00
Mark Vieira 0f68814e04
Ensure global test seed is used for all random testing tasks (#38991) (#39195)
This commit fixes a bug which resulted in every RandomizedTestingTask
generating its own unique test seed rather than using the global test
seed generated by the the RandomizedTestingPlugin.

The issue didn't affect build invocations which explicitly ran with
-Dtest.seed. In those cases, the Ant junit task correctly uses the
global seed.

Since the Ant random testing target looks for an Ant project property
for determining the existing seed we now explicitly set this inside
RandomizedTestingPlugin. It just so happens that, incidentally, Ant will
inherit system properties, thus wy the -Dtest.seed mechanism continued
to work.
2019-02-21 08:52:14 -08:00