Commit Graph

1554 Commits

Author SHA1 Message Date
Alpar Torok 2b16d7bcf8
Backport testclusters all (#47565)
* Bwc testclusters all (#46265)

Convert all bwc projects to testclusters

* Fix bwc versions config

* WIP fix rolling upgrade

* Fix bwc tests on old versions

* Fix rolling upgrade
2019-10-04 16:12:53 +03:00
Alpar Torok 97a0b7dcbc Make All OS tests run on GCP instances (#46924)
This PR makes the necesary adaptations to the tests and adds a power shell script to
invoke the OS tests on GCP instances connected as CI workers.

Also noticed that logs were not being produced by the tests and that theses were not using log4j so fixed that too.

One of the difficulties in working on theses tests was that the tests just stalled with no indication where the problem is.
To ease with the debugging, after process explorer suggested that the tests are running some commands, we now have multiple timeouts: one for the tests ( which will generate a thread dump ) and one for individual commands ( that bails with the command being ran and output and error so far ) to make it easier to see what went wrong.

The tests were blocking because apparently the pipes to the sub-process were not closing, thus the threads were blocking on them and we were blocking indefinitely on the join. I'm not sure why this doesn't happen in vagrant, but we now properly deal with it.
2019-10-04 08:46:52 +03:00
Alpar Torok 392013cf05 Fix double start of test cluster 2019-10-03 18:14:58 +03:00
Alpar Torok 510d7d84b2 bwc tests: make sure first node is only started once 2019-10-03 16:34:20 +03:00
Alpar Torok 93032c9dea Fix bwcTests for versions < 6.5.0 2019-10-03 15:22:01 +03:00
Alpar Torok 0a14bb174f Remove eclipse conditionals (#44075)
* Remove eclipse conditionals

We used to have some meta projects with a `-test` prefix because
historically eclipse could not distinguish between test and main
source-sets and could only use a single classpath.
This is no longer the case for the past few Eclipse versions.

This PR adds the necessary configuration to correctly categorize source
folders and libraries.
With this change eclipse can import projects, and the visibility rules
are correct e.x. auto compete doesn't offer classes from test code or
`testCompile` dependencies when editing classes in `main`.

Unfortunately the cyclic dependency detection in Eclipse doesn't seem to
take the difference between test and non test source sets into account,
but since we are checking this in Gradle anyhow, it's safe to set to
`warning` in the settings. Unfortunately there is no setting to ignore
it.

This might cause problems when building since Eclipse will probably not
know the right order to build things in so more wirk might be necesarry.
2019-10-03 11:55:00 +03:00
Mark Vieira 6acc5ca8d1
Remove groovy test code from buildSrc (#47416) 2019-10-02 11:05:04 -07:00
Alpar Torok a032f9b2d5
Backport testclusters fix bwc (#47363)
* Add support for bwc for testclusters and convert full cluster restart (#45374)

* Testclusters fix bwc (#46740)

Additions to make testclsuters work with lather versions of ES

* Do common node config on bwc tests

Before this PR we always ever ran `ElasticsearchCluster.start` once, and
the common node config was never done.
This becomes apparent in upgrading from `6.x` to `7.x` as the new config
is missing preventing the cluster from starting.

* Do common node config on bwc tests

Before this PR we always ever ran `ElasticsearchCluster.start` once, and
the common node config was never done.
This becomes apparent in upgrading from `6.x` to `7.x` as the new config
is missing preventing the cluster from starting.

* Fix logic to pick up snapshot from 6.x

* Make sure ports are cleared

* Fix test

* Don't clear all the config as we rely on it

* Fix removal of keys
2019-10-02 14:37:00 +03:00
Mark Vieira ff15495b98
Remove empty buildSrc subproject (#47415) 2019-10-01 16:34:31 -07:00
Alpar Torok 3648481eb2 Replace heavy weight build-tools integ tests with a simple unit test. (#44056)
Remove heavy build-tool integ test.
Add a  unit test for the plugin builder plugin.

Closes #41256 #41256
2019-09-30 10:37:52 +03:00
Rory Hunter 1c7dc3a5bf Format Java source files automatically (#46745)
This commit adds a Java source formatter and checker into the build process.
This is not yet enabled for any sub-projects - to format and check a
sub-project, add its Gradle path into `build.gradle` and run:

    ./gradlew spotlessApply

to format, and:

    ./gradlew spotlessJavaCheck
    # or:
    ./gradlew precommit

to verify formatting.
2019-09-30 10:35:47 +03:00
Mark Vieira d0495b8f93
Track enabled test task candidate class files as task input (#47054) 2019-09-25 11:20:02 -07:00
James Rodewig 365aa30b7f
[DOCS] Remove support for `// CONSOLE` magic comment (#46936) (#47110)
#46180 added support for the `[source,console]`
language for snippets which should be tested.

This removes support for the `// CONSOLE` magic comment,
which serve a similar purpose.

Snippets that include the `// CONSOLE` magic comment will return
an exception.
2019-09-25 12:35:48 -04:00
Tim Brooks 6720c56bdd
Set netty system properties in BuildPlugin (#45881)
Currently in production instances of Elasticsearch we set a couple of
system properties by default. We currently do not apply all of these
system properties in tests. This commit applies these properties in the
tests.
2019-09-24 10:49:36 -06:00
Alpar Torok 5fd7505efc Testfixtures allow a single service only (#46780)
This PR adds some restrictions around testfixtures to make sure the same service ( as defiend in docker-compose.yml ) is not shared between multiple projects.
Sharing would break running with --parallel.

Projects can still share fixtures as long as each has it;s own service within.
This is still useful to share some of the setup and configuration code of the fixture.

Project now also have to specify a service name when calling useCluster to refer to a specific service.
If this is not the case all services will be claimed and the fixture can't be shared.
For this reason fixtures have to explicitly specify if they are using themselves ( fixture and tests in the same project ).
2019-09-23 14:13:49 +03:00
Mark Vieira ef0b75765b Add explicit build flag for experimenting with test execution cacheability (#42649)
* Add build flag for ignoring random test seed as task input

* Fix checkstyle violations
2019-09-23 14:13:49 +03:00
Mark Vieira f51b2592a4
Use composition instead of inheritance for extending Gradle plugins (#46888) 2019-09-20 14:51:36 -07:00
Alpar Torok 0294b8fc60 Fix SymbolicLinkPreservingTarIT on windows (#46908)
Broken path comparison on windows:
 https://scans.gradle.com/s/7y3zl23orfp7w/tests/nr3ff3qzwqzsw-xfh7lvi5orvq4
With the fix:
 https://gradle-enterprise.elastic.co/s/4dcc2nv3haaqw
2019-09-20 17:32:14 +03:00
Jason Tedor 4a9943a906
Bundle AdoptOpenJDK 13 (#46860)
This commit switches to use AdoptOpenJDK 13 for bundling the JDK,
instead of Oracle OpenJDK 13.
2019-09-19 08:56:17 -04:00
Mark Vieira 507d879b70
Disable bwc distribution caching in master branch (#46686)
This commit disables caching of BWC snapshot distributions in the "trunk" (aka master) branch.
Since the previous major release branches move quickly we rarely get cache hits for these 
tasks, and the artifacts themselves are very large. This means the overhead here is high and
savings basically zero. We conditionally disable task output caching in this scenario in CI to 
avoid excessive build cache overhead as well as causing too much turn in the cache itself which
would lead to lots of cache entry evictions.
2019-09-18 11:52:17 -07:00
Jason Tedor 7e71b2dbb3
Update bundled JDK to JDK 13 (#46795)
With the next minor release of Elasticsearch we will drop support for
JDK 12 and bump to JDK 13. While we want to use AdoptOpenJDK as the
bundled JDK, we are waiting for a release there. This commit moves to
OpenJDK 13 for now, and we will move to AdoptOpenJDK 13 as soon as its
available. Since macOS Catalina is delayed until October, we have some
time to update this.
2019-09-17 17:51:49 -07:00
Jason Tedor cd71d4a83b
Use AdoptOpenJDK as the bundled JDK (#46470) (#46785)
This commit teaches the build how to bundle AdoptOpenJDK with our
artifacts, and switches to AdoptOpenJDK as the bundled JDK. We keep the
functionality to also bundle Oracle OpenJDK distributions.
2019-09-17 13:40:35 -07:00
Alpar Torok 9229b87bd2 Upgrade to Gradle 5.6 (#45005)
* Upgrade to Gradle 5.6.2
2019-09-12 16:18:41 +03:00
Alpar Torok 9f2c5632fe Add support for tests.jvm.argline in testclusters (#46540)
We have CI jobs that make use of this for testing alternative garbadge
collectors.
2019-09-12 12:24:10 +03:00
Luca Cavanna e57756492a Update http-core and http-client dependencies (#46549)
Relates to #45808
Closes #45577
2019-09-12 09:45:29 +02:00
Jason Tedor 941c215d52
Favor java.vendor.version over java.vendor (#46615)
In some cases (for example some AdoptOpenJDK builds), the java.vendor is
mistakenly populated as "Oracle Corporation" while the real value is
under "java.vendor.version". Since "java.vendor.version" is mandatory
since JDK 10, this commit changes to use "java.vendor.version" as the
favored system property to find the JVM vendor, and we fallback to
"java.vendor" if this is not populated (as happens in some Oracle
builds). Ugh.
2019-09-11 15:20:46 -04:00
Mark Vieira e5d315f6e1
Ensure rest api specs are always copied before using test classpath (#46514)
(cherry picked from commit 45202903b4fea3a43f62594fd357ab3c98c3dd15)
2019-09-10 08:31:21 -07:00
Alpar Torok 31bee53fdd Fix the JVM we use for bwc nodes (#46314)
Before this change we would run bwc nodes with their bundled jdk if
these supported it, so the passed in runtime JDK was not honored.
This became obvius when running with FIPS.

Closes #41721
2019-09-09 15:24:58 +03:00
Armin Braun 7a9af874ad
Enable Debug Logging for Master and Coordination Packages (#46363) (#46374)
In order to track down #46091:
* Enables debug logging in REST tests for `master` and `coordination` packages
since we suspect that issues are caused by failed and then retried publications
2019-09-05 14:03:38 +02:00
Nik Everett d5ad86dad9
Build: Enable testing without magic comments (backports #46180) (#46325)
Previously we only turned on tests if we saw either `// CONSOLE` or
`// TEST`. These magic comments are difficult for the docs build to deal
with so it has moved away from using them where possible. We should
catch up. This adds another trigger to enable testing: marking a snippet
with the `console` language. It looks like this:

```
[source,console]
----
GET /
----
```

This saves a line which is nice, I guess. But it is more important to me
that this is consistent with the way the docs build works now.

Similarly this enables response testing when you mark a snippet with the
language `console-result`. That looks like:
```
[source,console-result]
----
{
  "result": "0.1"
}
----
```

`// TESTRESPONSE` is still available for situations like `// TEST`: when
the response isn't *in* the console-result language (like `_cat`) or
when you want to perform substitutions on the generated test.

Should unblock #46159.
2019-09-04 15:19:20 -04:00
Paul Sanwald 8bdbc7d9bf
Bump version from 7.4 to 7.5 (#46142) 2019-08-29 15:03:26 -04:00
Ryan Ernst 564b80303d Fix rest-api-spec dep for external plugins (#45949)
This commit fixes the maven coordinates for the rest-api-spec jar. It
was accidentally by #45107.

closes #45891
2019-08-28 10:54:03 -07:00
Nik Everett b3f76d6f22
Build: Support `console-result` language (#45937) (#45976)
This adds support for verifying that snippets with the `console-result`
language are valid json. It also switches the response snippets on the
`docs/get` page from `js` to `console-result` which will allow clients
to provide "alternatives" for them like they can now do with
`// CONSOLE` snippets.
2019-08-26 16:38:02 -04:00
William Brafford ff7fd9b9e2
Pass COMPUTERNAME env var to elasticsearch.bat (#45763)
* Pass COMPUTERNAME env var to elasticsearch.bat

When we run bin/elasticsearch with bash, we get a $HOSTNAME builtin that
contains the hostname of the machine the script is running on. When
there's no provided nodename, Elasticsearch uses the HOSTNAME to create
a nodename. On Windows, Powershell provides a $COMPUTERNAME variable for
the same purpose. CMD.EXE provides the same thing, except it's called
%COMPUTERNAME%. bin/elasticsearch.bat sets $HOSTNAME to the value of
$COMPUTERNAME. However, when testclusters invokes bin/elasticsearch.bat,
the COMPUTERNAME variable doesn't get passed in, leaving HOSTNAME null
and breaking an integration test on Windows.

This commit sets COMPUTERNAME in the environment so that our tests get
the value that Elasticsearch would have when bin/elasticsearch.bat is
invoked from the shell.

* Add null check to protect in non-Windows case

What good is it a developer to gain the whole Windows if they forfeit
their Unix? The value that fixes things on Windows is null on
Linux/Darwin, so let's null-check it.

* Override system hostnames for testclusters

Rather than relying on variable system behavior, let's just override
HOSTNAME and COMPUTERNAME and test for correct values in the integration
test that was originally failing.

* Rename constants for clarity

Since we are setting HOSTNAME and COMPUTERNAME regardless of whether the
tests are running on Windows or Linux, we shouldn't imply that constants
are only used in one case or the other.
2019-08-26 11:44:31 -04:00
Jason Tedor 9672ff388a
Consider artifact repositories backed by S3 secure (#45950)
Since credentials are required to access such a repository, and these
repositories are accessed over an encrypted protocol (https), this
commit adds support to consider S3-backed artifact repositories as
secure. Additionally, we add tests for this functionality.
2019-08-26 09:40:49 -04:00
Jason Tedor 599bf2d68b
Deprecate the pidfile setting (#45938)
This commit deprecates the pidfile setting in favor of node.pidfile.
2019-08-23 21:31:35 -04:00
Alexander Reelsen ecafe4f4ad Update joda to 2.10.3 (#45495) 2019-08-23 10:39:39 +02:00
Ryan Ernst 2b8d79c470 Add destructiveDistroTest meta task (#45762)
This commit adds a destructiveDistroTest task which depends on all of
the distribution specific destructive tasks, which can be used by CI.

closes #45769
2019-08-21 11:25:40 -07:00
Ryan Ernst 18fb63209b
Separate distro tests to be per distribution (#45565)
The java based distribution tests currently have a single Tests class
which encapsulates all of the tests for a particular distribution. The
test task in gradle then depends on all distributions being built, and
each individual tests class looks for the particular distribution it is
trying to test. This means that reproducing a single test failure
triggers all the distributions to be built, even though only one is
needed for the test.

This commit reworks the java distribution tests to pass in a particular
distribution to be tested, and changes the base test classes to be
actual test classes which have assumptions around which distributions
they operate on. For example, the archives tests will be skipped when
run with an rpm distribution, and vice versa for the package tests. This
makes reproduction much more granular. It also also better splitting up
tests around a particular use case. For example, all tests for systemd
behavior can be in one test class, and run independently of all tests
against rpm/deb distributions.
2019-08-20 13:12:15 -07:00
Alpar Torok c6b30b8883 Add input and outut tracking of built bwc versions (#45694)
* Add input and outut tracking of built bwc versions

This PR adds tracking of the bwc versions git has as input and all the
expected files as output.
The effect is that `gradlew` is not called at all when the git has
doesn't change and the version was allready built.
Previusly gradlew would be called for the bwc version and it would have
to configure the project and go trough up to date checks to figure out
that nothing changed.
This helps when working on bwc tests locally needing to run the test
multiple times.
This should also help in CI not re-build bwc versions across different
runs.

* Enable caching of bwc builds
2019-08-20 10:05:33 +03:00
Jason Tedor 200579bfce
Simplify finding jps (#45677)
This commit simplifies how we find jps to instead use built-in Gradle
functionality for the same.
2019-08-16 18:33:46 -04:00
Mark Vieira 0e375e6cd7
Disable test runner task when not running with testclusters (#45659) 2019-08-16 10:30:36 -07:00
Andrey Ershov dbc90653dc transport.publish_address should contain CNAME (#45626)
This commit adds CNAME reporting for transport.publish_address same way
it's done for http.publish_address.

Relates #32806
Relates #39970

(cherry picked from commit e0a2558a4c3a6b6fbfc6cd17ed34a6f6ef7b15a9)
2019-08-16 17:42:00 +02:00
Alpar Torok 59d378fc7b Toggle flag to tail cluster log on failure
The flag was toggled by accident in a refactoring.
This PR configures it so that we tail the cluster log on failure again.
2019-08-16 16:51:04 +03:00
Mark Vieira 1f6cd5c6e1
Fix backwards compatibility with pre-Java11 build-tools users (#45633) 2019-08-15 18:39:11 -07:00
Jason Tedor 6b72d42cb8
Read git revision directly from repository (#45547)
Today we shell out to git rev-parse to read the git revision. Forking
another process is slower than reading the revision directly. This
commit changes to directly read the git revision from the repository,
avoiding to fork another process.
2019-08-15 15:27:06 -04:00
Mark Vieira e31212b0aa
Remove leftover debugging logic (#45556) 2019-08-14 11:52:03 -07:00
Ryan Ernst 84bf98e9cd Fix bats distro test to build distributions (#45529)
The dependency on copying distributions was accidentally masked by an
earlier refactoring. This commit fixes the copyDistributions task to be
run before bats tests run.
2019-08-13 22:50:59 -07:00
Mark Vieira 1403a389f8
Fix remote cache misses for test tasks (#45521)
(cherry picked from commit 14486c73128feefc19d1c6a4cad4864d03ecaa9d)
2019-08-13 19:09:37 -07:00
Mark Vieira b17bf70e04
Fix remote cache misses for checkstyle tasks (#45512) 2019-08-13 16:03:11 -07:00
Ryan Ernst 80a3aeaef1
Split off upgrade and plugins bats tests (#45360) (#45508)
The bats tests currently require many additional artifacts to be built.
In addition to the current distributions, they need all the plugins to
be installed, as well as a randomly chosen bwc distribution. This commit
splits these two cases into their own bats task, so the dependencies do
not slow down other tasks like distroTests which do not need them.
2019-08-13 12:45:29 -07:00
Ryan Ernst 884f26a1dc
Make distro test plugin apply to the top level project (#45406) (#45481)
The distro test plugin was originally designed to be applied within each
subproject, per operating system we run in a VM with vagrant. However,
for efficiency, and also ease of having a single task to run in CI when
launching within individual OS VMs, having the "destructive" tasks in a
single place is more convenient. This commit reworks the distro test
plugin to be applied to the qa/vagrant project, which now creates only
the wrapper tasks in each of the subprojects for each vagrant VM.
2019-08-13 08:19:18 -07:00
Ryan Ernst 488673f636
Guard distro tests for pre oss in distribution filesnames (#45478)
Before #45064, the bats tests skipped the upgrade tests when the random
upgrade version is before 6.3.0. This commit restores that behavior.

closes #45476
2019-08-13 00:13:20 -07:00
Ryan Ernst 97efb6a403
Convert vagrant tests to per platform projects (#45064)
The vagrant based tests currently reside in a single project, creating
dozens of tasks to manage starting and stopping the vagrant VM along
with running java and bats tests within each image. This all-in-one
pattern makes parallelizing packaging tests difficult.

This commit rewrites the vagrant testing infrastructure to be
independent of the actual test runners, thus allowing each platform to
be handled in a separate subproject. Additionally, the java and bats
tests are changed to be run through a "destructive" gradle task, which
is run inside the VM. The combination of these will allow
parallelization both locally (through running several VMs at once) as
well as running the destructive tasks in CI machines dedicated to each
platform (thus removing the need for vagrant in CI).
2019-08-12 16:01:53 -07:00
Mark Vieira 2a1f0c7b4a
Fix location of reaper jar on Windows (#45464)
(cherry picked from commit 5021c5d8bf815ed11a2fe9cf4d3a2c2ed2e2e0b3)
2019-08-12 13:25:37 -07:00
Mark Vieira 491880edde
Fix build cache misses caused by embedded reaper jar (#45404)
(cherry picked from commit 788feced760bc2a5f453ebb07f1dbb288c6232b2)
2019-08-09 21:10:48 -07:00
Alpar Torok 634a070430 Restrict which tasks can use testclusters (#45198)
* Restrict which tasks can use testclusters

This PR fixes a problem between the interaction of test-clusters and
build cache.
Before this any task could have used a cluster without tracking it as
input.
With this change a new interface is introduced to track the tasks that
can use clusters and we do consider the cluster as input for all of
them.
2019-08-09 13:38:01 +03:00
Armin Braun a501d68f23
Upgrade to Netty 4.1.38 (#45132) (#45364)
* A number of fixes to buffer handling in the .37 and .38 -> we should stay up to date
2019-08-09 03:38:14 +02:00
Ryan Ernst 1794718e8e Make git revision loading lazy (#45358)
This commit makes the gitRevision property a lazy loaded value by
returning an Object implementing toString(). The Dockerfile template is
also changed to use groovy templates instead of the mavenfilter hack, so
converting to String will not happen until runtime.
2019-08-08 17:08:07 -07:00
Mark Vieira 214cbb28df
Fix for build runtime classpath instability (#45347)
(cherry picked from commit dee4ee2f0d4190ab54d0a4f0aa251d8c03e9db6d)
2019-08-08 12:41:17 -07:00
Mark Vieira 0ae103c40f
Avoid unnecessary eager creation of Gradle tasks (#45098) (#45310) 2019-08-08 10:50:09 -07:00
Alpar Torok b3076adae6 Fix reaper build failures on Windows (#45205)
We configure the service ID as the node's toString but this containes
characters that Windows doesn't like.
This PR fixes it by allowing only alphanumeric characters
2019-08-06 14:36:36 +03:00
Mark Vieira 87fb487299
Fix failed dependency resolution with external build-tools users (#45107) (#45219) 2019-08-05 15:52:46 -07:00
Jason Tedor 89b21102e5
Upgrade to JDK 12.0.2 (#45172)
This commit bumps the embedded JDK from 12.0.1 to 12.0.2.
2019-08-04 19:22:31 -04:00
Jason Tedor e708710f38
Simplify handling of git revision (#45175)
This commit simplifies the handling of git revision in the build. In
particular we remove pushing git revision through the generate build
info and print build info tasks as the git revision does not need to be
cached.
2019-08-04 11:09:11 -04:00
Jason Tedor 78aaf1b2d5
Lift build date to global build info (#45166)
This commit adds the build date to global build info mainly so that it
is accessible as an extension property.
2019-08-03 14:49:46 -04:00
Jason Tedor d78ecd9c09
Use the full hash in build info (#45163)
This commit switches to using the full hash to build into the JAR
manifest, which is used in node startup and the REST main action to
display the build hash.
2019-08-03 11:27:53 -04:00
Ryan Ernst 925bd4faa1
Use reaper process instead of shutdown hooks for testclusters (#44927)
Testclusters currently provides protection from clusters living past the
life of a build by adding a shutdown hook to java. While this works in
some cases, it does not cover all cases like where the daemon is killed
with SIGKILL.

To handle these other cases, this commit replaces the shutdown hooks with
a separate process (one per build) that manages reaping external services
if gradle dies.
2019-08-02 18:58:04 -07:00
Jason Tedor 5dc22e29c6
Add build hash to global build info (#45162)
This commit adds the commit hash to the global build info, and adds the
git revision as an extension. There are a couple motivations for this
change:
 - the current mechanism of getting the build hash does not work with
   git worktrees (because jgit does not understand them)
 - a follow-up will want to use the git revision when building the
   Docker images, so we want it available as an extension
 - it allows us to simplify our usage around the build hash as we no
   longer have to hack around silliness in the info-scm plugin

A follow-up will also stop using the short hash in the product build, so
that we use the full hash there. We already know that short hashes in
our codebase do collide, so we should move to the full hash to avoid
this problem.
2019-08-02 19:02:08 -04:00
Andrey Ershov 5a0bd696fc
Snapshot tool S3 cleanup 7.x backport (#44575)
Backport of #44551
2019-07-30 11:02:08 +02:00
Ignacio Vera 821f6f893b
Upgrade to Lucene 8.2.0 release (#44859) (#44892) 2019-07-26 08:14:59 +02:00
Mark Vieira aebfdf1477
Fix issue with Gradle daemons hanging indefinitely on shutdown (#44867) (#44878) 2019-07-25 15:33:51 -07:00
Ryan Ernst b3d8b39f29 Convert logging related gradle classes to java (#44771)
This commit converts the logging related classes (only used for vagrant)
to java from groovy.

relates #34459
2019-07-24 11:20:51 -07:00
Ryan Ernst 226a753e93
Restore setting up temp dir for windows service (#44541) (#44661)
In https://github.com/elastic/elasticsearch/pull/41913 setting up the
temp dir for ES was moved from the env script to individual cli scripts.
However, moving it to the windows service cli was missed. This commit
restores setting up the temp dir for the windows service control script.
2019-07-21 13:54:46 -07:00
Marios Trivyzas 8745d38dc8
Compile subset of build-tools classes with Java 8 target compatibility (#44560)
Backport of #43177 so that VersionProperties is Java 8 compatible and
can be used by https://github.com/elastic/elasticsearch-hadoop
to retrieve snapshot versions for Lucene.

(cherry picked from commit ec3ac9b62452f04ce44dea0a904a6e2b31dd8076)
2019-07-18 21:15:56 +03:00
Andrey Ershov ef6ddd15c6 Revert "Snapshot tool: S3 orphaned files
cleanup (#44551)"

This reverts commit 09edeeb3
2019-07-18 17:21:45 +02:00
Andrey Ershov 09edeeb38e Snapshot tool: S3 orphaned files cleanup (#44551)
A tool to work with snapshots.
Co-authored by @original-brownbear.
This commit adds snapshot tool and the single command cleanup, that
cleans up orphaned files for S3.
Snapshot tool lives in x-pack/snapshot-tool.

(cherry picked from commit fc4aed44dd975d83229561090f957a95cc76b287)
2019-07-18 16:38:00 +02:00
Alpar Torok 9f97319b3c Detect process third party audit being killed by OOM (#44269)
* Detect process third party audit being killed by OOM

It's very common for the third party audit to be killed by the OOM
killer when the system is running low on memory.
Since the forbidden APIs call is expected to fail, we were ignoring
these and incorrectly interpreting the partial output.
With this change we detect and provide a proper error message when this
happens.
2019-07-18 10:07:23 +03:00
Ignacio Vera eb348d2593
Upgrade to lucene-8.2.0-snapshot-6413aae226 (#44480) 2019-07-17 13:28:28 +02:00
david raistrick ae5a917efe Add clarification around TESTSETUP docs and error message (#43306) 2019-07-16 14:58:16 +02:00
William Brafford 673c63bb00 Restrict testCreateEmptyDirNoPermissions to Unix (#44282) (#44297)
The test EmptyDirTaskTests#testCreateEmptyDirNoPermissions may fail on
Windows. However, the test is only meaningful for Unix permissions
structures, so we should assume a Unix-family OS and skip the test on
Windows.

Fixes #44064
2019-07-16 08:48:12 +03:00
Ryan Ernst 7e06888bae
Convert testclusters to use distro download plugin (#44253) (#44362)
Test clusters currently has its own set of logic for dealing with
finding different versions of Elasticsearch, downloading them, and
extracting them. This commit converts testclusters to use the
DistributionDownloadPlugin.
2019-07-15 17:53:05 -07:00
Yogesh Gaikwad 91c342a888
fix and enable repository-hdfs secure tests (#44044) (#44199)
Due to recent changes are done for converting `repository-hdfs` to test
clusters (#41252), the `integTestSecure*` tasks did not depend on
`secureHdfsFixture` which when running would fail as the fixture
would not be available. This commit adds the dependency of the fixture
to the task.

The `secureHdfsFixture` is a `AntFixture` which is spawned a process.
Internally it waits for 30 seconds for the resources to be made available.
For my local machine, it took almost 45 seconds to be available so I have
added the wait time as an input to the `AntFixture` defaults to 30 seconds
 and set it to 60 seconds in case of secure hdfs fixture.

The integ test for secure hdfs was disabled for a long time and so
the changes done in #42090 to fix the tests are also done in this commit.
2019-07-12 12:44:01 +10:00
Mark Vieira 5698f4dc27
Pass tests.jvms system property to test tasks for maxParallelForks (#44237) 2019-07-11 12:41:32 -07:00
Mark Vieira 7a82106de6
Ignore test seed when flag is passed (#44234) 2019-07-11 12:30:21 -07:00
Nick Knize 374030a53f
Upgrade to lucene-8.2.0-snapshot-860e0be5378 (#44171) (#44184)
Upgrades lucene library to lucene-8.2.0-snapshot-860e0be5378
2019-07-11 09:17:22 -05:00
Alpar Torok 7ba18732f7 Run some REST tests against a cluster running in docker containers (#39515)
* Run REST tests against a cluster running on docker

Closes #38053
2019-07-11 15:28:33 +03:00
Alpar Torok 47ab2bda72 Improve how log is tailed in testclusters on failure (#40600)
* Improoce how log is tailed in testclusters on failure

- only print last few lines
- print all errors and warnings
- compact repeating errors and warnings
2019-07-11 15:04:08 +03:00
Mark Vieira 7c2e4b2857
[Backport] Enable caching of rest tests which use integ-test distribution (#44181) 2019-07-10 15:42:28 -07:00
Alpar Torok bde5802ad6 Test fixtures improovements (#43956)
* Test fixtures improovements

Don't disable some of the precommit tasks on fixtures.
This no longer makes sense now that a project can both produce and use a
fixture.

In order for this to be possible, had to add an additional configuration
to make JarHell class accessible to the task even if it's not a
dependency of the project and fix some of the third party audit fallout
from  #43671 which wasn't detected at the time due to the issue being
fixed here.

Closes #43918
2019-07-10 21:21:06 +03:00
Alpar Torok 0c8294e633 Make sure the clean task doesn't break test fixtures (#43641)
Use a dedicated fixture dir.
2019-07-08 17:58:27 +03:00
Adrien Grand 680edbe3f1
Bump current version to 7.4. (#43927) 2019-07-03 20:32:04 +02:00
Mark Vieira b5d47e5643
Upgrade to Gradle 5.5 (#43788) (#43832) 2019-07-01 11:54:58 -07:00
Alpar Torok b4303290d9 Testclusters: improove timeout handling (#43440) 2019-07-01 11:39:53 +03:00
Alpar Torok a38f509284 Testclusters: convert left-overs from checkPart1 (#43370)
* Testclusters: convert left-overs from checkPart1
2019-06-25 19:14:45 +03:00
Ryan Ernst 97cd417829 Distribution download plugin cleanup (#43461)
This commit addresses some post-review comments.

relates #43247
2019-06-24 09:34:53 -07:00
Alpar Torok b4f5882fbe Improove error on waiting for cluster to come up (#43416)
Always include all messages from exceptions to make the reason for
failure more visible.
2019-06-24 12:12:21 +03:00
Alpar Torok ea44da6069 Testclusters: conver remaining x-pack (#43335)
Convert x-pack tests
2019-06-24 12:07:42 +03:00
Alpar Torok 5a9c48369b TestClusters: Convert the security plugin (#43242)
* TestClusters: Convert the security plugin

This PR moves security tests to use TestClusters.
The TLS test required support in testclusters itself, so the correct
wait condition is configgured based on the cluster settings.

* PR review
2019-06-18 11:55:44 +03:00
Mark Vieira 74813360ab Fix inverted condition so we never cache rest integ tests 2019-06-18 07:33:19 +03:00