Commit Graph

1641 Commits

Author SHA1 Message Date
Rafael Acevedo eb0d8f3383 update gradle to 5.6.4 (#48872) 2019-11-11 15:30:31 +02:00
Rory Hunter 35e21f85f3
Reenable Docker tests again (#48936)
Backport of #48898.

We no longer configure distributions for prior versions for Docker. This
is because doing so prompts Gradle to try and resolve the Docker
dependencies, which doesn't work as they can't be downloaded via Ivy
(configured in DistributionDownloadPlugin). Since we need these for the
BATS upgrade tests, and those tests only cover .rpm and .deb, it's OK to
omit creating such distributions in the first place. We may need to
revisit this in the future, to allow upgrade testing using Docker
containers.
2019-11-11 11:43:32 +00:00
Rory Hunter df16ff777e
Disable docker packaging tests again (#48896)
Backport of #48883.

Per elastic/infra#15864, the Elasticsearch CI images are failing due to
a packer_cache failure. This is because Gradle is trying to resolve
a `.docker` file through the Ivy repository, which doesn't work. Disable
the Docker tests again until we figure out the way forward.
2019-11-07 14:28:33 +00:00
Rory Hunter 24f7d4e83b
Add Docker packaging tests on 7.x (#48857)
Backport of #46599 and #47640. Add packaging tests for Docker.

* Introduce packaging tests for Docker (#46599)

Closes #37617. Add packaging tests for our Docker images, similar to what
we have for RPMs or Debian packages. This works by running a container and
probing it e.g. via `docker exec`. Test can also be run in Vagrant, by
exporting the Docker images to disk and loading them again in VMs. Docker
is installed via `Vagrantfile` in a selection of boxes.

* Only define Docker pkg tests if Docker is available (#47640)

Closes #47639, and unmutes tests that were muted in b958467.

The Docker packaging tests were being defined irrespective of whether
Docker was actually available in the current environment. Instead,
implement exclude lists so that in environments where Docker is not
available, no Docker packaging tests are defined. For CI hosts, the build
checks `.ci/dockerOnLinuxExclusions`. The Vagrant VMs can defined the
extension property `shouldTestDocker` property to opt-in to packaging
tests.

As part of this, define a seperate utility class for checking Docker,
and call that instead of defining checks in-line in BuildPlugin.groovy
2019-11-05 15:17:59 +00:00
Mark Vieira 6ab4645f4e
[7.x] Introduce type-safe and consistent pattern for handling build globals (#48818)
This commit introduces a consistent, and type-safe manner for handling
global build parameters through out our build logic. Primarily this
replaces the existing usages of extra properties with static accessors.
It also introduces and explicit API for initialization and mutation of
any such parameters, as well as better error handling for uninitialized
or eager access of parameter values.

Closes #42042
2019-11-01 11:33:11 -07:00
Mark Vieira 6e6b939fc3
Remove unnecessary logic for fixing generated POMs (#48721)
This commit eliminates some custom logic we have in place for post-hoc
cleanup of POM files generated by Gradle. There were to main issues this
logic was meant to address:

First, for dependencies marked as `transitive = false`, Gradle by
default creates a "wildcard" exclusion in the generated POM file. It
turns out that Ivy didn't handle these types of exclusions well, even
though they are perfectly valid and dealt with by Gradle and Maven as
expected. We've since confirmed that this issues is indeed resolved in
the most recent Ivy release (2.5.0-rc1) so going forward the suggestion
to folks consuming Elasticsearch dependencies with Ivy will be to use
this version.

Second, earlier versions of Gradle would incorrectly assign compile
dependencies to the "runtime" scope in the publish POM file. This could
cause issues if the dependencies were indeed needed at compile time
because their APIs were exposed. This has since been fixed and these
dependencies are correctly marked as "compile" scope in the POM.

Since these two issues have been resolved in their respective projects
we can eliminate this logic and all the supporting code, such as having
to create lots of "internal" configurations for tracking transitive
dependencies.
2019-10-31 10:20:00 -07:00
Alexander Reelsen 4ecf234617 Upgrade to joda 2.10.4 (#47805) 2019-10-31 14:49:50 +01:00
Alpar Torok f2d68934e8 Only log vagrant output on failure (#48402) 2019-10-30 10:40:44 +02:00
Alpar Torok d004a560df Upgrade to Gradle 5.6.3 (#48235)
This PR upgrades to [5.6.3](https://github.com/gradle/gradle/releases/tag/v5.6.3).
Nothing particualrily of interest for us, just keeping up to date.
2019-10-29 14:48:03 +02:00
Mark Vieira e5c6440a4f
Simplify usage of Gradle Shadow plugin (#48478) (#48597)
This commit simplifies and standardizes our usage of the Gradle Shadow
plugin to conform more to plugin conventions. The custom "bundle" plugin
has been removed as it's not necessary and performs the same function
as the Shadow plugin's default behavior with existing configurations.

Additionally, this removes unnecessary creation of a "nodeps" artifact,
which is unnecessary because by default project dependencies will in
fact use the non-shadowed JAR unless explicitly depending on the
"shadow" configuration.

Finally, we've cleaned up the logic used for unit testing, so we are
now correctly testing against the shadow JAR when the plugin is applied.
This better represents a real-world scenario for consumers and provides
better test coverage for incorrectly declared dependencies.

(cherry picked from commit 3698131109c7e78bdd3a3340707e1c7b4740d310)
2019-10-28 12:11:55 -07:00
Jason Tedor 97f48168d9
Bump bundled JDK to JDK 13.0.1+9 (#48587)
This commit bumps the bundled JDK to 13.0.1+9. Since AdoptOpenJDK did
not release 13.0.1+9 for Windows, this commit also enables that the
bundled JDK version can vary by platform.
2019-10-28 12:31:33 -04:00
Tim Brooks 45e42f4e18
Upgrade to Netty 4.1.43 (#48484)
With this update we can remove the mitigation in our custom allocator
which forces heap buffer allocations.
2019-10-25 10:17:25 -06:00
Przemyslaw Gomulka aaa6209be6
[7.x] [Java.time] Calculate week of a year with ISO rules BACKPORT(#48209) (#48349)
Reverting the change introducing IsoLocal.ROOT and introducing IsoCalendarDataProvider that defaults start of the week to Monday and requires minimum 4 days in first week of a year. This extension is using java SPI mechanism and defaults for Locale.ROOT only.
It require jvm property java.locale.providers to be set with SPI,COMPAT

closes #41670
backport #48209
2019-10-23 17:39:38 +02:00
Alpar Torok cb9f45ad78 Always publish a build scan in CI (#48348)
* Always publish a build scan in CI

This PR changes the build scan configuration to alwasy publisha  build
scan when running in our CI.

We should alkready be passing these env vars into the Vagrant VM so this
will make it produce a build scan too.

The old properties to accept build scan ToS on the public server are
thus no longer relevant and will be cleaned up from the Jenkins config
once this is merged.

* Pass env vars to vagrant VM
* Enable running in parallel in the VM
* Add job name and build nomber as custom values
2019-10-23 13:27:56 +03:00
Alpar Torok 144e8a0f1a Use an env var for the classpath of jar hell task (#48240)
The classpath for some project could outgrow the max allowed command
line on Windows. Using an env var is not fault proof, but give more
breathing room
2019-10-22 11:17:19 +03:00
Tim Brooks 547e399dbf
Remove option to enable direct buffer pooling (#48310)
This commit removes the option to change the netty system properties to
reenable the direct buffer pooling. It also removes the need for us to
disable the buffer pooling in the system properties file. Instead, we
programmatically craete an allocator that is used by our networking
layer.

This commit does introduce an Elasticsearch property which allows the
user to fallback on the netty default allocator. If they choose this
option, they can configure the default allocator how they wish using the
standard netty properties.
2019-10-21 19:15:50 -06:00
Mark Vieira 178204703a
Add 'javadoc' task to lifecycle check tasks (#48214) 2019-10-21 09:45:01 -07:00
Alpar Torok 7d085ffbd9 Switch to debug with server=n (#48188)
Before this change one needed to re-start debugging several times, as we
launched multiple JVMs in debug mode.
With this option the IDE has the option to re-launch and listen for
connections again leading for to a more pleasant experience.
2019-10-21 15:07:04 +03:00
Alpar Torok c903ac2376
Fix download of 6.x rpm and deb packages (#48228) 2019-10-21 10:03:44 +03:00
Ignacio Vera b1224fca8c
upgrade to Lucene-8.3.0-snapshot-25968e3b75e (#48227) 2019-10-21 08:21:09 +02:00
Mark Vieira befc44c145
Cache all rest tests tasks so long as they don't use shared clusters (#48161) 2019-10-17 07:30:55 -07:00
jimczi b2dc98562b Bump version to 7.6 2019-10-16 15:57:12 +02:00
Uwe Schindler f0cb43fb96 Update forbiddenapis to v2.7 (#47969) 2019-10-15 16:58:55 +03:00
Alpar Torok 6c9305dc78 Partial Revert "Convert RunTask to use testclusers, remove ClusterFormationTasks (#47572)"
This reverts the removal of the ClusterFormationTaks from
commit 36d018c909 so they are usable for a
bit longer in the hadoop build.
2019-10-15 16:43:46 +03:00
Tim Brooks 8814bf07f1
Upgrade to Netty 4.1.42 (#48015)
Upgrades the netty version.
2019-10-14 13:54:02 -06:00
Ryan Ernst dc8080e88a Add snapshot support to distribution download plugin (#47837)
The distribution download plugin which handles finding built
distributions for testing currently only knows how to find locally built
snapshots. When an external Elasticsearch plugin uses build-tools, these
snapshots do not exist. This commit extends the download plugin so it
pulls from the Elastic snapshots service when used outside of the
Elasticsearch repository.

closes #47123
2019-10-11 16:14:43 -07:00
William Brafford fe8767789a
GlobalBuildInfo plugin should search packed references for commit IDs (#47464) (#47936)
* GlobalBuildInfo plugin searches packed references

In recent versions of Git, references may be packed in a packed-refs
file. If this happens, Gradle will need to look in that file to find
build information.
2019-10-11 14:47:34 -04:00
Alpar Torok 5cbc96ccb6
Fix dependency info tasks (#47857)
We fixed warnings related to task input and outputs in #45098.
This particular input was not considered, a warning was present for it
and Gradle didn't use it as part of task inputs.
As soon as we fixed it Gradle started considering it an input and
enforced that it exists.

With this change we make it optional as the task can work both wih and
without this directory.
2019-10-11 11:11:52 -05:00
Alpar Torok ec91fa5df1 Make the run task honor tests.es properties (#47860)
* Make the run task honor tests.es properties

Closes #47797

With this PR we now again honor tests.es and tests.heap.size

* remove debuging
2019-10-11 04:38:32 +03:00
Mark Vieira 5825d2df83
Resolve more Gradle task validation warnings (#47825)
(cherry picked from commit 82e3d5ea31101f4c27e69162684c3aa4fb2193a4)

# Conflicts:
#	buildSrc/src/main/groovy/org/elasticsearch/gradle/doc/RestTestsFromSnippetsTask.groovy
2019-10-09 15:42:49 -07:00
Ryan Ernst 14b979a7bc Make internal project flag overrideable by tests (#47757)
In order to work with external elasticsearch plugins, some parts of
build-tools need to know when the current build is part of the
elasticsearch project or an external build. We identify these "internal"
builds by a marker in our buildSrc classpath. However, some build-tools
integ tests need to override this flag to test both the external and
internal behavior.

This commit moves the storage of the flag indicating whether we are
running in an internal build to global build info. This will allow
testkit projects to override the flag.
2019-10-09 12:01:30 -07:00
Ryan Ernst 813f44cec8 Move test seed setup to global build info (#47763)
The global build info plugin prints high level information about the
build, including the test seed. However, BuildPlugin sets up the test
seed, which creates an odd implicit dependency on it. This commit moves
the intialization of the testSeed property into the global build info.
2019-10-09 09:59:19 -07:00
Armin Braun edc3e9f0ab
Fix --debug-jvm Gradle Arg (#47773) (#47783)
This fixes the `--debug-jvm` arg to work again for
the `run` task.
Seems a recent refactoring of `RunTask` introduced
this obvious type.
2019-10-09 13:25:16 +02:00
Ryan Ernst 54c2aec38a Filter out special gradle threads from leak control (#47713)
This commit adds a thread filter for gradle unit tests which omits
threads gradle may create that we have no control over shutting down.
The current example of this is for project.exec which gradle pools.

closes #47417
2019-10-08 17:00:28 -07:00
Alpar Torok 36d018c909 Convert RunTask to use testclusers, remove ClusterFormationTasks (#47572)
* Convert RunTask to use testclusers, remove ClusterFormationTasks

This PR adds a new RunTask and a way for it to start a
testclusters cluster out of band and block on it to replace
the old RunTask that used ClusterFormationTasks.

With this we can now remove ClusterFormationTasks.
2019-10-08 14:43:29 +03:00
Alpar Torok bc85b22c1f
Complete testclusters backport (#47623)
* Use versions specific distribution folders so we don't need to clean up (#46539)

* Retry deleting distro dir on windows

When retarting the cluster we clean up old distribution files that might
still be in use by the OS.
Windows closes resources of ded processes async, so we do a couple of
retries to get arround it.

Closes #46014

* Avoid having to delete the distro folder.

* Remove the use of ClusterFormationTasks form RestTestTask (#47022)

This PR removes a use-case of the ClusterFormationTasks and converts a
project that flew under the radar so far.
There's probably more clean-up possible here, but for now the goal is
to be able to remove that code after `RunTask` is also updated.

* Migrate some 7.x only projects
2019-10-07 11:43:57 +03:00
Mark Vieira d966e5a9b9
Eliminate Gradle task input warnings (#47538)
(cherry picked from commit e86d40ff4576fb20c64fe88f01f13e201f3b948f)
2019-10-04 15:11:41 -07:00
Alpar Torok 2b16d7bcf8
Backport testclusters all (#47565)
* Bwc testclusters all (#46265)

Convert all bwc projects to testclusters

* Fix bwc versions config

* WIP fix rolling upgrade

* Fix bwc tests on old versions

* Fix rolling upgrade
2019-10-04 16:12:53 +03:00
Alpar Torok 97a0b7dcbc Make All OS tests run on GCP instances (#46924)
This PR makes the necesary adaptations to the tests and adds a power shell script to
invoke the OS tests on GCP instances connected as CI workers.

Also noticed that logs were not being produced by the tests and that theses were not using log4j so fixed that too.

One of the difficulties in working on theses tests was that the tests just stalled with no indication where the problem is.
To ease with the debugging, after process explorer suggested that the tests are running some commands, we now have multiple timeouts: one for the tests ( which will generate a thread dump ) and one for individual commands ( that bails with the command being ran and output and error so far ) to make it easier to see what went wrong.

The tests were blocking because apparently the pipes to the sub-process were not closing, thus the threads were blocking on them and we were blocking indefinitely on the join. I'm not sure why this doesn't happen in vagrant, but we now properly deal with it.
2019-10-04 08:46:52 +03:00
Alpar Torok 392013cf05 Fix double start of test cluster 2019-10-03 18:14:58 +03:00
Alpar Torok 510d7d84b2 bwc tests: make sure first node is only started once 2019-10-03 16:34:20 +03:00
Alpar Torok 93032c9dea Fix bwcTests for versions < 6.5.0 2019-10-03 15:22:01 +03:00
Alpar Torok 0a14bb174f Remove eclipse conditionals (#44075)
* Remove eclipse conditionals

We used to have some meta projects with a `-test` prefix because
historically eclipse could not distinguish between test and main
source-sets and could only use a single classpath.
This is no longer the case for the past few Eclipse versions.

This PR adds the necessary configuration to correctly categorize source
folders and libraries.
With this change eclipse can import projects, and the visibility rules
are correct e.x. auto compete doesn't offer classes from test code or
`testCompile` dependencies when editing classes in `main`.

Unfortunately the cyclic dependency detection in Eclipse doesn't seem to
take the difference between test and non test source sets into account,
but since we are checking this in Gradle anyhow, it's safe to set to
`warning` in the settings. Unfortunately there is no setting to ignore
it.

This might cause problems when building since Eclipse will probably not
know the right order to build things in so more wirk might be necesarry.
2019-10-03 11:55:00 +03:00
Mark Vieira 6acc5ca8d1
Remove groovy test code from buildSrc (#47416) 2019-10-02 11:05:04 -07:00
Alpar Torok a032f9b2d5
Backport testclusters fix bwc (#47363)
* Add support for bwc for testclusters and convert full cluster restart (#45374)

* Testclusters fix bwc (#46740)

Additions to make testclsuters work with lather versions of ES

* Do common node config on bwc tests

Before this PR we always ever ran `ElasticsearchCluster.start` once, and
the common node config was never done.
This becomes apparent in upgrading from `6.x` to `7.x` as the new config
is missing preventing the cluster from starting.

* Do common node config on bwc tests

Before this PR we always ever ran `ElasticsearchCluster.start` once, and
the common node config was never done.
This becomes apparent in upgrading from `6.x` to `7.x` as the new config
is missing preventing the cluster from starting.

* Fix logic to pick up snapshot from 6.x

* Make sure ports are cleared

* Fix test

* Don't clear all the config as we rely on it

* Fix removal of keys
2019-10-02 14:37:00 +03:00
Mark Vieira ff15495b98
Remove empty buildSrc subproject (#47415) 2019-10-01 16:34:31 -07:00
Alpar Torok 3648481eb2 Replace heavy weight build-tools integ tests with a simple unit test. (#44056)
Remove heavy build-tool integ test.
Add a  unit test for the plugin builder plugin.

Closes #41256 #41256
2019-09-30 10:37:52 +03:00
Rory Hunter 1c7dc3a5bf Format Java source files automatically (#46745)
This commit adds a Java source formatter and checker into the build process.
This is not yet enabled for any sub-projects - to format and check a
sub-project, add its Gradle path into `build.gradle` and run:

    ./gradlew spotlessApply

to format, and:

    ./gradlew spotlessJavaCheck
    # or:
    ./gradlew precommit

to verify formatting.
2019-09-30 10:35:47 +03:00
Mark Vieira d0495b8f93
Track enabled test task candidate class files as task input (#47054) 2019-09-25 11:20:02 -07:00
James Rodewig 365aa30b7f
[DOCS] Remove support for `// CONSOLE` magic comment (#46936) (#47110)
#46180 added support for the `[source,console]`
language for snippets which should be tested.

This removes support for the `// CONSOLE` magic comment,
which serve a similar purpose.

Snippets that include the `// CONSOLE` magic comment will return
an exception.
2019-09-25 12:35:48 -04:00
Tim Brooks 6720c56bdd
Set netty system properties in BuildPlugin (#45881)
Currently in production instances of Elasticsearch we set a couple of
system properties by default. We currently do not apply all of these
system properties in tests. This commit applies these properties in the
tests.
2019-09-24 10:49:36 -06:00
Alpar Torok 5fd7505efc Testfixtures allow a single service only (#46780)
This PR adds some restrictions around testfixtures to make sure the same service ( as defiend in docker-compose.yml ) is not shared between multiple projects.
Sharing would break running with --parallel.

Projects can still share fixtures as long as each has it;s own service within.
This is still useful to share some of the setup and configuration code of the fixture.

Project now also have to specify a service name when calling useCluster to refer to a specific service.
If this is not the case all services will be claimed and the fixture can't be shared.
For this reason fixtures have to explicitly specify if they are using themselves ( fixture and tests in the same project ).
2019-09-23 14:13:49 +03:00
Mark Vieira ef0b75765b Add explicit build flag for experimenting with test execution cacheability (#42649)
* Add build flag for ignoring random test seed as task input

* Fix checkstyle violations
2019-09-23 14:13:49 +03:00
Mark Vieira f51b2592a4
Use composition instead of inheritance for extending Gradle plugins (#46888) 2019-09-20 14:51:36 -07:00
Alpar Torok 0294b8fc60 Fix SymbolicLinkPreservingTarIT on windows (#46908)
Broken path comparison on windows:
 https://scans.gradle.com/s/7y3zl23orfp7w/tests/nr3ff3qzwqzsw-xfh7lvi5orvq4
With the fix:
 https://gradle-enterprise.elastic.co/s/4dcc2nv3haaqw
2019-09-20 17:32:14 +03:00
Jason Tedor 4a9943a906
Bundle AdoptOpenJDK 13 (#46860)
This commit switches to use AdoptOpenJDK 13 for bundling the JDK,
instead of Oracle OpenJDK 13.
2019-09-19 08:56:17 -04:00
Mark Vieira 507d879b70
Disable bwc distribution caching in master branch (#46686)
This commit disables caching of BWC snapshot distributions in the "trunk" (aka master) branch.
Since the previous major release branches move quickly we rarely get cache hits for these 
tasks, and the artifacts themselves are very large. This means the overhead here is high and
savings basically zero. We conditionally disable task output caching in this scenario in CI to 
avoid excessive build cache overhead as well as causing too much turn in the cache itself which
would lead to lots of cache entry evictions.
2019-09-18 11:52:17 -07:00
Jason Tedor 7e71b2dbb3
Update bundled JDK to JDK 13 (#46795)
With the next minor release of Elasticsearch we will drop support for
JDK 12 and bump to JDK 13. While we want to use AdoptOpenJDK as the
bundled JDK, we are waiting for a release there. This commit moves to
OpenJDK 13 for now, and we will move to AdoptOpenJDK 13 as soon as its
available. Since macOS Catalina is delayed until October, we have some
time to update this.
2019-09-17 17:51:49 -07:00
Jason Tedor cd71d4a83b
Use AdoptOpenJDK as the bundled JDK (#46470) (#46785)
This commit teaches the build how to bundle AdoptOpenJDK with our
artifacts, and switches to AdoptOpenJDK as the bundled JDK. We keep the
functionality to also bundle Oracle OpenJDK distributions.
2019-09-17 13:40:35 -07:00
Alpar Torok 9229b87bd2 Upgrade to Gradle 5.6 (#45005)
* Upgrade to Gradle 5.6.2
2019-09-12 16:18:41 +03:00
Alpar Torok 9f2c5632fe Add support for tests.jvm.argline in testclusters (#46540)
We have CI jobs that make use of this for testing alternative garbadge
collectors.
2019-09-12 12:24:10 +03:00
Luca Cavanna e57756492a Update http-core and http-client dependencies (#46549)
Relates to #45808
Closes #45577
2019-09-12 09:45:29 +02:00
Jason Tedor 941c215d52
Favor java.vendor.version over java.vendor (#46615)
In some cases (for example some AdoptOpenJDK builds), the java.vendor is
mistakenly populated as "Oracle Corporation" while the real value is
under "java.vendor.version". Since "java.vendor.version" is mandatory
since JDK 10, this commit changes to use "java.vendor.version" as the
favored system property to find the JVM vendor, and we fallback to
"java.vendor" if this is not populated (as happens in some Oracle
builds). Ugh.
2019-09-11 15:20:46 -04:00
Mark Vieira e5d315f6e1
Ensure rest api specs are always copied before using test classpath (#46514)
(cherry picked from commit 45202903b4fea3a43f62594fd357ab3c98c3dd15)
2019-09-10 08:31:21 -07:00
Alpar Torok 31bee53fdd Fix the JVM we use for bwc nodes (#46314)
Before this change we would run bwc nodes with their bundled jdk if
these supported it, so the passed in runtime JDK was not honored.
This became obvius when running with FIPS.

Closes #41721
2019-09-09 15:24:58 +03:00
Armin Braun 7a9af874ad
Enable Debug Logging for Master and Coordination Packages (#46363) (#46374)
In order to track down #46091:
* Enables debug logging in REST tests for `master` and `coordination` packages
since we suspect that issues are caused by failed and then retried publications
2019-09-05 14:03:38 +02:00
Nik Everett d5ad86dad9
Build: Enable testing without magic comments (backports #46180) (#46325)
Previously we only turned on tests if we saw either `// CONSOLE` or
`// TEST`. These magic comments are difficult for the docs build to deal
with so it has moved away from using them where possible. We should
catch up. This adds another trigger to enable testing: marking a snippet
with the `console` language. It looks like this:

```
[source,console]
----
GET /
----
```

This saves a line which is nice, I guess. But it is more important to me
that this is consistent with the way the docs build works now.

Similarly this enables response testing when you mark a snippet with the
language `console-result`. That looks like:
```
[source,console-result]
----
{
  "result": "0.1"
}
----
```

`// TESTRESPONSE` is still available for situations like `// TEST`: when
the response isn't *in* the console-result language (like `_cat`) or
when you want to perform substitutions on the generated test.

Should unblock #46159.
2019-09-04 15:19:20 -04:00
Paul Sanwald 8bdbc7d9bf
Bump version from 7.4 to 7.5 (#46142) 2019-08-29 15:03:26 -04:00
Ryan Ernst 564b80303d Fix rest-api-spec dep for external plugins (#45949)
This commit fixes the maven coordinates for the rest-api-spec jar. It
was accidentally by #45107.

closes #45891
2019-08-28 10:54:03 -07:00
Nik Everett b3f76d6f22
Build: Support `console-result` language (#45937) (#45976)
This adds support for verifying that snippets with the `console-result`
language are valid json. It also switches the response snippets on the
`docs/get` page from `js` to `console-result` which will allow clients
to provide "alternatives" for them like they can now do with
`// CONSOLE` snippets.
2019-08-26 16:38:02 -04:00
William Brafford ff7fd9b9e2
Pass COMPUTERNAME env var to elasticsearch.bat (#45763)
* Pass COMPUTERNAME env var to elasticsearch.bat

When we run bin/elasticsearch with bash, we get a $HOSTNAME builtin that
contains the hostname of the machine the script is running on. When
there's no provided nodename, Elasticsearch uses the HOSTNAME to create
a nodename. On Windows, Powershell provides a $COMPUTERNAME variable for
the same purpose. CMD.EXE provides the same thing, except it's called
%COMPUTERNAME%. bin/elasticsearch.bat sets $HOSTNAME to the value of
$COMPUTERNAME. However, when testclusters invokes bin/elasticsearch.bat,
the COMPUTERNAME variable doesn't get passed in, leaving HOSTNAME null
and breaking an integration test on Windows.

This commit sets COMPUTERNAME in the environment so that our tests get
the value that Elasticsearch would have when bin/elasticsearch.bat is
invoked from the shell.

* Add null check to protect in non-Windows case

What good is it a developer to gain the whole Windows if they forfeit
their Unix? The value that fixes things on Windows is null on
Linux/Darwin, so let's null-check it.

* Override system hostnames for testclusters

Rather than relying on variable system behavior, let's just override
HOSTNAME and COMPUTERNAME and test for correct values in the integration
test that was originally failing.

* Rename constants for clarity

Since we are setting HOSTNAME and COMPUTERNAME regardless of whether the
tests are running on Windows or Linux, we shouldn't imply that constants
are only used in one case or the other.
2019-08-26 11:44:31 -04:00
Jason Tedor 9672ff388a
Consider artifact repositories backed by S3 secure (#45950)
Since credentials are required to access such a repository, and these
repositories are accessed over an encrypted protocol (https), this
commit adds support to consider S3-backed artifact repositories as
secure. Additionally, we add tests for this functionality.
2019-08-26 09:40:49 -04:00
Jason Tedor 599bf2d68b
Deprecate the pidfile setting (#45938)
This commit deprecates the pidfile setting in favor of node.pidfile.
2019-08-23 21:31:35 -04:00
Alexander Reelsen ecafe4f4ad Update joda to 2.10.3 (#45495) 2019-08-23 10:39:39 +02:00
Ryan Ernst 2b8d79c470 Add destructiveDistroTest meta task (#45762)
This commit adds a destructiveDistroTest task which depends on all of
the distribution specific destructive tasks, which can be used by CI.

closes #45769
2019-08-21 11:25:40 -07:00
Ryan Ernst 18fb63209b
Separate distro tests to be per distribution (#45565)
The java based distribution tests currently have a single Tests class
which encapsulates all of the tests for a particular distribution. The
test task in gradle then depends on all distributions being built, and
each individual tests class looks for the particular distribution it is
trying to test. This means that reproducing a single test failure
triggers all the distributions to be built, even though only one is
needed for the test.

This commit reworks the java distribution tests to pass in a particular
distribution to be tested, and changes the base test classes to be
actual test classes which have assumptions around which distributions
they operate on. For example, the archives tests will be skipped when
run with an rpm distribution, and vice versa for the package tests. This
makes reproduction much more granular. It also also better splitting up
tests around a particular use case. For example, all tests for systemd
behavior can be in one test class, and run independently of all tests
against rpm/deb distributions.
2019-08-20 13:12:15 -07:00
Alpar Torok c6b30b8883 Add input and outut tracking of built bwc versions (#45694)
* Add input and outut tracking of built bwc versions

This PR adds tracking of the bwc versions git has as input and all the
expected files as output.
The effect is that `gradlew` is not called at all when the git has
doesn't change and the version was allready built.
Previusly gradlew would be called for the bwc version and it would have
to configure the project and go trough up to date checks to figure out
that nothing changed.
This helps when working on bwc tests locally needing to run the test
multiple times.
This should also help in CI not re-build bwc versions across different
runs.

* Enable caching of bwc builds
2019-08-20 10:05:33 +03:00
Jason Tedor 200579bfce
Simplify finding jps (#45677)
This commit simplifies how we find jps to instead use built-in Gradle
functionality for the same.
2019-08-16 18:33:46 -04:00
Mark Vieira 0e375e6cd7
Disable test runner task when not running with testclusters (#45659) 2019-08-16 10:30:36 -07:00
Andrey Ershov dbc90653dc transport.publish_address should contain CNAME (#45626)
This commit adds CNAME reporting for transport.publish_address same way
it's done for http.publish_address.

Relates #32806
Relates #39970

(cherry picked from commit e0a2558a4c3a6b6fbfc6cd17ed34a6f6ef7b15a9)
2019-08-16 17:42:00 +02:00
Alpar Torok 59d378fc7b Toggle flag to tail cluster log on failure
The flag was toggled by accident in a refactoring.
This PR configures it so that we tail the cluster log on failure again.
2019-08-16 16:51:04 +03:00
Mark Vieira 1f6cd5c6e1
Fix backwards compatibility with pre-Java11 build-tools users (#45633) 2019-08-15 18:39:11 -07:00
Jason Tedor 6b72d42cb8
Read git revision directly from repository (#45547)
Today we shell out to git rev-parse to read the git revision. Forking
another process is slower than reading the revision directly. This
commit changes to directly read the git revision from the repository,
avoiding to fork another process.
2019-08-15 15:27:06 -04:00
Mark Vieira e31212b0aa
Remove leftover debugging logic (#45556) 2019-08-14 11:52:03 -07:00
Ryan Ernst 84bf98e9cd Fix bats distro test to build distributions (#45529)
The dependency on copying distributions was accidentally masked by an
earlier refactoring. This commit fixes the copyDistributions task to be
run before bats tests run.
2019-08-13 22:50:59 -07:00
Mark Vieira 1403a389f8
Fix remote cache misses for test tasks (#45521)
(cherry picked from commit 14486c73128feefc19d1c6a4cad4864d03ecaa9d)
2019-08-13 19:09:37 -07:00
Mark Vieira b17bf70e04
Fix remote cache misses for checkstyle tasks (#45512) 2019-08-13 16:03:11 -07:00
Ryan Ernst 80a3aeaef1
Split off upgrade and plugins bats tests (#45360) (#45508)
The bats tests currently require many additional artifacts to be built.
In addition to the current distributions, they need all the plugins to
be installed, as well as a randomly chosen bwc distribution. This commit
splits these two cases into their own bats task, so the dependencies do
not slow down other tasks like distroTests which do not need them.
2019-08-13 12:45:29 -07:00
Ryan Ernst 884f26a1dc
Make distro test plugin apply to the top level project (#45406) (#45481)
The distro test plugin was originally designed to be applied within each
subproject, per operating system we run in a VM with vagrant. However,
for efficiency, and also ease of having a single task to run in CI when
launching within individual OS VMs, having the "destructive" tasks in a
single place is more convenient. This commit reworks the distro test
plugin to be applied to the qa/vagrant project, which now creates only
the wrapper tasks in each of the subprojects for each vagrant VM.
2019-08-13 08:19:18 -07:00
Ryan Ernst 488673f636
Guard distro tests for pre oss in distribution filesnames (#45478)
Before #45064, the bats tests skipped the upgrade tests when the random
upgrade version is before 6.3.0. This commit restores that behavior.

closes #45476
2019-08-13 00:13:20 -07:00
Ryan Ernst 97efb6a403
Convert vagrant tests to per platform projects (#45064)
The vagrant based tests currently reside in a single project, creating
dozens of tasks to manage starting and stopping the vagrant VM along
with running java and bats tests within each image. This all-in-one
pattern makes parallelizing packaging tests difficult.

This commit rewrites the vagrant testing infrastructure to be
independent of the actual test runners, thus allowing each platform to
be handled in a separate subproject. Additionally, the java and bats
tests are changed to be run through a "destructive" gradle task, which
is run inside the VM. The combination of these will allow
parallelization both locally (through running several VMs at once) as
well as running the destructive tasks in CI machines dedicated to each
platform (thus removing the need for vagrant in CI).
2019-08-12 16:01:53 -07:00
Mark Vieira 2a1f0c7b4a
Fix location of reaper jar on Windows (#45464)
(cherry picked from commit 5021c5d8bf815ed11a2fe9cf4d3a2c2ed2e2e0b3)
2019-08-12 13:25:37 -07:00
Mark Vieira 491880edde
Fix build cache misses caused by embedded reaper jar (#45404)
(cherry picked from commit 788feced760bc2a5f453ebb07f1dbb288c6232b2)
2019-08-09 21:10:48 -07:00
Alpar Torok 634a070430 Restrict which tasks can use testclusters (#45198)
* Restrict which tasks can use testclusters

This PR fixes a problem between the interaction of test-clusters and
build cache.
Before this any task could have used a cluster without tracking it as
input.
With this change a new interface is introduced to track the tasks that
can use clusters and we do consider the cluster as input for all of
them.
2019-08-09 13:38:01 +03:00
Armin Braun a501d68f23
Upgrade to Netty 4.1.38 (#45132) (#45364)
* A number of fixes to buffer handling in the .37 and .38 -> we should stay up to date
2019-08-09 03:38:14 +02:00
Ryan Ernst 1794718e8e Make git revision loading lazy (#45358)
This commit makes the gitRevision property a lazy loaded value by
returning an Object implementing toString(). The Dockerfile template is
also changed to use groovy templates instead of the mavenfilter hack, so
converting to String will not happen until runtime.
2019-08-08 17:08:07 -07:00
Mark Vieira 214cbb28df
Fix for build runtime classpath instability (#45347)
(cherry picked from commit dee4ee2f0d4190ab54d0a4f0aa251d8c03e9db6d)
2019-08-08 12:41:17 -07:00
Mark Vieira 0ae103c40f
Avoid unnecessary eager creation of Gradle tasks (#45098) (#45310) 2019-08-08 10:50:09 -07:00
Alpar Torok b3076adae6 Fix reaper build failures on Windows (#45205)
We configure the service ID as the node's toString but this containes
characters that Windows doesn't like.
This PR fixes it by allowing only alphanumeric characters
2019-08-06 14:36:36 +03:00
Mark Vieira 87fb487299
Fix failed dependency resolution with external build-tools users (#45107) (#45219) 2019-08-05 15:52:46 -07:00
Jason Tedor 89b21102e5
Upgrade to JDK 12.0.2 (#45172)
This commit bumps the embedded JDK from 12.0.1 to 12.0.2.
2019-08-04 19:22:31 -04:00
Jason Tedor e708710f38
Simplify handling of git revision (#45175)
This commit simplifies the handling of git revision in the build. In
particular we remove pushing git revision through the generate build
info and print build info tasks as the git revision does not need to be
cached.
2019-08-04 11:09:11 -04:00
Jason Tedor 78aaf1b2d5
Lift build date to global build info (#45166)
This commit adds the build date to global build info mainly so that it
is accessible as an extension property.
2019-08-03 14:49:46 -04:00
Jason Tedor d78ecd9c09
Use the full hash in build info (#45163)
This commit switches to using the full hash to build into the JAR
manifest, which is used in node startup and the REST main action to
display the build hash.
2019-08-03 11:27:53 -04:00
Ryan Ernst 925bd4faa1
Use reaper process instead of shutdown hooks for testclusters (#44927)
Testclusters currently provides protection from clusters living past the
life of a build by adding a shutdown hook to java. While this works in
some cases, it does not cover all cases like where the daemon is killed
with SIGKILL.

To handle these other cases, this commit replaces the shutdown hooks with
a separate process (one per build) that manages reaping external services
if gradle dies.
2019-08-02 18:58:04 -07:00
Jason Tedor 5dc22e29c6
Add build hash to global build info (#45162)
This commit adds the commit hash to the global build info, and adds the
git revision as an extension. There are a couple motivations for this
change:
 - the current mechanism of getting the build hash does not work with
   git worktrees (because jgit does not understand them)
 - a follow-up will want to use the git revision when building the
   Docker images, so we want it available as an extension
 - it allows us to simplify our usage around the build hash as we no
   longer have to hack around silliness in the info-scm plugin

A follow-up will also stop using the short hash in the product build, so
that we use the full hash there. We already know that short hashes in
our codebase do collide, so we should move to the full hash to avoid
this problem.
2019-08-02 19:02:08 -04:00
Andrey Ershov 5a0bd696fc
Snapshot tool S3 cleanup 7.x backport (#44575)
Backport of #44551
2019-07-30 11:02:08 +02:00
Ignacio Vera 821f6f893b
Upgrade to Lucene 8.2.0 release (#44859) (#44892) 2019-07-26 08:14:59 +02:00
Mark Vieira aebfdf1477
Fix issue with Gradle daemons hanging indefinitely on shutdown (#44867) (#44878) 2019-07-25 15:33:51 -07:00
Ryan Ernst b3d8b39f29 Convert logging related gradle classes to java (#44771)
This commit converts the logging related classes (only used for vagrant)
to java from groovy.

relates #34459
2019-07-24 11:20:51 -07:00
Ryan Ernst 226a753e93
Restore setting up temp dir for windows service (#44541) (#44661)
In https://github.com/elastic/elasticsearch/pull/41913 setting up the
temp dir for ES was moved from the env script to individual cli scripts.
However, moving it to the windows service cli was missed. This commit
restores setting up the temp dir for the windows service control script.
2019-07-21 13:54:46 -07:00
Marios Trivyzas 8745d38dc8
Compile subset of build-tools classes with Java 8 target compatibility (#44560)
Backport of #43177 so that VersionProperties is Java 8 compatible and
can be used by https://github.com/elastic/elasticsearch-hadoop
to retrieve snapshot versions for Lucene.

(cherry picked from commit ec3ac9b62452f04ce44dea0a904a6e2b31dd8076)
2019-07-18 21:15:56 +03:00
Andrey Ershov ef6ddd15c6 Revert "Snapshot tool: S3 orphaned files
cleanup (#44551)"

This reverts commit 09edeeb3
2019-07-18 17:21:45 +02:00
Andrey Ershov 09edeeb38e Snapshot tool: S3 orphaned files cleanup (#44551)
A tool to work with snapshots.
Co-authored by @original-brownbear.
This commit adds snapshot tool and the single command cleanup, that
cleans up orphaned files for S3.
Snapshot tool lives in x-pack/snapshot-tool.

(cherry picked from commit fc4aed44dd975d83229561090f957a95cc76b287)
2019-07-18 16:38:00 +02:00
Alpar Torok 9f97319b3c Detect process third party audit being killed by OOM (#44269)
* Detect process third party audit being killed by OOM

It's very common for the third party audit to be killed by the OOM
killer when the system is running low on memory.
Since the forbidden APIs call is expected to fail, we were ignoring
these and incorrectly interpreting the partial output.
With this change we detect and provide a proper error message when this
happens.
2019-07-18 10:07:23 +03:00
Ignacio Vera eb348d2593
Upgrade to lucene-8.2.0-snapshot-6413aae226 (#44480) 2019-07-17 13:28:28 +02:00
david raistrick ae5a917efe Add clarification around TESTSETUP docs and error message (#43306) 2019-07-16 14:58:16 +02:00
William Brafford 673c63bb00 Restrict testCreateEmptyDirNoPermissions to Unix (#44282) (#44297)
The test EmptyDirTaskTests#testCreateEmptyDirNoPermissions may fail on
Windows. However, the test is only meaningful for Unix permissions
structures, so we should assume a Unix-family OS and skip the test on
Windows.

Fixes #44064
2019-07-16 08:48:12 +03:00
Ryan Ernst 7e06888bae
Convert testclusters to use distro download plugin (#44253) (#44362)
Test clusters currently has its own set of logic for dealing with
finding different versions of Elasticsearch, downloading them, and
extracting them. This commit converts testclusters to use the
DistributionDownloadPlugin.
2019-07-15 17:53:05 -07:00
Yogesh Gaikwad 91c342a888
fix and enable repository-hdfs secure tests (#44044) (#44199)
Due to recent changes are done for converting `repository-hdfs` to test
clusters (#41252), the `integTestSecure*` tasks did not depend on
`secureHdfsFixture` which when running would fail as the fixture
would not be available. This commit adds the dependency of the fixture
to the task.

The `secureHdfsFixture` is a `AntFixture` which is spawned a process.
Internally it waits for 30 seconds for the resources to be made available.
For my local machine, it took almost 45 seconds to be available so I have
added the wait time as an input to the `AntFixture` defaults to 30 seconds
 and set it to 60 seconds in case of secure hdfs fixture.

The integ test for secure hdfs was disabled for a long time and so
the changes done in #42090 to fix the tests are also done in this commit.
2019-07-12 12:44:01 +10:00
Mark Vieira 5698f4dc27
Pass tests.jvms system property to test tasks for maxParallelForks (#44237) 2019-07-11 12:41:32 -07:00
Mark Vieira 7a82106de6
Ignore test seed when flag is passed (#44234) 2019-07-11 12:30:21 -07:00
Nick Knize 374030a53f
Upgrade to lucene-8.2.0-snapshot-860e0be5378 (#44171) (#44184)
Upgrades lucene library to lucene-8.2.0-snapshot-860e0be5378
2019-07-11 09:17:22 -05:00
Alpar Torok 7ba18732f7 Run some REST tests against a cluster running in docker containers (#39515)
* Run REST tests against a cluster running on docker

Closes #38053
2019-07-11 15:28:33 +03:00
Alpar Torok 47ab2bda72 Improve how log is tailed in testclusters on failure (#40600)
* Improoce how log is tailed in testclusters on failure

- only print last few lines
- print all errors and warnings
- compact repeating errors and warnings
2019-07-11 15:04:08 +03:00
Mark Vieira 7c2e4b2857
[Backport] Enable caching of rest tests which use integ-test distribution (#44181) 2019-07-10 15:42:28 -07:00
Alpar Torok bde5802ad6 Test fixtures improovements (#43956)
* Test fixtures improovements

Don't disable some of the precommit tasks on fixtures.
This no longer makes sense now that a project can both produce and use a
fixture.

In order for this to be possible, had to add an additional configuration
to make JarHell class accessible to the task even if it's not a
dependency of the project and fix some of the third party audit fallout
from  #43671 which wasn't detected at the time due to the issue being
fixed here.

Closes #43918
2019-07-10 21:21:06 +03:00
Alpar Torok 0c8294e633 Make sure the clean task doesn't break test fixtures (#43641)
Use a dedicated fixture dir.
2019-07-08 17:58:27 +03:00
Adrien Grand 680edbe3f1
Bump current version to 7.4. (#43927) 2019-07-03 20:32:04 +02:00
Mark Vieira b5d47e5643
Upgrade to Gradle 5.5 (#43788) (#43832) 2019-07-01 11:54:58 -07:00
Alpar Torok b4303290d9 Testclusters: improove timeout handling (#43440) 2019-07-01 11:39:53 +03:00
Alpar Torok a38f509284 Testclusters: convert left-overs from checkPart1 (#43370)
* Testclusters: convert left-overs from checkPart1
2019-06-25 19:14:45 +03:00
Ryan Ernst 97cd417829 Distribution download plugin cleanup (#43461)
This commit addresses some post-review comments.

relates #43247
2019-06-24 09:34:53 -07:00
Alpar Torok b4f5882fbe Improove error on waiting for cluster to come up (#43416)
Always include all messages from exceptions to make the reason for
failure more visible.
2019-06-24 12:12:21 +03:00
Alpar Torok ea44da6069 Testclusters: conver remaining x-pack (#43335)
Convert x-pack tests
2019-06-24 12:07:42 +03:00
Alpar Torok 5a9c48369b TestClusters: Convert the security plugin (#43242)
* TestClusters: Convert the security plugin

This PR moves security tests to use TestClusters.
The TLS test required support in testclusters itself, so the correct
wait condition is configgured based on the cluster settings.

* PR review
2019-06-18 11:55:44 +03:00
Mark Vieira 74813360ab Fix inverted condition so we never cache rest integ tests 2019-06-18 07:33:19 +03:00
Ryan Ernst d0a4d23027 Add elasticsearch distribution plugin (#43247)
Several types of distributions are built and tested in elasticsearch,
ranging from the current version, to building or downloading snapshot or
released versions. Currently tests relying on these have to contain
logic deciding where and how to pull down these distributions.

This commit adds an distributiond download plugin for each project to
manage which versions and variants the project needs. It abstracts away
all need for knowing where a particular version comes from, like a local
distribution or bwc project, or pulling from the elastic download
service. This will be used in a followup PR by the testclusters and
vagrant tests.
2019-06-17 17:53:39 -07:00
Alpar Torok 023a126b57 Correctly configure testclsuters for fips (#43186)
Configuration was not being triggered for all projects with the way it
was previusly set up.
With this change we do it for each project directly.
2019-06-14 10:55:04 +03:00
Alpar Torok 4ba94a5051 Testclusters: convert ccr tests (#42313) 2019-06-13 19:19:36 +03:00
Alpar Torok 167e51335d Convert ILM tests to use testclusters (#43076)
Also improove the error message when bin scripts are not found
2019-06-13 12:24:48 +03:00
Vincent Boulaye 209a493b27 convert EmptyDirTask.groovy to .java (#34672) 2019-06-13 12:21:23 +03:00
Gabryel Monteiro 47d861c23a Converting DependencyLicensesTask and UpdateShasTask to java (#41921) 2019-06-13 12:15:51 +03:00
Rafiullah Hamedy a1b8d48f17 Convert PluginPropertiesExtension Groovy to Java (#39605) 2019-06-13 09:28:12 +03:00
Jason Tedor 0eeb080a72
Do not use runtime Java when starting BWC nodes (#43071)
When starting BWC nodes, it could be that runtime Java home is set. Yet,
runtime Java home can advance beyond what a BWC node might be compatible
with. For example, if runtime Java home is set to JDK 13 and we are
starting a 7.1.2 node, we do not have any guarantees that 7.1.2 is
compatible with JDK 13 (since we never did any work to make it so). This
will continue to be the case as JDK releases advance, but we still need
to test against BWC nodes. This commit stops applying runtime Java home
when starting a BWC node. Instead, we would use the bundled JDK.
2019-06-11 10:26:57 -04:00
James Rodewig 5913723788 [DOCS] Change `// TESTRESPONSE[_cat]` to `// TESTRESPONSE[non_json]` (#43006) 2019-06-10 09:53:05 -04:00
Alpar Torok 9def454ea9 Clean up configuration when docker isn't available (#42745)
We initially added `requireDocker` for a way for tasks to say that they
absolutely must have it, like the  build docker image tasks.
Projects using the test fixtures plugin are not in this both, as the
intent with these is that they will be skipped if docker and docker-compose
is not available.

Before this change we were lenient, the docker image build would succeed
but produce nothing. The implementation was also confusing as it was not
immediately obvious this was the case due to all the indirection in the
code.

The reason we have this leniency is that when we added the docker image
build, docker was a fairly new requirement for us, and we didn't have
it deployed in CI widely enough nor had CI configured to prefer workers
with docker when possible. We are in a much better position now.
The other reason was other stack teams running `./gradlew assemble`
in their respective CI and the possibility of breaking them if docker is
not installed. We have been advocating for building specific distros for
some time now and I will also send out an additional notice

The PR also removes the use of `requireDocker` from tests that actually
use test fixtures and are ok without it, and fixes a bug in test
fixtures that would cause incorrect configuration and allow some tasks
to run when docker was not available and they shouldn't have.

Closes  #42680 and #42829  see also #42719
2019-06-10 13:44:15 +03:00
Alpar Torok a5d4591af7 Don't run build-tools integ tests on FIPS (#42986)
These run Gradle and FIPS isn't supported

Closes #41721
2019-06-10 11:52:59 +03:00
Mark Vieira 12433434ef
Skip installation of pre-bundled integ-test modules (#42900)
(cherry picked from commit 67bedf6c7c5d6db1394c045958668ac930a92d57)
2019-06-05 13:34:22 -07:00
Mark Vieira be23658114
Avoid clobbering shared testcluster JAR files when installing modules (#42879)
(cherry picked from commit 6da9aa29170c840bba08637c106c85bf16359979)
2019-06-04 17:56:55 -07:00