Commit Graph

93 Commits

Author SHA1 Message Date
Ryan Ernst 374ce3e6a8 Add gradle plugin for downloading jdk (#41461)
We currently download 3 variants of the same version of the jdk for
bundling into the distributions. Additionally, the vagrant images do
their own downloading. This commit moves the jdk downloading into a
utility gradle plugin. This will be used in a future PR by the packaging
tests.

The new plugin exposes a "jdks" project extension which allows creating
named jdks. Once the jdk version and platform are set for a named jdk,
the jdk object may be used as a lazy String for the jdk home path, or a
file collection for copying.
2019-05-08 07:16:44 -07:00
Jason Tedor d7fd51a84e
Provide names for all artifact repositories (#41857)
This commit adds a name for each Maven and Ivy repository used in the
build.
2019-05-07 06:35:28 -04:00
Alpar Torok 2f8bbce85d Clean up build tool dependencies (#41336)
We are no longer using these dependencies.

Relates to #41061 since the class that seems to be leaking is both part
of Gradle and the logging jar.
2019-04-19 09:22:38 +03:00
Mark Vieira 1287c7d91f
[Backport] Replace usages RandomizedTestingTask with built-in Gradle Test (#40978) (#40993)
* Replace usages RandomizedTestingTask with built-in Gradle Test (#40978)

This commit replaces the existing RandomizedTestingTask and supporting code with Gradle's built-in JUnit support via the Test task type. Additionally, the previous workaround to disable all tasks named "test" and create new unit testing tasks named "unitTest" has been removed such that the "test" task now runs unit tests as per the normal Gradle Java plugin conventions.

(cherry picked from commit 323f312bbc829a63056a79ebe45adced5099f6e6)

* Fix forking JVM runner

* Don't bump shadow plugin version
2019-04-09 11:52:50 -07:00
Tim Vernum 9d34164ce6
Add build utility to check cluster health over ssl (#40713)
By default, in integ tests we wait for the standalone cluster to start
by using the ant Get task to retrieve the cluster health endpoint.
However the ant task has no facilities for customising the trusted
CAs for a https resource, so if the integ test cluster has TLS enabled
on the http interface (using a custom CA) we need a separate utility
for that purpose.

Backport of: #40573
2019-04-04 21:44:03 +11:00
Alpar Torok 49f18811c3 Run the build integ test in parallel (#39788)
* Run the build integ test in parallel

Because the randomized runner lives in buildSrc, we run these tests with
the Gradle runner, and had no parallelism configured so far.

* Handle Windows and "auto" better
2019-03-29 08:11:20 +02:00
Jim Ferenczi 7a7658707a
Upgrade to Lucene release 8.0.0 (#39998)
This commit upgrades to the GA release of Lucene 8

Closes #39640
2019-03-13 18:11:50 +01:00
Ryan Ernst 8e5f9c4b14
Add OS/architecture classifier to distributions (#37881)
This commit adds classifiers to the distributions indicating the
OS (for archives) and platform. The current OSes are for windows, darwin (ie
macos) and linux. This change will allow future OS/architecture specific
changes to the distributions. Note the docs using distribution links
have been updated, but will be reworked in a followup to make OS
specific instructions for the archives.
2019-01-29 11:18:30 -08:00
Alpar Torok 37768b7eac
Testing conventions now checks for tests in main (#37321)
* Testing conventions now checks for tests in main

This is the last outstanding feature of the old NamingConventionsTask,
so time to remove it.

* PR review
2019-01-24 17:30:50 +02:00
Ryan Ernst c9c088ff02
Use ivy pattern for elasticsearch artifacts download (#37557)
The rpm, deb and tar distributions were removed some time ago from maven
central. The zip distribution still exists there, but it does not need
to. Instead, this commit sets up an ivy repository with pattern pointing
to the elasticsearch artifacts download service. Note that the
integ-test-zip remains in maven central, since it is not present in the
download service.
2019-01-20 23:33:11 -08:00
Alpar Torok 6344e9a3ce
Testing conventions: add support for checking base classes (#36650) 2019-01-08 13:39:03 +02:00
Armin Braun 31c33fdb9b
MINOR: Remove some Deadcode in Gradle (#37160) 2019-01-07 09:21:25 +01:00
Alpar Torok e9ef5bdce8
Converting randomized testing to create a separate unitTest task instead of replacing the builtin test task (#36311)
- Create a separate unitTest task instead of Gradle's built in 
- convert all configuration to use the new task 
- the  built in task is now disabled
2018-12-19 08:25:20 +02:00
Alpar Torok c00d0fc814
Test fixtures improovements (#36037)
* Upgrae plugin to latest and expose udp
* Explicit check for windows
* Rename the properties for the port numbers
* Tasks for pre and pos container actions
2018-12-12 12:00:47 +02:00
James Baiera a07fb312fc
Move LoggedExec to minimumRuntime source set (#36453)
This moves LoggedExec to the minimumRuntime source set so that it correctly 
compiles for earlier Java versions, namely Java 8.
2018-12-11 13:49:24 -05:00
Christian Schneider 00eadd93eb converting ForbiddenPatternsTask to .java (#36194)
* converting ForbiddenPatternsTask to java impl & unit tests
2018-12-11 14:15:44 +02:00
Alpar Torok 9ae5ee2497 Fix typo in comment 2018-12-06 09:14:50 +02:00
Alpar Torok d531fab4b3 Run Build tests with proper java version 2018-12-05 14:47:15 +02:00
Alpar Torok 60e45cd81d
Testing conventions task part 2 (#36107)
Closes #35435

- make it easier to add additional testing tasks with the proper configuration and add some where they were missing.
- mute or fix failing tests
- add a check as part of testing conventions to find classes not included in any testing task.
2018-12-05 14:20:01 +02:00
Alpar Torok d036e0ca89
Testclusters: implement starting, waiting for and stopping single cluster nodes (#35599) 2018-12-04 10:16:51 +02:00
Alpar Torok 45db829039
Replace fixtures with docker-compose (#35651)
Creates a new plugin to manage docker-compose based test fixtures. 
Convert the smb-fixture as a first example.
2018-11-29 09:43:16 +02:00
Alpar Torok 212c202709 Remove hard coded version string in tests
- The current version was hard coded in the test, causing it to fail as
we removed the qualifier
- also applied the base plugin to the root project to get the `clean`
task to work as expected. This was preventing the failure from
reproducing locally.
2018-11-14 14:14:03 +02:00
Alpar Torok dc0e05f35a
Fix Eclipse configuration (#35452) 2018-11-14 11:11:46 +02:00
Alpar Torok b49f461bf1
Swithc to jcenter and hope to have fewer network failures in builds (#35427)
* Swithc to jcenter

Jcenter suposedly operates on a CDN.
It should be faster and more reliable.
It's the default in Andorid Studio currently
( it switched from mavenCentral ).

This change is safe because jcenter is a superset of mavenCentral.

* update comment
2018-11-14 10:50:08 +02:00
Alpar Torok ca24d9a07e
Switch to default to no build qualifier (#35415) 2018-11-12 14:43:42 +02:00
Alpar Torok 366f79cef2
Check for java 11 in buildSrc (#35260) 2018-11-06 14:43:48 +02:00
Alpar Torok 58f3e81555
Add missing up-to-date configuration (#35255)
* Add missing up-to-date configuration

The source properties file and the dynamic elasticsearch version
(set based on properties ) were missing from task outputs leading
to the task being incorrectly considered up to date.

Closes #35204
2018-11-06 14:00:40 +02:00
Alpar Torok f22700812e
Introduce build qualifier parameter (#35155)
* Introduce property to set version qualifier

- VersionProperties.elasticsearch is now a string which can have qualifier
and snapshot too
- The Version class in the build no longer cares about snapshot and
qualifier.
2018-11-02 05:27:40 +02:00
Alpar Torok 13ca942152
Switch build-tools to latest target version (#34746)
- we already require Java 11 to build, yet we target the minimum
supported version in build-tools ( currently 8 )
- this is because we have some checks that are executed in a new JVM
which could be running the minimum version.
- For everything else it would be nice to be able to use new features,
like the new process API.

With this change, we selectively compile the few classes that need an
older target version and move everything over to Java 10.
Unfortunately the current Gradle version does not support 11 as a target
version yet.
2018-10-25 10:03:23 +03:00
Alpar Torok 0536635c44
Upgrade forbiddenapis to 2.6 (#33809)
* Upgrade forbiddenapis to 2.6

Closes #33759

* Switch forbiddenApis back to official plugin

* Remove CLI based task

* Fix forbiddenApisJava9
2018-10-23 12:06:46 +03:00
Alpar Torok db3d32ce91
Fix pom for build-tools (#33300)
Looks like `java-gradle-plugin` reconfigures the pom.
Stop using it since we don't publish to Gradle plugin portal.
2018-08-31 16:48:00 +03:00
Alpar Torok 5cf6e0d4bc
Ignore module-info in jar hell checks (#33011)
* Ignore module-info in JarHell checks
* Add unit test
* integration test to test that jarhell is ran with precommit
2018-08-30 11:41:39 +03:00
Alpar Torok 9c541b8f72
Upgrade to latest Gradle 4.10 (#32801)
Upgrade to Gradle 4.10
2018-08-30 10:20:38 +03:00
Alpar Torok f097446066
Fix/30904 cluster formation part2 (#32877)
Gradle integration for the Cluster formation plugin with ref counting
2018-08-30 10:17:42 +03:00
Alpar Torok 2cc611604f
Run Third party audit with forbidden APIs CLI (part3/3) (#33052)
The new implementation is functional equivalent with the old, ant based one.
It parses task standard error to get the missing classes and violations in the same way.
I considered re-using ForbiddenApisCliTask but Gradle makes it hard to build inheritance with tasks that have task actions , since the order of the task actions can't be controlled.
This inheritance isn't dully desired either as the third party audit task is much more opinionated and we don't want to expose some of the configuration.
We could probably extract a common base class without any task actions, but probably more trouble than it's worth.

Closes #31715
2018-08-28 10:03:30 +03:00
Alpar Torok d16562eab5
RFC: Test that example plugins build stand-alone (#32235)
Add tests for build-tools to make sure example plugins build stand-alone using it.

This will catch issues such as referencing files from the buildSrc directly, breaking external uses of build-tools.
2018-08-17 09:41:39 +03:00
Alpar Torok bb3eacf384
Revert "cluster formation DSL - Gradle integration - part 2 (#32028)" (#32876)
This reverts commit e680a64434.
2018-08-15 19:17:46 +03:00
Alpar Torok e680a64434
cluster formation DSL - Gradle integration - part 2 (#32028)
* Implement Version in java

- This allows to move all  all .java files from .groovy.
- Will prevent eclipse from tangling up in this setup
- make it possible to use Version from Java

* PR review comments

* Cluster formation plugin with reference counting

```
> Task :plugins:ingest-user-agent:listElasticSearchClusters
Starting cluster: myTestCluster
   * myTestCluster: /home/alpar/work/elastic/elasticsearch/plugins/ingest-user-agent/foo
Asked to unClaimAndStop myTestCluster, since cluster still has 1 claim it will not be stopped

> Task :plugins:ingest-user-agent:testme UP-TO-DATE
Stopping myTestCluster, since no of claims is 0
```

- Meant to auto manage the clusters lifecycle
- Add integration test for cluster formation

* Fix rebase

* Change to `useCluster` method on task
2018-08-15 19:16:10 +03:00
Nik Everett 0f783ce94b
Build: Stop double generating buildSrc pom (#32408)
When we added the `java-gradle-plugin` to `buildSrc` it added a second
task to generate the pom that duplicates the publishing work that we
configure in `BuildPlugin`. Not only does it dupliciate the pom, it
creates a pom that is missing things like `name` and `description` which
are required for publishing to maven central.

This change disables the duplicate pom generation.
2018-07-26 13:19:05 -04:00
Alpar Torok b6c14935d7
Determine the minimum gradle version based on the wrapper (#32226)
* Determine the minimum gradle version based on the wrapper

This is restrictive and forces users of the plugin to move together with
us, but without integration tests it's close to impossible to make sure
that the claimed compatability is really there.
If we do want to offer more flexibility, we should add those tests
first.

* Track gradle version in individual file

* PR review
2018-07-25 06:21:03 +00:00
Alpar Torok a9a9598e02
Require Gradle 4.9 as minimum version (#32200)
Do the check in the build plugin as well to be more informative to users
of  build-tools.

We use APIs specific to Gradle 4.9 so earlier will not work.
2018-07-20 05:20:35 +00:00
Nik Everett 1b97652a4c
Build: Move shadow customizations into common code (#32014)
Moves the customizations to the build to produce nice shadow jars and
javadocs into common build code, mostly BuildPlugin with a little into
the root build.gradle file. This means that any project that applies the
shadow plugin will automatically be set up just like the high level rest
client:
* The non-shadow jar will not be built
* The shadow jar will not have a "classifier"
* Tests will run against the shadow jar
* Javadoc will include all of the shadowed classes
* Service files in `META-INF/services` will be merged
2018-07-17 14:20:41 -04:00
Alpar Torok ba3c0f2478
Move build integration tests out of :buildSrc project (#31961)
This way building buildSrc will not be delayed for all invocations, and
will be less disruptive if integration tests fail
2018-07-12 08:41:07 +00:00
Alpar Torok f702086df1 Build: Fix naming conventions task (#31681)
Move min compiler and runtime version to files so we can fix the source
and target comparability of the build script to those

Closes #31665
2018-06-29 14:41:02 -04:00
Alpar Torok 0afec8f31c
Remove deprecation warnings to prepare for Gradle 5 (sourceSets.main.output.classesDirs) (#30389)
* Remove deprecation warnings to prepare for Gradle 5

Gradle replaced `project.sourceSets.main.output.classesDir` of type
`File` with `project.sourceSets.main.output.classesDirs` of type
`FileCollection`
(see [SourceSetOutput](https://github.com/gradle/gradle/blob/master/subprojects/plugins/src/main/java/org/gradle/api/tasks/SourceSetOutput.java))
Build output is now stored on a per language folder.

There are a few places where we use that, here's these and how it's
fixed:

- Randomized Test execution
    - look in all test folders ( pass the multi dir configuration to the
    ant runner )
    - DRY the task configuration by introducing `basedOn` for
      `RandomizedTestingTask` DSL
- Extend the naming convention test to support passing in multiple
  directories
- Fix the standalon test plugin, the dires were not passed trough,
  checked with a debuger and the statement had no affect due to a
  missing `=`.

Closes #30354

* Only check Java tests, PR feedback

- Name checker was ran for Groovy tests that don't adhere to the same
  convections causing the check to fail
- implement PR feedback

* Replace `add` with `addAll`

This worked because the list is passed to `project.files` that does the
right thing.

* Revert "Only check Java tests, PR feedback"

This reverts commit 9bd9389875d8b88aadb50df57a45cd0d2b073241.

* Remove `basedOn` helper

* Bring some changes back

Previus revert accidentally reverted too much

* Fix negation

* add back public

* revert name check changes

* Revert "revert name check changes"

This reverts commit a2800c0b363168339ea65e2a79ec8256e5883e6d.

* Pass all dirs to name check

Only run on Java for build-tools, this is safe because it's a self test.
It needs more work before we could pass in the Groovy classes as well as
these inherit from `GroovyTestCase`

* remove self tests from name check

The self complicates the task setup and disable real checks on
build-tools.
With this change there are no more self tests, and the build-tools tests
adhere to the conventions.
The self test will be replaced by gradle test kit, thus the addition of
the Gradle plugin builder plugin.

* First test to run a Gradle build

* Add tests that replace the name check self test

* Clean up integ test base class

* Always run tests

* Align with test naming conventions

* Make integ. test case inherit from unit test case

The check requires this

* Remove `import static org.junit.Assert.*`
2018-06-28 15:14:34 +03:00
Alpar Torok 8557bbab28
Upgrade gradle wrapper to 4.8 (#31525)
* Move to Gradle 4.8 RC1

* Use latest version of plugin

The current does not work with Gradle 4.8 RC1

* Switch to Gradle GA

* Add and configure build compare plugin

* add work-around for https://github.com/gradle/gradle/issues/5692

* work around https://github.com/gradle/gradle/issues/5696

* Make use of Gradle build compare with reference project

* Make the manifest more compare friendly

* Clear the manifest in compare friendly mode

* Remove animalsniffer from buildscript classpath

* Fix javadoc errors

* Fix doc issues

* reference Gradle issues in comments

* Conditionally configure build compare

* Fix some more doclint issues

* fix typo in build script

* Add sanity check to make sure the test task was replaced

Relates to #31324. It seems like Gradle has an inconsistent behavior and
the taks is not always replaced.

* Include number of non conforming tasks in the exception.

* No longer replace test task, create implicit instead

Closes #31324. The issue has full context in comments.

With this change the `test` task becomes nothing more than an alias for `utest`.
Some of the stand alone tests that had a `test` task now have `integTest`, and a
few of them that used to have `integTest` to run multiple tests now only
have `check`.
This will also help separarate unit/micro tests from integration tests.

* Revert "No longer replace test task, create implicit instead"

This reverts commit f1ebaf7d93e4a0a19e751109bf620477dc35023c.

* Fix replacement of the test task

Based on information from gradle/gradle#5730 replace the task taking
into account the task providres.
Closes #31324.

* Only apply build comapare plugin if needed

* Make sure test runs before integTest

* Fix doclint aftter merge

* PR review comments

* Switch to Gradle 4.8.1 and remove workaround

* PR review comments

* Consolidate task ordering
2018-06-28 08:13:21 +03:00
Uwe Schindler 6578d8a2a8 Update to forbiddenapis 2.5 (#29285) 2018-03-28 20:29:02 -07:00
Martijn van Groningen 7d0eb3292b
upgraded jna from 4.4.0-1 to 4.5.1 (#28183)
Closes #28172
2018-01-11 17:31:10 +01:00
Maxime Gréau 771defb97c
Build: Add 3rd party dependencies report generation (#27727)
* Adds task dependenciesInfo to BuildPlugin to generate a CSV file with dependencies information (name,version,url,license)
* Adds `ConcatFilesTask.groovy` to concatenates multiple files into one
* Adds task `:distribution:generateDependenciesReport` to concatenate `dependencies.csv` files into a single file (`es-dependencies.csv` by default)

 # Examples:
      $ gradle dependenciesInfo :distribution:generateDependenciesReport

 ## Use `csv` system property to customize the output file path
     $ gradle dependenciesInfo :distribution:generateDependenciesReport -Dcsv=/tmp/elasticsearch-dependencies.csv

 ## When branch is not master, use `build.branch` system property to generate correct licenses URLs
     $ gradle dependenciesInfo :distribution:generateDependenciesReport -Dbuild.branch=6.x -Dcsv=/tmp/elasticsearch-dependencies.csv
2017-12-26 10:51:47 +01:00
Yannick Welsch 7b71cdd34e
Fix Gradle 4.3.1 compatibility for logging (#27382)
The build currently does not work with Gradle 4.3.1 as the Gradle team stopped publishing the gradle-logging dependency to jcenter, starting with 4.3.1 (not sure why). There are two options:

- Add the repository managed by Gradle team (https://repo.gradle.org/gradle/libs-releases-local) to our build
- Use an older version (4.3) of the dependency when running with Gradle 4.3.1.

Not to depend on another external repo, I've chosen solution 2. Note that this solution could break on future versions, but as this is a compileOnly dependency, and the interface we use has been super stable since forever, I don't envision this to be an issue (and easily detected by a breaking build).
2017-11-15 14:50:13 +01:00