Commit Graph

282 Commits

Author SHA1 Message Date
Jason Tedor 92a48fa6b8
Add script to cache dependencies ()
With the introduction of immutable workers into our CI, we now have a
problem where we would have to download dependencies on every single
build. To this end our infrastructure team has introduced the
possibility to download dependencies one time and then bake these into
the base immutable worker image. For this, we introduce our version of
this special script which runs a task that downloads all dependencies of
all configurations. With this script, our infrastructure team will be
rebuilding these images on an at-least daily basis. This helps us avoid
having to download the dependencies for every single build.
2018-09-14 16:14:03 -04:00
Alpar Torok 7f7e8fd733
Disable assemble task instead of removing it () 2018-09-04 07:32:14 +03:00
Alpar Torok 3c367a2c46
Fix extractjar task ci ()
Remove tasks to check license and notice and add build integration test instead.

Closes 
2018-09-03 10:18:34 +03:00
Alpar Torok f6a570880c
Work around to be able to generate eclipse projects ()
* Work around to be able to generate eclipse projects

https://github.com/gradle/gradle/issues/6582
2018-08-31 16:01:54 +03:00
Nik Everett 2c81d7f77e
Build: Rework shadow plugin configuration ()
This reworks how we configure the `shadow` plugin in the build. The major
change is that we no longer bundle dependencies in the `compile` configuration,
instead we bundle dependencies in the new `bundle` configuration. This feels
more right because it is a little more "opt in" rather than "opt out" and the
name of the `bundle` configuration is a little more obvious.

As an neat side effect of this, the `runtimeElements` configuration used when
one project depends on another now contains exactly the dependencies needed
to run the project so you no longer need to reference projects that use the
shadow plugin like this:

```
testCompile project(path: ':client:rest-high-level', configuration: 'shadow')
```

You can instead use the much more normal:

```
testCompile "org.elasticsearch.client:elasticsearch-rest-high-level-client:${version}"
```
2018-08-21 20:03:28 -04:00
Alpar Torok d16562eab5
RFC: Test that example plugins build stand-alone ()
Add tests for build-tools to make sure example plugins build stand-alone using it.

This will catch issues such as referencing files from the buildSrc directly, breaking external uses of build-tools.
2018-08-17 09:41:39 +03:00
Alpar Torok b6c14935d7
Determine the minimum gradle version based on the wrapper ()
* Determine the minimum gradle version based on the wrapper

This is restrictive and forces users of the plugin to move together with
us, but without integration tests it's close to impossible to make sure
that the claimed compatability is really there.
If we do want to offer more flexibility, we should add those tests
first.

* Track gradle version in individual file

* PR review
2018-07-25 06:21:03 +00:00
Nik Everett e6b9f59e4e
Build: Shadow x-pack:protocol into x-pack:plugin:core ()
This bundles the x-pack:protocol project into the x-pack:plugin:core
project because we'd like folks to consider it an implementation detail
of our build rather than a separate artifact to be managed and depended
on. It is now bundled into both x-pack:plugin:core and
client:rest-high-level. To make this work I had to fix a few things.

Firstly, I had to make PluginBuildPlugin work with the shadow plugin.
In that case we have to bundle only the `shadow` dependencies and the
shadow jar.

Secondly, every reference to x-pack:plugin:core has to use the `shadow`
configuration. Without that the reference is missing all of the
un-shadowed dependencies. I tried to make it so that applying the shadow
plugin automatically redefines the `default` configuration to mirror the
`shadow` configuration which would allow us to use bare project references
to the x-pack:plugin:core project but I couldn't make it work. It'd *look*
like it works but then fail for transitive dependencies anyway. I think
it is still a good thing to do but I don't have the willpower to do it
now.

Finally, I had to fix an issue where Eclipse and IntelliJ didn't properly
reference shadowed transitive dependencies. Neither IDE supports shadowing
natively so they have to reference the shadowed projects. We fix this by
detecting `shadow` dependencies when in "Intellij mode" or "Eclipse mode"
and adding `runtime` dependencies to the same target. This convinces
IntelliJ and Eclipse to play nice.
2018-07-24 11:53:04 -04:00
Alpar Torok 38e2e1d553
Detect and prevent configuration that triggers a Gradle bug ()
* Detect and prevent configuration that triggers a Gradle bug

As we found in , this can lead to a lot of wasted time as it's not
immediatly obvius what's going on.
Givent how many projects we have it's getting increasingly easier to run
into .
2018-07-19 06:46:58 +00:00
Nik Everett 91d8371325 Build: Skip jar tests if jar disabled
The shadow plugin disables the jar task but we still attempted to extract
the jar to see if it had the right license and notice file. This skips
the extraction and those tests if the jar is built for any reason which
fixes projects that use the shadow plugin.
2018-07-17 20:14:51 -04:00
Nik Everett 1b97652a4c
Build: Move shadow customizations into common code ()
Moves the customizations to the build to produce nice shadow jars and
javadocs into common build code, mostly BuildPlugin with a little into
the root build.gradle file. This means that any project that applies the
shadow plugin will automatically be set up just like the high level rest
client:
* The non-shadow jar will not be built
* The shadow jar will not have a "classifier"
* Tests will run against the shadow jar
* Javadoc will include all of the shadowed classes
* Service files in `META-INF/services` will be merged
2018-07-17 14:20:41 -04:00
Alpar Torok b43fe560a4
Updates the build to gradle 4.9 ()
There are fixes to the dependency report, most importantly for us,
it still works even if `failOnVersionConflict` would fail the build.
2018-07-17 11:41:31 +00:00
Alpar Torok eb8c82a66b
Work around reported problem in eclipse ()
The Gradle plugin
https://docs.gradle.org/current/userguide/java_gradle_plugin.html
added recently adds a folder to the CP, which is not created for the
eclipse import, ausing eclipse to complain.
2018-07-12 08:55:31 +00:00
Tal Levy 60c1ea2c00 re-enable backcompat tests 2018-07-09 16:06:57 -04:00
Tal Levy 68a8d13828
add support for is_write_index in put-alias body parsing ()
* add support for is_write_index in put-alias body parsing

The Rest Put-Alias Action does separate parsing of the alias body
to construct the IndicesAliasesRequest. This extra parsing
was missed in .

* test flag was not just ignored by the parser

* disable backcompat tests
2018-07-09 16:03:17 -04:00
Nik Everett 994a251075
Build: Fix detection of Eclipse Compiler Server ()
It looks like Eclipse's compiler server recently changed something so
our "eclipse detector" stopped working for it. I've updated the detector
so it ought to work now.
2018-07-06 12:48:00 -04:00
Martijn van Groningen c6e7df3ac2
Re-enable bwc tests now that has been backported and 6.x intake
build succeeded.
2018-07-06 18:03:25 +02:00
Martijn van Groningen cae304406e
Temporarily disable bwc test in order to backport 2018-07-06 10:08:11 +02:00
Costin Leau f40581caa0
Fix license header generation on Windows ()
Updates the build.gradle to take into account the OS differences for
Windows (in particular line separator and project naming)
2018-07-05 12:28:40 +03:00
Sohaib Iftikhar dc869aa149 Build: re-enabled bwc ()
-- It was disabled by 
2018-07-03 12:13:28 -04:00
Sohaib Iftikhar a5fd4a7709 Implemented XContent serialisation for GetIndexResponse ()
This PR does the server side work for adding the Get Index API to the REST
high-level-client, namely moving resolving default settings to the
transport action. A follow up would be the client side changes.
2018-07-03 08:08:50 -04:00
Alpar Torok 200e1f45f2
Fix gradle4.8 deprecation warnings ()
* remove explicit wrapper task

It's created by Gradle and triggers a deprecation warning
Simplify configuration

* Upgrade shadow plugin to get rid of Gradle deprecation

* Move compile configuration to base plugin

Solves Gradle deprecation warning from earlier Gradle versions

* Enable stable publishing in the Gradle build

* Replace usage of deprecated property

* bump Gradle version in build compare
2018-06-29 19:17:19 +00:00
Alpar Torok 8557bbab28
Upgrade gradle wrapper to 4.8 ()
* Move to Gradle 4.8 RC1

* Use latest version of plugin

The current does not work with Gradle 4.8 RC1

* Switch to Gradle GA

* Add and configure build compare plugin

* add work-around for https://github.com/gradle/gradle/issues/5692

* work around https://github.com/gradle/gradle/issues/5696

* Make use of Gradle build compare with reference project

* Make the manifest more compare friendly

* Clear the manifest in compare friendly mode

* Remove animalsniffer from buildscript classpath

* Fix javadoc errors

* Fix doc issues

* reference Gradle issues in comments

* Conditionally configure build compare

* Fix some more doclint issues

* fix typo in build script

* Add sanity check to make sure the test task was replaced

Relates to . It seems like Gradle has an inconsistent behavior and
the taks is not always replaced.

* Include number of non conforming tasks in the exception.

* No longer replace test task, create implicit instead

Closes . The issue has full context in comments.

With this change the `test` task becomes nothing more than an alias for `utest`.
Some of the stand alone tests that had a `test` task now have `integTest`, and a
few of them that used to have `integTest` to run multiple tests now only
have `check`.
This will also help separarate unit/micro tests from integration tests.

* Revert "No longer replace test task, create implicit instead"

This reverts commit f1ebaf7d93e4a0a19e751109bf620477dc35023c.

* Fix replacement of the test task

Based on information from  replace the task taking
into account the task providres.
Closes .

* Only apply build comapare plugin if needed

* Make sure test runs before integTest

* Fix doclint aftter merge

* PR review comments

* Switch to Gradle 4.8.1 and remove workaround

* PR review comments

* Consolidate task ordering
2018-06-28 08:13:21 +03:00
Nik Everett 6dd81ead74
Build: Fix the license in the pom zip and tar ()
For 6.3 we renamed the `tar` and `zip` distributions to `oss-tar` and
`oss-zip`. Then we added new `tar` and `zip` distributions that contain
x-pack and are licensed under the Elastic License. Unfortunately we
accidentally generated POM files along side the new `tar` and `zip`
distributions that incorrectly claimed that they were Apache 2 licensed.
Oooops.

This fixes the license on the POMs generated for the `tar` and `zip`
distributions.
2018-06-14 16:22:00 -04:00
Adrien Grand 2c17ab7d2a Fix version detection. 2018-06-13 22:29:45 +02:00
Jason Tedor 65c107b47d
Fix unknown licenses ()
The goal of this commit is to address unknown licenses when producing
the dependencies info report. We have two different checks that we run
on licenses. The first check is whether or not we have stashed a copy of
the license text for a dependency in the repository. The second is to
map every dependency to a license type (e.g., BSD 3-clause). The problem
here is that the way we were handling licenses in the second check
differs from how we handle licenses in the first check. The first check
works by finding a license file with the name of the artifact followed
by the text -LICENSE.txt. Yet in some cases we allow mapping an artifact
name to another name used to check for the license (e.g., we map
lucene-.* to lucene, and opensaml-.* to shibboleth. The second check
understood the first way of looking for a license file but not the
second way. So in this commit we teach the second check about the
mappings from artifact names to license names. We do this by copying the
configuration from the dependencyLicenses task to the dependenciesInfo
task and then reusing the code from the first check in the second
check. There were some other challenges here though. For example,
dependenciesInfo was checking too many dependencies. For now, we should
only be checking direct dependencies and leaving transitive dependencies
from another org.elasticsearch artifact to that artifact (we want to do
this differently in a follow-up). We also want to disable
dependenciesInfo for projects that we do not publish, users only care
about licenses they might be exposed to if they use our assembled
products. With all of the changes in this commit we have eliminated all
unknown licenses. A follow-up will enforce that when we add a new
dependency it does not get mapped to unknown, these will be forbidden in
the future. Therefore, with this change and earlier changes are left
having no unknown licenses and two custom licenses; custom here means it
does not map to an SPDX license type. Those two licenses are xz and
ldapsdk. A future change will not allow additional custom licenses
unless they are explicitly whitelisted. This ensures that if a new
dependency is added it is mapped to an SPDX license or mapped to custom
because it does not have an SPDX license.
2018-06-09 07:28:41 -04:00
Jason Tedor d49c85d2e8
Add back dropped substitution on merge
This was dropped accidentally during merge conflict resolution. This
commit adds back the substitution for elasticsearch-cli.
2018-06-07 17:40:47 -04:00
Jason Tedor 5296c11e4f
Rename elasticsearch-nio to nio ()
This commit renames :libs:elasticsearch-nio to :libs:nio.
2018-06-07 17:00:00 -04:00
Jason Tedor 94be9b471f
Rename elasticsearch-core to core ()
This commit renames :libs:elasticsearch-core to :libs:core.
2018-06-07 16:50:21 -04:00
Jason Tedor b32cbc1baa
Move cli sub-project out of server to libs ()
This commit moves the cli sub-project out of server to libs where it
makes more sense.
2018-06-07 16:35:34 -04:00
Jason Tedor 805648848d
Add check for feature aware implementations ()
This commit adds a check that any class in X-Pack that is a feature
aware custom also implements the appropriate mix-in interface in
X-Pack. These interfaces provide a default implementation of
FeatureAware#getRequiredFeature that returns that x-pack is the required
feature. By implementing this interface, this gives a consistent way for
X-Pack feature aware customs to return the appopriate required feature
and this check enforces that all such feature aware customs return the
appropriate required feature.
2018-06-05 19:56:22 -04:00
Alpar Torok ff62638ba6
Accept Gradle build scan agreement ()
* Accept Gradle build scan argreement

Scans will be produced only when passing
`--scan`

* Condition TOS acceptance with property

* Switch to boolean flags
2018-05-22 07:21:16 +03:00
Colin Goodheart-Smithe 09b918545d
Adds Eclipse config for xpack licence headers ()
Uses a filter on the copy task for the eclipse settings files to
replace the token @@LICENSE_HEADER_TEXT@@ with the correct licence
header from the new buildSrc/src/main/resources/license-headers
directory
2018-05-03 08:48:49 +01:00
Ryan Ernst a799939630 Revert "Build: Move gradle wrapper jar to a dot dir ()"
This reverts commit a324cd41f7.

A non default gradle wrapper location causes intellij to fail in
mysterious ways....
2018-05-01 16:46:58 -07:00
Ryan Ernst a324cd41f7
Build: Move gradle wrapper jar to a dot dir ()
This commit moves the gradle wrapper jar file to a hidden directory, so
that it does not clutter the top level names seen when doing an ls in
the project. The actual jar file is never manually edited, and only
changed by running `./gradlew wrapper ...` so it is not important for
this directory to be "visible".
2018-05-01 08:47:53 -07:00
Nik Everett a8f40b3e04 Build: Assert jar LICENSE and NOTICE files match
Adds tasks that check that the all jars that we build have LICENSE.txt
and NOTICE.txt files and that the files are correct. Sets check to
depend on these task.

This is mostly there for extra parnoia because we automatically
configure all Jar tasks to include the LICENSE.txt and NOTICE.txt
files anyway. But it is quite possible to add configuration to those
tasks that would override either file.

This causes check to depend on several more things than it used to.
Take, for example, javadoc:

check depends on the new verifyJavadocJarNotice which depends on
extractJavadocJar which depends on javadocJar which depends on
javadoc, this check now depends on javadoc.
2018-04-25 19:53:24 -04:00
Ryan Ernst 0d8aa7527e Reorganize license files
This commit moves the apache and elastic license files into a new
root level `licenses` directory and rewrites the top level LICENSE.txt
to clarify the repository has a mix of apache and elastic licensed code.
2018-04-20 15:33:59 -07:00
Ryan Ernst fab5e21e7d Build: Split distributions into oss and default
This commit makes x-pack a module and adds it to the default
distrubtion. It also creates distributions for zip, tar, deb and rpm
which contain only oss code.
2018-04-20 15:33:57 -07:00
Nik Everett 69aabb7e40
Build: Fail if any libs depend on non-core libs ()
Fails the build if any subprojects of `:libs` have dependencies in `:libs`
except for `:libs:elasticsearch-core`.

Since we now have three places where we resolve project substitutions
I've added `dependencyToProject` to `project.ext` in all projects. It
resolves both `project` style dependencies and "external" style (like
"org.elasticsearch:elasticsearch-core:${version}") dependencies to
`Project`s using the `projectSubstitutions`. I use this new function all
three places where resovle project substitutions.

Finally this pulls `apply plugin: 'elasticsearch.build'` out of
`libs/*/build.gradle` and into a subprojects clause in
`libs/build.gradle`. I do this entirely so that I can call
`tasks.precommit.dependsOn checkDependencies` without waiting for the
subprojects to be evaluated or worrying about whether or not they have
`precommit` set up in a normal way.
2018-04-16 11:49:27 -04:00
Jason Tedor efa823bd79 Simplify snapshot check in root build file
Rather than checking a substring match, now that
VersionProperties#elasticsearch is a strongly-typed instance of Version,
we can use the Version#isSnapshot convenience method. This commit
switches the root build file to do this.
2018-04-13 17:11:37 -04:00
Jason Tedor 85ac541ab3
Make NodeInfo#nodeVersion strongly-typed as Version ()
Today we have a nodeVersion property on the NodeInfo class that we use
to carry around information about a standalone node that we will start
during tests. This property is a String which we usually end up parsing
to a Version anyway to do various checks on it. This can end up
happening a lot during configuration so it would be more efficient and
safer to have this already be strongly-typed as a Version and parsed
from a String only once for each instance of NodeInfo. Therefore, this
commit makes NodeInfo#nodeVersion strongly-typed as a Version.
2018-04-13 16:57:59 -04:00
Lee Hinman 6b2167f462
Begin moving XContent to a separate lib/artifact ()
* Begin moving XContent to a separate lib/artifact

This commit moves a large portion of the XContent code from the `server` project
to the `libs/xcontent` project. For the pieces that have been moved, some
helpers have been duplicated to allow them to be decoupled from ES helper
classes. In addition, `Booleans` and `CheckedFunction` have been moved to the
`elasticsearch-core`  project.

This decoupling is a move so that we can eventually make things like the
high-level REST client not rely on the entire ES jar, only the parts it needs.

There are some pieces that are still not decoupled, in particular some of the
XContent tests still remain in the server project, this is because they test a
large portion of the pluggable xcontent pieces through
`XContentElasticsearchException`. They may be decoupled in future work.
Additionally, there may be more piecese that we want to move to the xcontent lib
in the future that are not part of this PR, this is a starting point.

Relates to 
2018-04-02 15:58:31 -06:00
Jason Tedor daf430c006 Revert "Disable BWC tests for build issues"
This reverts commit 45470945d2.
2018-02-22 06:36:27 -05:00
Jason Tedor 45470945d2 Disable BWC tests for build issues
This is a temporary disabling of BWC tests while waiting for some build
issues to be addressed.
2018-02-21 13:25:33 -05:00
Ryan Ernst 714efb2bcf Remove leftover PR link for previously disabled bwc tests 2018-02-16 20:30:02 -08:00
Colin Goodheart-Smithe d58226be8f
[TEST] Renables bwc tests 2018-02-16 10:17:55 +00:00
Ryan Ernst 53c38cc8fe
Build: Group archive and package distribution projects ()
This commit adds intermediate gradle projects for archive based
distributions (zip, tar) and package based distributions (rpm, deb). The
grouping allows the common distribution build file to be considerably
shorter and clearly separated from the common zip/tar and rpm/deb
configuration.
2018-02-13 22:49:53 -08:00
Michael Basnight e0bea70070
Generalize BWC logic ()
Generalizing BWC building so that there is less code to modify for a release. This ensures we do not
need to think about what major or minor version is in the gradle code. It follows the general rules of the
elastic release structure. For more information on the rules, see the VersionCollection's javadoc.

This also removes the additional bwc snapshots that will never be released, such as 6.0.2, which were
being built and tested against every time we ran bwc tests.

Additionally, it creates 4 new projects that correspond to the different types of snapshots that may exist
for a given version. Its possible to now run those individual tasks to work out bwc logic whereas
previously it was impossible and the entire suite of bwc tests had to be run to work out any logic
changes in the build tools' bwc project. Please note that if the project does not make sense for the 
version that is current, that an error will be thrown from that individual project if an attempt is made to 
run it.

This should allow for automating the version bumps as well, since it removes all the hardcoded version
logic from the configs.
2018-02-09 14:55:10 -06:00
Ryan Ernst 20c37efea2
Build: Replace provided configuration with compileOnly ()
When elasticsearch was originally moved to gradle, the "provided" equivalent in maven had to be done through a plugin. Since then, gradle added the "compileOnly" configuration. This commit removes the provided plugin and replaces all uses with compileOnly.
2018-02-09 11:30:24 -08:00
Ryan Ernst 7e65cd8ca8
Build: Add warning when bwc tests are disabled ()
The bwc tests can be disabled in order to facilitate commits necessary
to older branches to maintain backcompat. A check already exists to
ensure this flag is not left disabled too long (once a day by the
branchConsistency check). However, it can be surprising if you try
running bwc tests explicitly and they look like nothing is happening.
This commit adds a warning during configuration to ensure it is clear
the bwc tests are disabled and enforces a link to a PR which is in the
process of being backported.
2018-02-09 11:26:22 -08:00