Commit Graph

905 Commits

Author SHA1 Message Date
Steven Goossens a62c5ae11b Upgrade Centos version in Dockerfile to 8 (#59019) 2020-08-06 13:44:58 +01:00
Rene Groeschke bdd7347bbf
Merge test runner task into RestIntegTest (7.x backport) (#60600)
* Merge test runner task into RestIntegTest (#60261)
* Merge test runner task into RestIntegTest
* Reorganizing Standalone runner and RestIntegTest task
* Rework general test task configuration and extension
* Fix merge issues
* use former 7.x common test configuration
2020-08-04 14:46:32 +02:00
Rene Groeschke ed4b70190b
Replace immediate task creations by using task avoidance api (#60071) (#60504)
- Replace immediate task creations by using task avoidance api
- One step closer to #56610
- Still many tasks are created during configuration phase. Tackled in separate steps
2020-07-31 13:09:04 +02:00
Rene Groeschke 79ec0f2442
Split internal distribution handling into separate internal plugin (7.x backport) (#60270)
* Split internal distribution handling into separate internal plugin (#60295)
* Provide proper failure if unexpected non jdk bundled bwc version is requested
2020-07-31 09:30:21 +02:00
Mark Vieira 39fa1c4df0
Add compatibility testing for JDBC driver (#60409)
This commit adds compatibility testing of our JDBC driver against
different Elasticsearch versions. Although we are really testing the
forwards compatibility nature of the JDBC driver we model the testing
the same as we do existing BWC tests, that is, with the current branch
fetching the earlier versions of the artifact that is to be tested. In
this case, that's the JDBC driver itself.

Because the tests include the JDBC driver jar on it's classpath we had
to change the packaging of the driver jar in order to avoid jarhell and
other conflicting dependency issues when using an old JDBC driver with
later branches. For this we simply relocate all driver dependencies in
the shadow jar under a "shadowed" package. This allows the JDBC driver
to use the correct version of Elasticsearch libs classes, while the
tests themselves use their versions. Since this required a change to the
driver jar compatibility testing can only go back as far as that version
which at the time of this commit is 7.8.1.
2020-07-29 10:45:11 -07:00
Ryan Ernst 07aede135d
Set the systemd initial timeout to 75 seconds (#60345)
For systemd, while we are starting up, we notify the system every 15
seconds that we are still in the middle of starting up. However, if
initial startup before plugin initialization is slower than 15 seconds,
we won't ever get the chance to run the first timeout extension. This
commit sets the initial timeout to 75 seconds, up from the default 30
seconds used by systemd.

closes #60140
2020-07-28 16:57:36 -07:00
Rene Groeschke d5c11479da
Remove remaining deprecated api usages (#59231) (#59498)
- Fix duplicate path deprecation by removing duplicate test resources
- fix deprecated non annotated input property in LazyPropertyList
- fix deprecated usage of AbstractArchiveTask.version
- Resolve correct test resources
2020-07-14 10:25:00 +02:00
Rene Groeschke 68dd431bc9
Fix deprecated unsave project outputs resolution (#59088) (#59356)
- Fixes how libs in distribution are resolved
- Required minor rework on common repository setup to allow distribution projects
to resolve thirdparty artifacts
- Use Default configurations when resolving tools for distribution packaging
- Related to #57920
2020-07-10 17:16:47 +02:00
Rene Groeschke 6a7ac3830a
Verify distro archives do not contain plain class files (#59073) (#59229)
This is a test to ensure we do not run into a regression like we did in
https://github.com/elastic/elasticsearch/issues/59031
2020-07-08 17:06:36 +02:00
Jason Tedor b2e5a08798
Use https for JDK sources (#59111)
This commit switches to using https to refer to the JDK sources.
2020-07-07 13:39:50 -04:00
Rene Groeschke e8181fc627
Fix implicit duplicate duplicatesStrategy in processResources (#58929) (#59127)
* Fix implicit duplicate duplicatesStrategy in processResources
* Fix duplicates strategy in docker distribution setup
2020-07-07 13:45:36 +02:00
Jake Landis 604c6dd528
7.x - Create plugin for yamlTest task (#56841) (#59090)
This commit creates a new Gradle plugin to provide a separate task name
and source set for running YAML based REST tests. The only project
converted to use the new plugin in this PR is distribution/archives/integ-test-zip.
For which the testing has been moved to :rest-api-spec since it makes the most
sense and it avoids a small but awkward change to the distribution plugin.

The remaining cases in modules, plugins, and x-pack will be handled in followups.

This plugin is distinctly different from the plugin introduced in #55896 since
the YAML REST tests are intended to be black box tests over HTTP. As such they
should not (by default) have access to the classpath for that which they are testing.

The YAML based REST tests will be moved to separate source sets (yamlRestTest).
The which source is the target for the test resources is dependent on if this
new plugin is applied. If it is not applied, it will default to the test source
set.

Further, this introduces a breaking change for plugin developers that
use the YAML testing framework. They will now need to either use the new source set
and matching task, or configure the rest resources to use the old "test" source set that
matches the old integTest task. (The former should be preferred).

As part of this change (which is also breaking for plugin developers) the
rest resources plugin has been removed from the build plugin and now requires
either explicit application or application via the new YAML REST test plugin.

Plugin developers should be able to fix the breaking changes to the YAML tests
by adding apply plugin: 'elasticsearch.yaml-rest-test' and moving the YAML tests
under a yamlRestTest folder (instead of test)
2020-07-06 14:16:26 -05:00
Rene Groeschke 56136b75dc
Fix security-cli distribution packaging (#59048)
- This fixes https://github.com/elastic/elasticsearch/issues/59031
- do not use compileclasspath in distribution packaging as it uses by default
plain class files
2020-07-06 09:46:22 +02:00
Rene Groeschke 70713a0a19
Remove deprecated AbstractArchiveTask Gradle API usages (#58657) (#58894)
* Fix deprecated ArchiveTask configurations
2020-07-02 13:08:34 +02:00
Rene Groeschke d952b101e6
Replace compile configuration usage with api (7.x backport) (#58721)
* Replace compile configuration usage with api (#58451)

- Use java-library instead of plugin to allow api configuration usage
- Remove explicit references to runtime configurations in dependency declarations
- Make test runtime classpath input for testing convention
  - required as java library will by default not have build jar file
  - jar file is now explicit input of the task and gradle will ensure its properly build

* Fix compile usages in 7.x branch
2020-06-30 15:57:41 +02:00
Ryan Ernst eb16ad4574
Add plugin installer test for plugin jar contents (#58287)
The plugin installer currently checks the fake plugins installed contain
a single jar file. This commit adds another test for a plugin which
contains multiple jar files, ensuring all jars exist in the installed
plugin.
2020-06-24 09:50:02 -07:00
Rene Groeschke bd2dd81bc6
Fix deprecated property usage in archive tasks (#58269) (#58308) 2020-06-23 09:11:46 +02:00
Stuart Tettemer 20abba8433
Scripting: Deprecate general cache settings (#55753) (#58283)
Backport: ef543b0
2020-06-18 11:54:23 -06:00
Rene Groeschke abc72c1a27
Unify dependency licenses task configuration (#58116) (#58274)
- Remove duplicate dependency configuration
- Use task avoidance api accross the build
- Remove redundant licensesCheck config
2020-06-18 08:15:50 +02:00
Mark Vieira 5eb6692b0f
Add OpenJDK distribution to external dependency report (#58187) 2020-06-16 15:08:04 -07:00
Stuart Tettemer 01795d1925
Revert "Scripting: Deprecate general cache settings (#55753)" (#58201)
This reverts commit 88e8b34fc2.
2020-06-16 14:58:18 -06:00
Stuart Tettemer 88e8b34fc2
Scripting: Deprecate general cache settings (#55753)
Backport: ef543b0
2020-06-16 13:06:59 -06:00
Rene Groeschke 01e9126588
Remove deprecated usage of testCompile configuration (#57921) (#58083)
* Remove usage of deprecated testCompile configuration
* Replace testCompile usage by testImplementation
* Make testImplementation non transitive by default (as we did for testCompile)
* Update CONTRIBUTING about using testImplementation for test dependencies
* Fail on testCompile configuration usage
2020-06-14 22:30:44 +02:00
Rene Groeschke 5226fef321 Update Gradle wrapper to 6.5 (#57580) (#57653)
* Update Gradle wrapper to 6.5
* Fix groovy incompatibility issue after gradle update
* Fix Gstring String incompatibility
2020-06-12 08:38:16 +02:00
Mark Vieira 9b0f5a1589
Include vendored code notices in distribution notice files (#57017) (#57569)
(cherry picked from commit 627ef279fd29f8af63303bcaafd641aef0ffc586)
2020-06-04 10:34:24 -07:00
Rene Groeschke 2c2d903277 Revert "Update Gradle wrapper to 6.5 (#57580)"
This reverts commit b7e39dd1c8.
2020-06-04 11:01:11 +02:00
Rene Groeschke b7e39dd1c8 Update Gradle wrapper to 6.5 (#57580)
* Update Gradle wrapper to 6.5
* Fix groovy incompatibility issue after gradle update
* Fix Gstring String incompatibility
2020-06-04 10:34:11 +02:00
Rene Groeschke 8584da40af
Move classes from build scripts to buildSrc (#57197) (#57512)
* Move classes from build scripts to buildSrc

- move Run task
- move duplicate SanEvaluator

* Remove :run workaround

* Some little cleanup on build scripts on the way
2020-06-02 15:33:53 +02:00
Ryan Ernst c15cd33c88
Remove DEBUG-level logging from actions in docker (#57389)
In #51459 DEBUG-level logging was removed from the default log4j
configuration. However, our docker build has its own log4j configuration
which was missed in that change. This commit removes the same from the
docker log4j configuration.

relates #51459
relates #51198
2020-06-01 18:01:29 -07:00
Rene Groeschke 28920a45f1
Improvement usage of gradle task avoidance api (#56627) (#56981)
Use gradle task avoidance api wherever it is possible as a drop in replacement in the es build
2020-05-25 09:37:33 +02:00
Mark Vieira 73c00a9ce4
Ensure dependency report task depends on upstream reporting tasks (#56853) 2020-05-18 07:20:08 -07:00
Ryan Ernst 9fb80d3827
Move publishing configuration to a separate plugin (#56727)
This is another part of the breakup of the massive BuildPlugin. This PR
moves the code for configuring publications to a separate plugin. Most
of the time these publications are jar files, but this also supports the
zip publication we have for integ tests.
2020-05-14 20:23:07 -07:00
Mark Vieira 0fd756d511
Enforce strict license distribution requirements (#56642) 2020-05-14 13:57:56 -07:00
Rory Hunter bf97679854 Push arch-specific logic from Gradle into Docker (#56503)
Docker informed us that for official multi-arch Docker builds, there
needs to be a single Dockerfile and build context that can be used for
each supported architecture. Therefore, rework the build to move the
relevant architecture logic into the Dockerfile, and merge the aarch64
/ x64 docker context builds.
2020-05-12 14:31:45 +01:00
Rene Groeschke c29bc87040
Move bwcVersions extension property to BuildParams (back port) (#56381)
* Move bwcVersions extension property to BuildParams (#56206)
* Fix :qa Task Using Broken BwC Versions Resolution (#56332)

Co-authored-by: Armin Braun <me@obrown.io>
2020-05-11 09:39:13 +02:00
Mark Vieira 91cba091ed
Update Gradle wrapper to 6.4 (#55338) 2020-05-06 14:53:53 -07:00
Igor Motov d8f9df771d
Expose agg usage in Feature Usage API (#55732) (#56048)
Counts usage of the aggs and exposes them on the _nodes/usage/.

Closes #53746
2020-04-30 12:53:36 -04:00
Ryan Ernst 70b499b7aa
Simplify java home verification (#55635)
* Simplify java home verification

At one time, all uses of java home were found through the getJavaHome
utility method on BuildPlugin. However, that was changed many
refactorings ago, but the complex support for registering a java home
version needed that fails at configuration time still exists. The only
remaining use of grabbing java home is within bwc tests, and must be at
runtime since that is when we have the checkout and know what version is
needed.

This commit consolidates the java home finding method into a utility
unassociated with BuildPlugin.

* fix checkstyle

* address feedback
2020-04-27 12:43:32 -07:00
Rory Hunter c2c14e8875
Use official checksums to verify Tini (#55717)
Firstly, backport the use of tini as the Docker entrypoint. This was supposed
to have been done following #50277, but was missed. It isn't a direct backport
as this branch will continue using root as the initial Docker user.

Secondly, backport #55491 to use the official checksums when downloading tini.
2020-04-24 16:47:24 +01:00
Rory Hunter 6d2a5378a0 Rename docker context artifacts to satisfy release-manager (#55692)
Our release tool expects artifacts to have a certain naming convention. Rename
the Docker context artifacts to match this convention.
2020-04-24 10:48:18 +01:00
Tal Levy 0844455505
Add geo_shape mapper supporting doc-values in Spatial Plugin (#55037) (#55500)
After #53562, the `geo_shape` field mapper is registered within
a module. This opens the door for introducing a new `geo_shape`
field mapper into the Spatial Plugin that has doc-values support.

This is very much an extension of server's GeoShapeFieldMapper,
but with the addition of the doc values implementation.
2020-04-22 08:12:54 -07:00
William Brafford 49e30b15a2
Deprecate disabling basic-license features (#54816) (#55405)
We believe there's no longer a need to be able to disable basic-license
features completely using the "xpack.*.enabled" settings. If users don't
want to use those features, they simply don't need to use them. Having
such features always available lets us build more complex features that
assume basic-license features are present.

This commit deprecates settings of the form "xpack.*.enabled" for
basic-license features, excluding "security", which is a special case.
It also removes deprecated settings from integration tests and unit
tests where they're not directly relevant; e.g. monitoring and ILM are
no longer disabled in many integration tests.
2020-04-17 15:04:17 -04:00
Ryan Ernst 742b69a7dc
Make windows JAVA_HOME handling consistent with linux (#55261) (#55362)
In bash, checking for whether an env variable exists uses the -z test,
against a stringified env var, so that the test is actually whether the
env var is empty, but not necessarily undefined. We use this to test
whether JAVA_HOME is set, to determine whether the bundled jdk should be
used. In windows, this test is an actual "undefined" check. This commit
brings the behavior on two systems in sync, opting to allow for an empty
JAVA_HOME in windows to indicate the bundled jdk should be used.

closes #55134
2020-04-16 16:17:41 -07:00
David Roberts ac11dd619c
Only ship Linux binaries for the correct architecture (#55280)
Following elastic/ml-cpp#1135 there are now Linux binaries
for both x86_64 and aarch64.  The code that finds the
correct binaries to ship with each distribution was
including both on every Linux distribution.  This change
alters that logic to consider the architecture as well
as the operating system.

Also, there is no need to disable ML on aarch64 now that
we have the native binaries available.  ML is still not
supported on aarch64, but the processes at least run up
and work at a superficial level.

Backport of #55256
2020-04-16 09:45:52 +01:00
Ryan Ernst 29b70733ae
Use task avoidance with forbidden apis (#55034)
Currently forbidden apis accounts for 800+ tasks in the build. These
tasks are aggressively created by the plugin. In forbidden apis 3.0, we
will get task avoidance
(https://github.com/policeman-tools/forbidden-apis/pull/162), but we
need to ourselves use the same task avoidance mechanisms to not trigger
these task creations. This commit does that for our foribdden apis
usages, in preparation for upgrading to 3.0 when it is released.
2020-04-15 13:27:53 -07:00
Jason Tedor acf2acba4a
Exclude JDK Contents/MacOS directory on Darwin (#55197)
This directory interferes with notarization and removing it before we
notarize allows us to have a properly notarized
distribution. Conceptually, this directory is only needed when building
a distribution to be installed by Installer (a so-called "pkg"). Since
we are not building such distributions, and this directory interferes
with notarization, we choose to exclude it here. We do this here, rather
than in our notarization process, to ensure that what we run through CI
for testing is also what we ship to the world.
2020-04-15 09:28:47 -04:00
Rory Hunter 70616cd76a
Define aarch64 packaging test tasks (#55228)
Backport of #55073.

We added tasks to build an ARM distribution and Docker image, but didn't
provide any way to run packaging tests against them. Add extra loops on
the possible Architecture values, and skip tasks that can't be run on
the current Architecture.
2020-04-15 13:39:18 +01:00
Rory Hunter 18dc2f7330 Rename some Docker projects for consistency (#55150)
Apply the :distribution:archives naming convention to some of the Docker
sub-projects, so that we have a more consistent naming scheme.

Also, we've seen some examples of Docker packaging tests failing sporadically
when they try to clean up the temp directory, citing a not-empty
directory. Ensure that any running container is removed before cleaning
up the temp dir, in an effort to avoid this problem.
2020-04-14 22:09:05 +01:00
Mark Vieira ce85063653
[7.x] Re-add origin url information to publish POM files (#55173) 2020-04-14 13:24:15 -07:00
Mark Vieira dd73a14d11
Improve total build configuration time (#54611) (#54994)
This commit includes a number of changes to reduce overall build
configuration time. These optimizations include:

- Removing the usage of the 'nebula.info-scm' plugin. This plugin
   leverages jgit to load read various pieces of VCS information. This
   is mostly overkill and we have our own minimal implementation for
   determining the current commit id.
- Removing unnecessary build dependencies such as perforce and jgit
   now that we don't need them. This reduces our classpath considerably.
- Expanding the usage lazy task creation, particularly in our
   distribution projects. The archives and packages projects create
   lots of tasks with very complex configuration. Avoiding the creation
   of these tasks at configuration time gives us a nice boost.
2020-04-08 16:47:02 -07:00