Commit Graph

1761 Commits

Author SHA1 Message Date
Ryan Ernst a2864191d6
Move common repository configuration to java plugin (#57057)
This commit moves the common maven repository configuration to the ES
java plugin. All java projects need this common set of repos. Note that
the Elastic download and maven repos are removed, as they are not
necessary anymore since distribution download was split into the
DistributionDownloadPlugin.
2020-05-26 14:12:29 -07:00
Jake Landis 41629b7bfd
[7.x] Move license checks back to BuildPlugin (#56975) (#57169) 2020-05-26 15:18:12 -05:00
Rene Groeschke 28920a45f1
Improvement usage of gradle task avoidance api (#56627) (#56981)
Use gradle task avoidance api wherever it is possible as a drop in replacement in the es build
2020-05-25 09:37:33 +02:00
markharwood eb8cb31d46
Update Lucene version to 8.6.0-snapshot-9d6c738ffce (#57024)
Same version as master
2020-05-21 11:28:16 +01:00
Ryan Ernst 94062df77c
Make internalClusterTest run after unit tests (#57016)
This commit adds an ordering rule to ensure unit tests are run first
when running a higher level task like check.
2020-05-20 19:23:34 -07:00
Ryan Ernst 33ca5ba44b
Add individual precommit task plugins (#56926) (#57015)
Each precommit task is currently registered inside the shared
PrecommitTasks class. Having a single class with all precommit tasks
makes individualizing which precommit tasks are needed based on type of
project difficult, where we often just disable somet tasks eg for all qa
projects. This commit creates plugins for each precommit task, and moves
the configuration of the task into each plugin. PrecommitTasks remains
for now, and just delegates to each plugin, but will be removed in a
future change.
2020-05-20 14:12:36 -07:00
Ryan Ernst acfa2e4241
Split dependenciesInfo task to its own plugin (#56957)
The dependenciesInfo task is used to gather information about all
dependencies in elasticsearch. This commit separates it into its own
gradle plugin.
2020-05-20 10:26:05 -07:00
Ryan Ernst 381ed235e9
Re-enable plugin and upgrade bats tests (#51565) (#56999)
While the tests remained in place, the execution of these tests were
lost when creating the DistroTestPlugin. This commit restores executing
them.
2020-05-20 08:34:05 -07:00
Ioannis Kakavas 35bc386ac4
Stop disabling disgnostic trust manager in settings (#56916)
Now that #56526 is merged, we do not need to explicitly disable
the diagnostic trust manager for all of our test clusters - we do
this dynamically in runtime if the combination of java version and
JSSE provider dictates that.
2020-05-18 21:49:00 +03:00
Rene Groeschke b21330b00c Update gradle wrapper to 6.4.1 (#56881) 2020-05-18 17:14:55 +02:00
Ryan Ernst 9fb80d3827
Move publishing configuration to a separate plugin (#56727)
This is another part of the breakup of the massive BuildPlugin. This PR
moves the code for configuring publications to a separate plugin. Most
of the time these publications are jar files, but this also supports the
zip publication we have for integ tests.
2020-05-14 20:23:07 -07:00
Mark Vieira 0fd756d511
Enforce strict license distribution requirements (#56642) 2020-05-14 13:57:56 -07:00
Ignacio Vera b4521d5183
upgrade to Lucene 8.6.0 snapshot (#56661) 2020-05-13 14:25:16 +02:00
Ryan Ernst 902fc546bd
Migrate remaining ESIntegTestCases to internalClusterTest (#56479) (#56563)
This commit migrates the ESIntegTestCase tests in x-pack to the
internalClusterTest source set.
2020-05-11 21:06:04 -07:00
Rene Groeschke c29bc87040
Move bwcVersions extension property to BuildParams (back port) (#56381)
* Move bwcVersions extension property to BuildParams (#56206)
* Fix :qa Task Using Broken BwC Versions Resolution (#56332)

Co-authored-by: Armin Braun <me@obrown.io>
2020-05-11 09:39:13 +02:00
Mark Vieira 0fb9bc5379
Always use archive base name as the pom artifact id (#56447) (#56467) 2020-05-08 16:11:19 -07:00
Ryan Ernst 582145a493
Upgrade forbidden apis to 3.0 (#56368)
This commit upgrades forbidden apis to the latest version, which also
means we now get task configuration avoidance.
2020-05-07 19:05:07 -07:00
Nik Everett ca9bec7c1a Build: fix eclipse after icTests (#56362)
We made a small mistake when breaking out the `ESIntegTestCase`
subclasses that confused eclipse. This makes it happy again. Poor
eclipse!

Relates #55896
2020-05-07 13:29:43 -04:00
Ryan Ernst 33d6a55d1d
Create plugin for internalClusterTest task (#56067)
This commit creates a new gradle plugin to provide a separate task name
and source set for running ESIntegTestCase tests. The only project
converted to use the new plugin in this PR is server, as an example. The
remaining cases in x-pack will be handled in followups.

backport of #55896
2020-05-06 17:20:52 -07:00
Mark Vieira 91cba091ed
Update Gradle wrapper to 6.4 (#55338) 2020-05-06 14:53:53 -07:00
Jason Tedor 33669c0420
Upgrade to Jackson 2.10.4 (#56188)
Another Jackson release is available. There are some CVEs addressed,
none of which impact us, but since we can now bump Jackson easily, let
us move along with the train to avoid the false positives from security
scanners.
2020-05-06 17:20:23 -04:00
Mark Vieira f28a12cbba
Add version 7.9.0
This reverts commit 350e930e
2020-05-06 13:34:57 -07:00
Rory Hunter 350e930e55 Revert "Add version 7.9.0"
This reverts commit b8b4ebd089.
2020-05-06 14:24:55 +01:00
Rory Hunter b8b4ebd089 Add version 7.9.0 2020-05-06 09:12:14 +01:00
Tim Brooks 6a51017cb2
Upgrade netty to 4.1.49.Final (#56059) 2020-05-05 10:40:23 -06:00
Mark Vieira 462c0337a1
Fix input snapshotting for test tasks (#55981)
(cherry picked from commit 3b7ae2d14fa755aa1e868acbd018941fdb8380a8)
2020-04-29 15:27:27 -07:00
Ryan Ernst f8db1a56f8
Guard java9+ warn option in test config 2020-04-28 14:32:40 -07:00
Ryan Ernst 3f1a983ecb
Fix spotless...whitespace 2020-04-28 14:10:10 -07:00
Ryan Ernst 07f8c0368e
Split java plugin elements out of BuildPlugin (#55834)
BuildPlugin is a catch all for any elasticsearch common build
infrastructure. Unfortunately that makes reusing parts of it difficult.
This commit splits the parts specific to all java based projects out to
our own elasticsearch.java plugin.
2020-04-28 13:50:40 -07:00
Ryan Ernst 70b499b7aa
Simplify java home verification (#55635)
* Simplify java home verification

At one time, all uses of java home were found through the getJavaHome
utility method on BuildPlugin. However, that was changed many
refactorings ago, but the complex support for registering a java home
version needed that fails at configuration time still exists. The only
remaining use of grabbing java home is within bwc tests, and must be at
runtime since that is when we have the checkout and know what version is
needed.

This commit consolidates the java home finding method into a utility
unassociated with BuildPlugin.

* fix checkstyle

* address feedback
2020-04-27 12:43:32 -07:00
Ioannis Kakavas 55485cfa17
Print runtime.java in test failures (#55515) (#55624)
This commit adds `runtime.java` as a system property in our
nonInputProperties so that it will be available to be printed
upon test failure by ReproduceInfoPrinter.
2020-04-23 18:53:58 +03:00
Jake Landis 25ea6a74f0
[7.x] Validate REST specs against schema (#55117) (#55563)
A JSON schema was recently introduced for the REST API specification. #54252
This PR introduces a 3rd party validation tool to ensure that the
REST specification conforms to the schema.

The task is applied to the 3 projects that contain REST API specifications.
The plugin wires this task into the precommit commit task, and should be
considered as part of the public API for the build tools for any plugin
developer to contribute their plugin's specification.

An ignore parameter has been introduced for the task to allow specific
file to be ignored from the validation. The ignored files in this PR
will soon get issues logged and a link so they can be fixed.

Closes #54314
2020-04-22 14:14:03 -05:00
Stuart Tettemer 41748f02a5
Test: don't modify defaultConfig on upgrade (#55560) (#55599)
Backport: 58ec9c3
2020-04-22 11:07:27 -06:00
Jim Crowley 592b5516c1
Use AdoptOpenJDK API to Download JDKs (#55127) 2020-04-17 11:03:52 -07:00
Jake Landis eb30cf5c89
[7.x] Move Watcher config out of RestResourcesPlugin (#55136) (#55336) 2020-04-17 12:38:01 -05:00
Ryan Ernst a49ce45f1b
Removing warning for forbidden apis not supporting java 14 (#55359)
While the current version of forbidden apis still does not support java
14, the warning message has become very noisy as we now require java 14
for the elasticsearch build. This commit replaces the warn log message
with a comment in the code.
2020-04-17 10:27:49 -07:00
Ryan Ernst bed422162c
Move old jdk download tests to openjdk tests (#55361)
The "old jdk" tests are just testing support for downloading from oracle
prior to java 12.0.1, when oracle added a hash to the url. This commit
moves these tests into the openjdk tests (ie oracle download tests),
since adoptopenjdk does not have any change in behavior that needs to be
tested.
2020-04-16 17:23:12 -07:00
Ryan Ernst 9fb30942e0
Add pom validation (#55272) (#55364)
The pom files for our published artifacts are sent to maven central
during Elastic's release process, but we may not found out until then
that we have inadvertently broken the pom structure, as has happened
several times before. This commit adds validation of the pom file
specifically for the rules required by maven central.
2020-04-16 17:21:27 -07:00
Ryan Ernst 29b70733ae
Use task avoidance with forbidden apis (#55034)
Currently forbidden apis accounts for 800+ tasks in the build. These
tasks are aggressively created by the plugin. In forbidden apis 3.0, we
will get task avoidance
(https://github.com/policeman-tools/forbidden-apis/pull/162), but we
need to ourselves use the same task avoidance mechanisms to not trigger
these task creations. This commit does that for our foribdden apis
usages, in preparation for upgrading to 3.0 when it is released.
2020-04-15 13:27:53 -07:00
Ignacio Vera a677b63daa
Upgrade to lucene 8.5.1 release (#55229) (#55235)
Upgrade to lucene 8.5.1 release that contains a bug fix for a bug that might introduce index corruption when deleting data from an index that was previously shrunk.
2020-04-15 17:35:42 +02:00
Rory Hunter 70616cd76a
Define aarch64 packaging test tasks (#55228)
Backport of #55073.

We added tasks to build an ARM distribution and Docker image, but didn't
provide any way to run packaging tests against them. Add extra loops on
the possible Architecture values, and skip tasks that can't be run on
the current Architecture.
2020-04-15 13:39:18 +01:00
Ryan Ernst 5243295788
Update gradle in vagrant to use jdk14 (#55130)
This commit updates the Java version used within vagrant tests by gradle
inside the VM to use JDK14.

closes #55044
2020-04-14 17:00:12 -07:00
Mark Vieira ce85063653
[7.x] Re-add origin url information to publish POM files (#55173) 2020-04-14 13:24:15 -07:00
Jake Landis a2fafa6af4
[7.x] Lazy test cluster module and plugins (#54852) (#55087)
This change converts the module and plugin parameters
for testClusters to be lazy. Meaning that the values
are not resolved until they are actually used. This
removes the requirement to use project.afterEvaluate to
be able to resolve the bundle artifact.

Note - this does not completely remove the need for afterEvaluate
since it is still needed for the custom resource extension.
2020-04-13 10:53:35 -05:00
Ioannis Kakavas 7a8a66d9ae
[7.x] Fix ReloadSecureSettings API to consume password (#54771) (#55059)
The secure_settings_password was never taken into consideration in
the ReloadSecureSettings API. This commit fixes that and adds
necessary REST layer testing. Doing so, it also:

- Allows TestClusters to have a password protected keystore
so that it can be set for tests.
- Adds a parameter to the run task so that elastisearch can
be run with a password protected keystore from source.
2020-04-13 09:50:55 +03:00
Mark Vieira dd73a14d11
Improve total build configuration time (#54611) (#54994)
This commit includes a number of changes to reduce overall build
configuration time. These optimizations include:

- Removing the usage of the 'nebula.info-scm' plugin. This plugin
   leverages jgit to load read various pieces of VCS information. This
   is mostly overkill and we have our own minimal implementation for
   determining the current commit id.
- Removing unnecessary build dependencies such as perforce and jgit
   now that we don't need them. This reduces our classpath considerably.
- Expanding the usage lazy task creation, particularly in our
   distribution projects. The archives and packages projects create
   lots of tasks with very complex configuration. Avoiding the creation
   of these tasks at configuration time gives us a nice boost.
2020-04-08 16:47:02 -07:00
Jake Landis 2b970e2a8d
[7.x] Allow different source sets from forbiddenApis (#54731) (#54983)
ForbiddenApis task via the precommit task currently makes an assumption
that only the test and main source sets are present for any given project.
This commit removes that assumption and allows for any project source set's
compileClasspath class path to be added to the forbiddenApis classpath
configuration.
2020-04-08 16:31:04 -05:00
Jason Tedor 6853d73e88
Require JDK 14 for compilation (#54696)
This commit bumps the minimum JDK required for compilation to JDK 14.
2020-04-08 09:26:29 -04:00
Ryan Ernst 37795d259a
Remove guava from transitive compile classpath (#54309) (#54695)
Guava was removed from Elasticsearch many years ago, but remnants of it
remain due to transitive dependencies. When a dependency pulls guava
into the compile classpath, devs can inadvertently begin using methods
from guava without realizing it. This commit moves guava to a runtime
dependency in the modules that it is needed.

Note that one special case is the html sanitizer in watcher. The third
party dep uses guava in the PolicyFactory class signature. However, only
calling a method on the PolicyFactory actually causes the class to be
loaded, a reference alone does not trigger compilation to look at the
class implementation. There we utilize a MethodHandle for invoking the
relevant method at runtime, where guava will continue to exist.
2020-04-07 23:20:17 -07:00
James Baiera 2fa0d4cf3c
Move Fixture to minimumRuntime source (#54732) 2020-04-03 15:08:25 -04:00
Rory Hunter 83e900e248 Tweak formatting config for exception lists (#54710)
Change how we format exceptions to only wrap them as necessary. While
the config's overall philosophy is to put items one-per-line when
wrapping, in practice this is a little cumbersome for exception lists.
2020-04-03 16:54:31 +01:00
Jake Landis d412fd1bbd
[7.x] Convert RestIntegTestTask to Java (#54528) (#54585)
This commit migrates the RestIntegTestTask from groovy to Java.
No changes to logic should be included, however the following changes
are needed:

* Move Fixture interface to Java (Java can not depend on Groovy classes)
* Support lazy evaluation of non-input System parameters (can not use Groovy strings)
* Use constants for system property names
* Remove dead System property pass through code (the build plugin does this already)
2020-04-01 17:33:30 -05:00
Mark Vieira 2f619ad7d0
Improve checkstyle performance and readability (#54308)
Drop a nasty regex in our checkstyle config that I wrote a long time ago
in favor of a checkstyle extension. This is better because:
* It is faster. It saves a little more than a minute across the entire
  build.
* It is easier to read. Who knew 100 lines of Java would be easier to
  read than a regex, but it is.
* It has tests.
2020-03-27 08:05:04 -07:00
Rory Hunter 24ef800b24 Rename formatter config and switch to an opt-out list
Backport of #54276.

Move and rename formatter config file, so that it is easier for
Eclipse users to import.

Also switch to an opt-out list for formatting. Instead of explcitly
listing projects that should be formatted, instead list projects
that should not be formatted. This means that any new projects will
automatically be formatted and checked.
2020-03-27 12:09:37 +00:00
Mark Vieira b5d04e4c55
Fix typo in task name 2020-03-26 14:01:28 -07:00
Ryan Ernst 5a5d6e9ef2
Invert license security disabled helper method (#54043) (#54239)
Xpack license state contains a helper method to determine whether
security is disabled due to license level defaults. Most code needs to
know whether security is enabled, not disabled, but this method exists
so that the security being explicitly disabled can be distinguished from
licence level defaulting to disabled. However, in the case that security
is explicitly disabled, the handlers in question are never registered,
so security is implicitly not disabled explicitly, and thus we can share
a single method to know whether licensing is enabled.
2020-03-25 19:20:10 -07:00
Mark Vieira 611d98a472
Define lifecycle tasks for running different types of packaging tests (#54134)
(cherry picked from commit fd54cb087413d0247f5b60df381d6229e96227d8)
2020-03-25 16:51:32 -07:00
Mark Vieira 6e2ec94086
Upgrade to Gradle 6.3 (#53499) 2020-03-25 15:07:08 -07:00
Bogdan Pintea 77da9dd040 Add version 7.8.0
Add version 7.8.0
2020-03-25 18:10:30 +01:00
Mark Vieira 7728ccd920
Encore consistent compile options across all projects (#54120)
(cherry picked from commit ddd068a7e92dc140774598664efdc15155ab05c2)
2020-03-25 08:24:21 -07:00
Alan Woodward 39d7d0dc10 Upgrade to lucene 8.5.0 release (#54077)
Upgrades our lucene dependency to the released 8.5.0 version.
2020-03-24 13:45:50 +00:00
Mark Vieira 70cfedf542
Refactor global build info plugin to leverage JavaInstallationRegistry (#54026)
This commit removes the configuration time vs execution time distinction
with regards to certain BuildParms properties. Because of the cost of
determining Java versions for configuration JDK locations we deferred
this until execution time. This had two main downsides. First, we had
to implement all this build logic in tasks, which required a bunch of
additional plumbing and complexity. Second, because some information
wasn't known during configuration time, we had to nest any build logic
that depended on this in awkward callbacks.

We now defer to the JavaInstallationRegistry recently added in Gradle.
This utility uses a much more efficient method for probing Java
installations vs our jrunscript implementation. This, combined with some
optimizations to avoid probing the current JVM as well as deferring
some evaluation via Providers when probing installations for BWC builds
we can maintain effectively the same configuration time performance
while removing a bunch of complexity and runtime cost (snapshotting
inputs for the GenerateGlobalBuildInfoTask was very expensive). The end
result should be a much more responsive build execution in almost all
scenarios.

(cherry picked from commit ecdbd37f2e0f0447ed574b306adb64c19adc3ce1)
2020-03-23 15:30:10 -07:00
Jason Tedor 5a52ee3ef8
Fix typo in jdk-download testKit build.gradle
This commit fixes a typo in the jdk-download testKit build.gradle file
where "architecture" was not spelled correctly.
2020-03-23 15:05:30 -04:00
Mark Vieira eda023e2c3
Don't include HLRC on downstream classpath twice (#53983) 2020-03-23 08:49:34 -07:00
Jason Tedor 27c8bcbbd1
Introduce aarch64 packaging (#53914) (#53926)
This commit introduces aarch64 packaging, including bundling an aarch64
JDK distribution. We had to make some interesting choices here:
 - ML binaries are not compiled for aarch64, so for now we disable ML on
   aarch64
 - depending on underlying page sizes, we have to disable class data
   sharing
2020-03-22 11:58:11 -04:00
Armin Braun 5d3cb46232
Fix StackOverflowError in CheckStyle for CRUDDocumentationIT (#53840) (#53922)
This file seems to randomly cause a stack overflow like other HLRC ITs that
already have this check suppressed on them.

Closes #37647
2020-03-21 20:35:42 +01:00
Luca Cavanna 03fca61fcb [DOCS] add docs for async search (#53675)
Relates to #49091

Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2020-03-20 14:46:38 +01:00
Ryan Ernst f63c32ff31 Make lucene snapshot repository exclusive to lucene packages (#53353)
When depending on lucene snapshots we point maven at our own s3 backed
repository. However, in this case lucene packages should only be
retrieved from this location, and no other packages should ever be found
in that repo. This commit makes the maven repository exclusive to lucene
packages.
2020-03-19 15:04:04 -07:00
Jake Landis cce60215d8
[7.x] Add Watcher to available rest resources (#53620) (#53764)
Prior to this commit Watcher explicitly copied test between two
projects with a copy task. This commit removes the explicit copy in favor
of adding the Watcher tests to the available restResources that may be
copied between projects.

This is how inter-project dependencies should be modeled. However, only
Watcher is included here since it is (currently) the only project with
inter-project test dependencies.
2020-03-19 12:29:36 -05:00
Jason Tedor 569323928b
Upgrade the bundled JDK to JDK 14 (#53748)
This commit upgrades the bundled JDK to JDK 14.
2020-03-18 16:18:12 -04:00
Ryan Ernst 5c472fcb47 Upgrade jackson to 2.10.3 and GeoIP to 2.13.1 (#53642)
Re-applies the change from #53523 along with test fixes.

closes #53626
closes #53624
closes #53622
closes #53625

Co-authored-by: Nik Everett <nik9000@gmail.com>
Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>
Co-authored-by: Jake Landis <jake.landis@elastic.co>
2020-03-17 10:28:51 -07:00
Mark Vieira 497250ba9c
Improve performance of docs snippet length checkstyle rule (#53636)
The multiline regex rule used to detect docs code snippets greater than
76 characters in length has considerable cost to. For the high level
rest client project alone, this was taking upwards of 3 minutes. This
commit updates the rule regex pattern to use non-greedy matching when
appropriate which results in a lot fewer backtracks and about a 70%
reduction in execution time on the high level rest client module.
2020-03-17 09:58:07 -07:00
Mark Vieira 2f0aca992b
Revert "Upgrade to Jackson 2.10.3 and GeoIP2 to 2.13.1 (#53576)"
This reverts commit b7dbadeea0.
2020-03-15 18:10:40 -07:00
Jason Tedor b7dbadeea0
Upgrade to Jackson 2.10.3 and GeoIP2 to 2.13.1 (#53576)
This commit upgrades our Jackson dependency to 2.10.3 and our GeoIP2
dependency to 2.13.1.

Relates #53523
2020-03-14 13:28:06 -04:00
Jason Tedor 32dd852210
Update jackson-databind to 2.8.11.6 (#53522)
This commit upgrades the jackson-databind depdendency to
2.8.11.6. Additionally, we revert a previous change that put
ingest-geoip on the version of jackson-databind from the version
properties file. This is because upgrading ingest-geoip to a later
version of jackson-databind also requires an upgrade to the geoip2
dependency which is currently blocked. Therefore, if we can get to a
point where we otherwise upgrade our Jackson dependencies, we do not
want ingest-geoip to automatically come along with it.
2020-03-12 20:15:13 -04:00
Ryan Ernst 4301c35783 Use exclusiveContent gradle feature for jdk and distro downloads (#53358)
The jdk and distribution download plugins create fake ivy repositories,
and use group based repository filtering to ensure no other artifacts
try to resolve against the fake repos. Currently this works by adding a
blanket exclude to all repositories for the given group name. This
commit changes to using the new exclusiveContent feature in
Gradle to do the exclusion.
2020-03-11 16:35:15 -07:00
Alan Woodward 5c861cfe6e Upgrade to final lucene 8.5.0 snapshot (#53293)
Lucene 8.5.0 release candidates are imminent. This commit upgrades master to use
the latest snapshot to check that there are no last-minute bugs or regressions.
2020-03-10 09:32:59 +00:00
Jake Landis 6a5d9195aa
[7.x] Ensure only plugin REST tests are run for plugins (#5318… (#53196)
This commit fixes ensures that for external builds
(e.g. plugin development) that the REST tests that are
copied are properly filtered to only include the API
by default.

The code prior to this change resulted in including both
the API and tests since the copy.include resulted as an
empty list by default since the stream is empty unless
explicitly configured.

related #52114
fixes #53183
2020-03-05 17:41:17 -06:00
Nhat Nguyen 5476a49833 Revert "upgrade to lucene-snapshot-fa75139efea (#53150) (#53151)"
This reverts commit 058113aa42.
2020-03-05 17:33:00 -05:00
Mark Vieira 8851fb2a08
Upgrade to Gradle 6.2.2 (#53136) 2020-03-05 09:27:03 -08:00
Ignacio Vera 058113aa42
upgrade to lucene-snapshot-fa75139efea (#53150) (#53151) 2020-03-05 10:04:05 +01:00
Nhat Nguyen e6755afeeb
Upgrade to Lucene 8.5.0-snapshot-c4475920b08 (#52950) (#52977)
To give LUCENE-9228 more CI cycles
2020-02-29 09:29:16 -05:00
Mark Vieira c642a97255
Support parallel building of Docker images (#52920) 2020-02-27 19:22:06 -08:00
Jake Landis 1c29140f24
[7.x] Fix incremental build support copying REST api and tests (#52896)
A recent PR #52114 introduced two new tasks to copy the REST api and tests.
A couple bugs were found in that initial PR that prevents the incremental
build from working as expected.

The pattern match of empty string is equivalent to match all and it was coded
as match none. Fixed with explicit checks against empty patterns.

The fileCollection.plus return value was ignored. Fixed by changing how the
input's fileTree is constructed.

If a project has an src/test/resources directory, and tests are being copied
without a rest-api-spec/test directory could result no-op. Masked by the other
bugs and fixed by minor changes to logic to determine if a project has tests.
2020-02-27 12:44:00 -06:00
Mark Vieira 305c8342ac
Fix caching of build-tools project tests (#52848)
We embed the :reaper project jar in the build-tools jar so we can spawn
a reaper process at build runtime. Due to this, the jar technically
isn't part of the test runtime classpath, but for input snapshotting
purposes, we should be treating it as such. Instead, because it lives
in META-INF, Gradle treats it as a normal file, which in practice means
its hash changes on every build (timestamps, etc).

This commit changes our input snapshotting strategy such that instead
we explicitly add the jar as an input to any test tasks using Gradle's
runtime classpath normalization strategy (ignore timestamps, jar entry
order, etc) and ignore the file in META-INF. This ensures that we can
properly cache test results for build-tools, why still ensuring that
changes to the :reaper project trigger reexecution of tests.
2020-02-26 13:56:08 -08:00
JaeGeun 59638bc405
Add UTF-8 encoding to build.gradle in BuildSrc/. (#52674) 2020-02-26 11:41:52 -08:00
Jake Landis 8d311297ca
[7.x] Smarter copying of the rest specs and tests (#52114) (#52798)
* Smarter copying of the rest specs and tests (#52114)

This PR addresses the unnecessary copying of the rest specs and allows
for better semantics for which specs and tests are copied. By default
the rest specs will get copied if the project applies
`elasticsearch.standalone-rest-test` or `esplugin` and the project
has rest tests or you configure the custom extension `restResources`.

This PR also removes the need for dozens of places where the x-pack
specs were copied by supporting copying of the x-pack rest specs too.

The plugin/task introduced here can also copy the rest tests to the
local project through a similar configuration.

The new plugin/task allows a user to minimize the surface area of
which rest specs are copied. Per project can be configured to include
only a subset of the specs (or tests). Configuring a project to only
copy the specs when actually needed should help with build cache hit
rates since we can better define what is actually in use.
However, project level optimizations for build cache hit rates are
not included with this PR.

Also, with this PR you can no longer use the includePackaged flag on
integTest task.

The following items are included in this PR:
* new plugin: `elasticsearch.rest-resources`
* new tasks: CopyRestApiTask and CopyRestTestsTask - performs the copy
* new extension 'restResources'
```
restResources {
  restApi {
    includeCore 'foo' , 'bar' //will include the core specs that start with foo and bar
    includeXpack 'baz' //will include x-pack specs that start with baz
  }
  restTests {
    includeCore 'foo', 'bar' //will include the core tests that start with foo and bar
    includeXpack 'baz' //will include the x-pack tests that start with baz
  }
}

```
2020-02-26 08:13:41 -06:00
Mark Vieira 528e77cd5e
Update Docker on linux exclusion list (#52736)
(cherry picked from commit 5cfbf4dac11225d9f48fc25919e29713dd53a279)
2020-02-25 09:24:38 -08:00
Mark Vieira ac654692d4
Upgrade to Gradle 6.2.1 (#52732) 2020-02-24 16:34:53 -08:00
Mark Vieira f06d692706
[Backport] Consolidate docker availability logic (#52656) 2020-02-21 15:24:05 -08:00
markharwood 96d603979b
Upgrade Lucene to 8.5.0-snapshot-b01d7cb (#52584)
Upgrading 7x to same Lucene 8.5 version used in master
2020-02-21 10:25:03 +00:00
Mark Vieira a99d8b71b1
Upgrade to Gradle 6.2 (#51824) 2020-02-18 15:35:23 -08:00
Tim Brooks e752221fc6
Upgrade netty to 4.1.45.Final (#51689)
Upgrade netty.
2020-02-18 09:11:29 -07:00
Ioannis Kakavas d9ce0e6733
Update BouncyCastle to 1.64 (#52185) (#52464)
This commit upgrades the bouncycastle dependency from 1.61 to 1.64.
2020-02-18 14:11:34 +02:00
Mark Vieira c61124a7b9
Fix input snapshotting error when using test clusters cliSetup (#52340) 2020-02-13 14:24:23 -08:00
Mark Vieira 28c56da754
Don't track absolute path as test input to improve cacheability (#52235) 2020-02-11 13:32:59 -08:00
Jason Tedor bb2e04bc16
Use absolute path for temporary directory in tests (#52228)
We explicitly set the path for the temporary directory to use in test
tasks, but today this path is a relative path, relative to the current
working directory of the test task. The fact that we are using a
relative path here appears to be legacy, simply leftover from the days
of the Maven build. An absolute path is preferred here, since it's
explicit and we do not have to rely on everyone resolving the path
properly relative to the working directory.
2020-02-11 15:17:45 -05:00
Jason Tedor 6ed3311443
Ensure test temporary directory exists (#52227)
Today we we set the test temporary directory explicitly by controling
java.io.tmpdir. Yet, we do not guarantee this directory exists, instead
relying on a test base class (LuceneTestCase) to create this directory
when it initializes. However, some of our tests do not rely on our test
framework, and thus do not have access to LuceneTestCase, instead
relying on RandomizedRunner directly. We should not be relying on the
temporary directory being implicitly created, instead guaranteeing that
it exists before test execution starts. This commit does that by
creating the test temporary directory before the test task executes (via
a doFirst).
2020-02-11 14:53:16 -05:00
Ryan Ernst 5a72b23716
Migrate SysV init tests from bats to java packaging (#51077) (#51498)
This commit converts the sysv init tests from bats tests into the java
packaging tests. Since it is the last oss specific test, the bats oss
test task is also removed.

relates #46005
2020-02-10 17:41:33 -05:00
Jason Tedor d188dda7eb
Move docker-compose logging statement to debug (#52107)
When docker-compose is required for a test fixture but is not
available, we warn log a message to this effect. This ends up being
noise during configuration, especially when working locally. This
commit changes the logging level of these messages to debug.
2020-02-10 13:13:36 -05:00