Commit Graph

1844 Commits

Author SHA1 Message Date
Rene Groeschke 3a8cfdc1f5
Extract distribution archive checks into plugin (7.x backport) (#61567)
- Added test coverage
- Removes build script cluttering
- Splits archive building and archive checking logic
- only rely on boost for now for ML licenses(tbd)
- Use Gradle build-in untar and unzip support

* Handle dynamic versions in func tests assertions
2020-08-26 15:04:12 +02:00
Rene Groeschke fac66a7528
Rework test cluster distribution handling (#61407) (#61566)
Driven by this issue 
https://github.com/elastic/elasticsearch/pull/60969#issuecomment-674962158
we apply some rework on how we handle distributions in our test cluster setups:

- If no custom modules, plugins or extra jar files are declared we do not create a cluster
specific distro folder and use the origin distribution folder instead.
- If a custom distribution folder is required, we fallback to file copy when hard linking
is not supported
2020-08-26 15:03:52 +02:00
Ryan Ernst d1031fd928
Move versions convenience property to java plugin (#61438)
Access the common versions map is done in a lot of places. While it can
be access through an import of VersionProperties, the vast majority of
places use it through the provided convenience property added by
BuildPlugin. This commit moves that convenience property to the base
java plugin, so further reduce dependence on the BuildPlugin.
2020-08-21 16:43:02 -07:00
Rory Hunter 890f943d0f
Fix tests that weren't expecting UBI Docker images (#61361)
Closes #61360.
2020-08-20 13:30:21 +01:00
Rory Hunter d8aacbdd31 Add UBI docker builds
Backport of #60742.

This PR resurrects support for building Docker images based on one of
Red Hat's UBI images. It also adds support for running the existing
Docker tests against the image. The image is named
`elasticsearch-ubi8:<version>`.

I also changed the Docker build file uses enums instead strings in a lot
of places, for added rigour.
2020-08-18 09:48:51 +01:00
Mark Vieira d26a3e144e
Revert "Simplify distribution download and extraction (7.x backport) (#61184)"
This reverts commit 8b7a0a1f64.
2020-08-17 08:54:08 -07:00
Rene Groeschke 8b7a0a1f64
Simplify distribution download and extraction (7.x backport) (#61184)
We leverage artifact transforms now when downloading and unpacking elasticsearch distributions.

This has the benefit of

- handcrafted extract tasks on the root project are not required. 
- the general tight coupling to the root project has been removed.
- the overall required configurations required to handle a distribution have been reduced
- ElasticsearchDistribution has been simplified by making Extracted an ordinary Configuration
downloaded and unpacked external distributions are reused in later builds by been cached
in the gradle user home.

DistributionDownloadPlugin functional tests have been extended and ported
to DistributionDownloadPluginFuncTest.

* Fix java8 compliant Path calculation
2020-08-17 10:10:32 +02:00
Rene Groeschke 9ca052d702
Use native Gradle support for --release flag (#61093) (#61151)
With Gradle 6.6 we can now use the native support for the --release compile options. 
As Gradle by default resolves the compatibility version from the release property we needed to workaround this in order to keep our current setup. An issue was raised at gradle to track this at https://github.com/gradle/gradle/issues/14141
2020-08-14 20:01:24 +02:00
Jake Landis cb9f4cdae2
Fix the REST FIPS tests (#61001)
Adds bouncycastle to classpath for tests and testclusters
2020-08-13 16:23:54 -07:00
Ryan Ernst ef796e589c
Fix typo in Taskkill in reaper (#61076)
The reaper is used only when spawned processes by the build like
testclusters or vagrant are left behind in a failed build. When this
occurs on Windows, the Taskkill command is used. However, there was a
typo in the command name. Since the reaper is for edgecases and mostly
protection for local builds (CI uses immutable hosts), this has been
broken since its inception.

closes #60503
2020-08-13 10:21:13 -07:00
Rene Groeschke a5ef38ca40
Update gradle wrapper to 6.6 (#59909) (#60949) 2020-08-11 11:03:19 +02:00
Rene Groeschke e03993d238
Make jdk download repo not consumable (#60875)
- fixes #60860
2020-08-10 14:22:42 +02:00
Ryan Ernst cf39a69be3
Exit gradle run task if any node dies (#60789)
The Elasticsearch `run` task in Gradle is backed by testclusters
spinning up a node and waiting indefinitely on output from that node.
However, this thread currently waits for an interrupt to exit. If
Elasticsearch fails to start, we hang indefinitely. This commit adds a
check after each poll of the process stdout to see if it is still alive,
and fails the build if it has died.

closes #60761
2020-08-06 15:54:43 -07:00
Ryan Ernst 479f32906d
Cleanup vagrant setup (#60697)
Though bats tests were recently removed, there remains a few unnecessary
setups needed for those tests in our vagrant files, as well as CI setup.
Additionally, we no longer rely on the vagrant images setting a
JAVA_HOME, instead relying on DistroTestPlugin to pull the appropriate
jdk when testing with no-jdk distributions.
2020-08-06 15:37:56 -07:00
Rene Groeschke 91e956e96a
Simplify jdk downloads via artifact transforms (#60713) (#60803)
* Simplify jdk downloads via artifact transforms

This reworks the jdk download plugin to use artifact transforms.
That allows us to simplify the jdk download plugin by a lot. The
benefits of using artifact transforms are:

- they transform an artifact to an upacked folder on the fly as part
of the dependenc resolution allowing us to remove all the custom created unpack tasks
and configurations

- Artifact transforms support gradle build cache. Requesting a jdk folder on a clean machine
will likely be resolved from the build cache

- The manual mingling about not extracting jdks multiple times by all jdks channeling through
root project configurations can be removed as they support up-to-date checking and build cache
which will ensure these archives are only resolved once per build at max.

Also the test coverage has been ported to Spock that supports data driven testing. This porting
includes an introduction to a wiremock fixture that can be used later on for mocking repository
urls in other integration tests

* Simplify artifact transform registration

* Change jdk finalization and repository setup

Jdk finalization is now done via `configuration.defaultDependencies`.
This has two benefits:
- we only configure and finalize the JDKs we actually use during the build
- afterEvaluate hooks are problematic in combination with task avoidance api.
Relying on task avoidance api in Gradle moves more configuration logic into the materialization
of tasks which is, with task avoidance api, is done ideally during the task graph calculation.
Anything created (e.g. jdks) created late in this task graph calculation would never be finalized
via afterEvaluate hooks as this has been fired before already with the jdk not there yet.

Furthermore we now only configure repositories in the projects where the jdk is declared
(aka the jdk-download plugin is applied) and only if the jdk is actually requested during the build.

* Fix javadoc

* Fix jdk download repo content filtering

* Minor cleanup

* Apply review feedback and cleanup
2020-08-06 12:34:20 +02:00
Ryan Ernst 7514526b6b
Convert packaging upgrade tests to java (#60560) (#60680)
This commit removes the last of the bats tests, converting the rpm/deb
upgrade tests to java. It adds a new pattern of tasks, similar in nature
but separate from the existing distro tests, named `distroUpgradeTest`.
For each index compatible version, a `distroUpgradeTest.VERSION` task
exxists. Each distribution then has a task, named
`distroUpgradeTest.VERSION.DISTRO`.

One thing to note is these new tests do not cover no-jdk versions of
the rpm/deb packages, since the distribution/bwc project does not
currently build those.

closes #59145
closes #46005
2020-08-05 14:05:37 -07:00
Jake Landis 5db283262e
[7.x] Apply new REST base plugin for the javaRestTest plugin (#60627) (#60672)
This commit applies the new base REST plugin for javaRestTest plugin.

related: #60624
2020-08-04 13:14:08 -05:00
Rene Groeschke bdd7347bbf
Merge test runner task into RestIntegTest (7.x backport) (#60600)
* Merge test runner task into RestIntegTest (#60261)
* Merge test runner task into RestIntegTest
* Reorganizing Standalone runner and RestIntegTest task
* Rework general test task configuration and extension
* Fix merge issues
* use former 7.x common test configuration
2020-08-04 14:46:32 +02:00
Yannick Welsch b0d601fa63 Adjust searchable snapshot license (#60578)
No longer needs Platinum license for testing on staging.
2020-08-03 13:19:53 +02:00
Rene Groeschke ed4b70190b
Replace immediate task creations by using task avoidance api (#60071) (#60504)
- Replace immediate task creations by using task avoidance api
- One step closer to #56610
- Still many tasks are created during configuration phase. Tackled in separate steps
2020-07-31 13:09:04 +02:00
Rene Groeschke 79ec0f2442
Split internal distribution handling into separate internal plugin (7.x backport) (#60270)
* Split internal distribution handling into separate internal plugin (#60295)
* Provide proper failure if unexpected non jdk bundled bwc version is requested
2020-07-31 09:30:21 +02:00
Jake Landis 6ce30bea08
[7.x] Convert most OSS plugins from integTest to [yaml | java]RestTest or internalClusterTest (#59444) (#60343)
For all OSS plugins (except repository-* and discovery-*) integTest
task is now a no-op and all of the tests are now executed via a test,
yamlRestTest, javaRestTest, or internalClusterTest.

This commit does NOT convert the discovery-* and repository-* since they
are bit more complex then the rest of tests and this PR is large enough.
Those plugins will be addressed in a future PR(s).

This commit also fixes a minor issue that did not copy the rest api
for projects that only had YAML TEST tests.

related: #56841
2020-07-29 13:06:13 -05:00
Mark Vieira 39fa1c4df0
Add compatibility testing for JDBC driver (#60409)
This commit adds compatibility testing of our JDBC driver against
different Elasticsearch versions. Although we are really testing the
forwards compatibility nature of the JDBC driver we model the testing
the same as we do existing BWC tests, that is, with the current branch
fetching the earlier versions of the artifact that is to be tested. In
this case, that's the JDBC driver itself.

Because the tests include the JDBC driver jar on it's classpath we had
to change the packaging of the driver jar in order to avoid jarhell and
other conflicting dependency issues when using an old JDBC driver with
later branches. For this we simply relocate all driver dependencies in
the shadow jar under a "shadowed" package. This allows the JDBC driver
to use the correct version of Elasticsearch libs classes, while the
tests themselves use their versions. Since this required a change to the
driver jar compatibility testing can only go back as far as that version
which at the time of this commit is 7.8.1.
2020-07-29 10:45:11 -07:00
Jake Landis 92ce41cfaf
[7.x] Introduce javaRestTest source set/task and convert modules (#59939) (#60026)
Introduce a javaRestTest source set and task to compliment the yamlRestTest.
javaRestTest differs such that the code is sourced from Java and may have
different dependencies and setup requirements for the test clusters. This also
allows the tests to run in parallel in different cluster instances to prevent any
cross test contamination between the two types of tests.

Included in this PR is all :modules no longer use the integTest task. The tests
are now driven by test, yamlRestTest, javaRestTest, and internalClusterTest.
Since only :modules (and :rest-api-spec) have been converted to yamlRestTest
we can now disable the integTest task if either yamlRestTest or javaRestTest have
been applied. Once all projects are converted, we can delete the integTest task.

related: #56841
related: #59444
2020-07-28 08:39:11 -05:00
Rene Groeschke 3fe6635b92
Remove stale gradle plugin descriptor 2020-07-22 09:10:01 +02:00
Rene Groeschke c6d3af35b9
Simplify gradle test task error reporting (#59760) (#59964)
* Simplify test error reporting

- avoid using extra plugin
- avoid extra task listener (should be avoided related to #57918 )
- keep all logic in the listener
2020-07-21 14:13:35 +02:00
Rene Groeschke 60d46f1d13
Remove superflous enforce deprecation failure plugin (#59770) (#59898)
- With enforcing the build to fail on all gradle deprecated api usage we do not need
this tailored plugin anymore
2020-07-21 12:54:17 +02:00
Rene Groeschke e31ebc96f9
Enforce fail on deprecated gradle usage (7.x backport) (#59758)
* Enforce fail on deprecated gradle usage (#59598)
* Fix branch specific deprecated gradle api usages
* Fix archiveVersion property usage
2020-07-20 08:52:30 +02:00
Ryan Ernst e963918830
Ensure precommit runs as part of check (#59476)
Precommit is setup to run as a dependency of the check task, but
unfortunately this wiring was only happening when the java plugin (but
not java-base plugin) was applied. This commit moves the wiring to occur
whenever the check task exists, which is with the lifecycle-base plugin.
2020-07-17 15:41:37 -07:00
Rene Groeschke afbc4678ae
Remove deprecated compileOnly resolution (#58857) (#59709)
- Ideally this should not be required but current runtime classpath
seems to require this. We need to revisit the classpath declarations in detail
to be able to remove this type of (in theory) superflous compile only references

Related to #57920
2020-07-17 12:06:57 +02:00
Rene Groeschke 3a228906a9
Fix license header checks on buildSrc (#58457) (#59705) 2020-07-16 16:33:30 +02:00
Rory Hunter 2e05ce5f88 Bump version to 7.10.0 2020-07-15 11:56:45 +01:00
Ignacio Vera f8037abf47
upgrade to lucene-8.6.0 release (#59596) (#59599) 2020-07-15 12:40:57 +02:00
Rene Groeschke d5c11479da
Remove remaining deprecated api usages (#59231) (#59498)
- Fix duplicate path deprecation by removing duplicate test resources
- fix deprecated non annotated input property in LazyPropertyList
- fix deprecated usage of AbstractArchiveTask.version
- Resolve correct test resources
2020-07-14 10:25:00 +02:00
Rene Groeschke 68dd431bc9
Fix deprecated unsave project outputs resolution (#59088) (#59356)
- Fixes how libs in distribution are resolved
- Required minor rework on common repository setup to allow distribution projects
to resolve thirdparty artifacts
- Use Default configurations when resolving tools for distribution packaging
- Related to #57920
2020-07-10 17:16:47 +02:00
Ryan Ernst 04a531dcfa
Use consistent system jdk in 7.x packaging tests (#59257)
In 7.x, we have java 8 as minimum jdk version. In the past, for
packaging tests, we relied on the system to provide an alternative jdk
to be used by the no-jdk distributions. Master switched to using a build
provided jdk, but 7.x was stuck relying on the system because it needed
a java 8 jdk. The jdk download plugin was updated a while ago to support
jdk 8, and so this PR converts the distro tests to use the build
provided jdk just as master branch does.

Note also this fixes a failure that would sometimes occur on older jdks
in windows where the expected gc filename can be different.
2020-07-08 15:23:59 -07:00
Rene Groeschke a896df53ac
Remove misc dependency related deprecation warnings (7.x backport) (#59122)
* Fix dependency related deprecations (#58892)
* Fix classpath setup for forbiddenapi usage
2020-07-07 17:10:31 +02:00
Ignacio Vera 5cc6457ed8
upgrade to lucene-8.6.0-snapshot-6a715e2ecc3 (#59091) (#59120) 2020-07-07 12:07:41 +02:00
Jake Landis 604c6dd528
7.x - Create plugin for yamlTest task (#56841) (#59090)
This commit creates a new Gradle plugin to provide a separate task name
and source set for running YAML based REST tests. The only project
converted to use the new plugin in this PR is distribution/archives/integ-test-zip.
For which the testing has been moved to :rest-api-spec since it makes the most
sense and it avoids a small but awkward change to the distribution plugin.

The remaining cases in modules, plugins, and x-pack will be handled in followups.

This plugin is distinctly different from the plugin introduced in #55896 since
the YAML REST tests are intended to be black box tests over HTTP. As such they
should not (by default) have access to the classpath for that which they are testing.

The YAML based REST tests will be moved to separate source sets (yamlRestTest).
The which source is the target for the test resources is dependent on if this
new plugin is applied. If it is not applied, it will default to the test source
set.

Further, this introduces a breaking change for plugin developers that
use the YAML testing framework. They will now need to either use the new source set
and matching task, or configure the rest resources to use the old "test" source set that
matches the old integTest task. (The former should be preferred).

As part of this change (which is also breaking for plugin developers) the
rest resources plugin has been removed from the build plugin and now requires
either explicit application or application via the new YAML REST test plugin.

Plugin developers should be able to fix the breaking changes to the YAML tests
by adding apply plugin: 'elasticsearch.yaml-rest-test' and moving the YAML tests
under a yamlRestTest folder (instead of test)
2020-07-06 14:16:26 -05:00
Rene Groeschke 0a6e6ef4be
Fix task input declaration when using LazyPropertyMap (#58850) (#58934) 2020-07-02 20:52:31 +02:00
Rene Groeschke 70713a0a19
Remove deprecated AbstractArchiveTask Gradle API usages (#58657) (#58894)
* Fix deprecated ArchiveTask configurations
2020-07-02 13:08:34 +02:00
Mark Vieira 402110c557
[7.x] Upgrade Gradle docker-compose plugin (#58873) 2020-07-01 16:08:46 -07:00
Rene Groeschke 4f1da31158
Remove minimumRuntime sourceSet from build-tools (7.x backport) (#58472)
* Remove minimumRuntime sourceSet from build-tools
* remove license change in loggedexec task
2020-07-01 15:12:59 +02:00
Rene Groeschke 2e278eaf3c
Detangle JdkJarHellCheck from build tool building (#58601) (#58755)
* Detangle JdkJarHellCheck from build tool building

- allows building the tool with same runtime as es
- allows building build tools with newer runtime version and keep ThirdPartyAuditTask
running with minimum runtime to ensure we check against correct jre
- add jdkjarhell test jar setup into fixture
2020-07-01 10:41:01 +02:00
Mark Vieira d81b455532
Fix comment formatting 2020-06-30 22:18:04 -07:00
Mark Vieira 119462dc85
Temporarily disable docker-compose test fixtures 2020-06-30 22:08:48 -07:00
Rene Groeschke d952b101e6
Replace compile configuration usage with api (7.x backport) (#58721)
* Replace compile configuration usage with api (#58451)

- Use java-library instead of plugin to allow api configuration usage
- Remove explicit references to runtime configurations in dependency declarations
- Make test runtime classpath input for testing convention
  - required as java library will by default not have build jar file
  - jar file is now explicit input of the task and gradle will ensure its properly build

* Fix compile usages in 7.x branch
2020-06-30 15:57:41 +02:00
Ryan Ernst 67a5a2bb60
Remove vagrant debug logging (#58629)
Previously we had odd issues in CI with vagrant which we attempted to
diagnose by enabling debug logging within our vagrant commands. However,
the debug logging didn't explain anything additional to the failures,
and it is extremely verbose. This commit removes the debug logging.
2020-06-29 13:33:07 -07:00
Ryan Ernst a524800b3e
Migrate plugin packaging tests to java (#58518)
This commit converts the bats tests for the plugin cli into the java
packaging test framework. The new tests only use the example plugin to
test the plugin cli. The tests for each individual plugin's contents
after being installed are handled by a new unit test for the plugin
installer added in #58287.
2020-06-25 14:16:33 -07:00
James Baiera 89243857ce
Update precommit to filter out project dependencies (#58189) (#58572)
If a project is pulling in an external org.elasticsearch dependency, the dependency
report generation would require a license file for the dependency to be present. 
This would break precommit because a license was present that it did not feel was
warranted. This un-reverts the update to the dependenciesInfo task, as well as the 
JNA license addition.
2020-06-25 16:33:25 -04:00
Rory Hunter e413de4203 Validate that REST API names do not contain keywords (#58452)
If an API name (or components of a name) overlaps with a reserved word in
the programming language for an ES client, then it's possible that the code
that is generated from the API will not compile. This PR adds validation to
check for such overlaps.
2020-06-25 09:48:54 +01:00
Rene Groeschke fc60cf6179
Introduce EnforceDeprecationFailuresPlugin (#58263) (#58309)
- extract fail on deprecated usage into its own plugin
- apply on all projects
- ensures we don't miss any project (missed xpack/plugin/eql/qa/security before)
2020-06-23 09:14:12 +02:00
Stuart Tettemer 20abba8433
Scripting: Deprecate general cache settings (#55753) (#58283)
Backport: ef543b0
2020-06-18 11:54:23 -06:00
Rene Groeschke abc72c1a27
Unify dependency licenses task configuration (#58116) (#58274)
- Remove duplicate dependency configuration
- Use task avoidance api accross the build
- Remove redundant licensesCheck config
2020-06-18 08:15:50 +02:00
Ryan Ernst c57a3f3e60
Cleanup jdk repro parameters (#57838)
The version of java printed when a test fails currently is passed in
from gradle. However, we already know this from java itself, so it is
not necessary. This commit changes how the runtime.java repro parameter
is found, as well as removes the compiler.java parameter which is no
longer relevant.

closes #57756
2020-06-17 20:53:00 -07:00
Jason Tedor b78b3edeea
Upgrade to JNA 5.5.0 (#58183)
This commit bumps our JNA dependency from 4.5.1 to 5.5.0, so that we are
now on the latest maintained line, and pick up a large collection of bug
fixes that have accumulated.
2020-06-17 07:35:08 -04:00
Mark Vieira 5eb6692b0f
Add OpenJDK distribution to external dependency report (#58187) 2020-06-16 15:08:04 -07:00
Stuart Tettemer 01795d1925
Revert "Scripting: Deprecate general cache settings (#55753)" (#58201)
This reverts commit 88e8b34fc2.
2020-06-16 14:58:18 -06:00
Stuart Tettemer 88e8b34fc2
Scripting: Deprecate general cache settings (#55753)
Backport: ef543b0
2020-06-16 13:06:59 -06:00
Rene Groeschke 9be89e8aca Make Gradle IT tests debuggable (#58093)
- This configures the testkit gradle runner to run with debug enabled automatically when
test is executed in debug mode (e.g. from the IDE)
- Allows step by step debugging of GradleIntegrationTestCase tests
2020-06-15 11:52:20 +02:00
Rene Groeschke 01e9126588
Remove deprecated usage of testCompile configuration (#57921) (#58083)
* Remove usage of deprecated testCompile configuration
* Replace testCompile usage by testImplementation
* Make testImplementation non transitive by default (as we did for testCompile)
* Update CONTRIBUTING about using testImplementation for test dependencies
* Fail on testCompile configuration usage
2020-06-14 22:30:44 +02:00
Jason Tedor ddf2ee3f23
Revert "Make dependenciesInfo task filter out project dependencies (#58015) (#58057)"
This reverts commit e8e351769f.
2020-06-12 17:04:57 -04:00
James Baiera e8e351769f
Make dependenciesInfo task filter out project dependencies (#58015) (#58057)
By default we filter out projects from dependencies info task based on their 
group names. We should be filtering based on if the dependency is a project 
dependency.
2020-06-12 13:41:29 -04:00
Rene Groeschke cb78c4d29d
Fix info message for RunTask in debug mode (#57974) (#58003)
Fixes #57860

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-06-12 13:32:03 +02:00
Rene Groeschke 5226fef321 Update Gradle wrapper to 6.5 (#57580) (#57653)
* Update Gradle wrapper to 6.5
* Fix groovy incompatibility issue after gradle update
* Fix Gstring String incompatibility
2020-06-12 08:38:16 +02:00
Alan Woodward 16e230dcb8 Update to lucene snapshot e7c625430ed (#57981)
Includes LUCENE-9148 and LUCENE-9398, which splits the BKD metadata, index and data into separate files and keeps the index off-heap.
2020-06-11 14:51:53 +01:00
Mark Vieira 9b0f5a1589
Include vendored code notices in distribution notice files (#57017) (#57569)
(cherry picked from commit 627ef279fd29f8af63303bcaafd641aef0ffc586)
2020-06-04 10:34:24 -07:00
Rene Groeschke 751f16858b
Remove duplicate ssl setup in sql/qa projects (#57319) (#57643)
* Remove duplicate ssl setup in sql/qa projects
* Fix enforcement of task instances
* Use static data for cert generation
* Move ssl testing logic into a plugin
* Document test cert creation
2020-06-04 14:53:23 +02:00
Rene Groeschke 2c2d903277 Revert "Update Gradle wrapper to 6.5 (#57580)"
This reverts commit b7e39dd1c8.
2020-06-04 11:01:11 +02:00
Rene Groeschke b7e39dd1c8 Update Gradle wrapper to 6.5 (#57580)
* Update Gradle wrapper to 6.5
* Fix groovy incompatibility issue after gradle update
* Fix Gstring String incompatibility
2020-06-04 10:34:11 +02:00
Rene Groeschke 8584da40af
Move classes from build scripts to buildSrc (#57197) (#57512)
* Move classes from build scripts to buildSrc

- move Run task
- move duplicate SanEvaluator

* Remove :run workaround

* Some little cleanup on build scripts on the way
2020-06-02 15:33:53 +02:00
Ioannis Kakavas 02bf5fff0f
Disable testingConventions in build tools in fips (#57357) (#57503)
Necessary since we also disable unit tests and 
testingConventions would fail in FIPS 140 mode.
2020-06-02 10:28:58 +03:00
Ryan Ernst 88ea1f97b8
Remove buildResources from BuildPlugin (#57482)
The shared buildResources task is a catch all for resources needing to
be copied from the build-tools jar at runtime. Utilizing this for all
resources causes any tasks using resources from this to be triggered on
any changes to any of those files. This commit creates separate export
tasks per usage, and removes the buildResources task.
2020-06-01 15:11:45 -07:00
Ryan Ernst 6ccdceec79
Move test fips configuration to script plugin (#57251)
This commit moves the configuration of all test jvms for fips to a
script plugin. Fips testing is something very specific to the
Elasticsearch build and does not need to be passed on to plugin authors.
2020-06-01 10:43:54 -07:00
Rene Groeschke 7a4a6360b7
Fix up-to-date checks for precommit related tasks (#57203) (#57291)
* Fix up-to-date checks for precommit related tasks

- Do not use lambdas for doFirst / doLast action declarations as this is not supported by gradle up-to-date check
- Use marker output folder for dependencies license task to make task incremental build compliant

* Tweak formatting
2020-05-28 17:12:17 +02:00
Rene Groeschke 51158a2d8b
Fix deprecation warning in ThirdpartyAuditTask (#57123) (#57224) 2020-05-28 14:41:42 +02:00
Ryan Ernst 1b59e9ab22
Move test error reporting to java plugin (#57259)
This commit moves the global hook for reporting failed test cases to the
ElasticsearchJavaPlugin. It should always be applied for all java
projects since the Test class is what emits the failures logged.
2020-05-27 17:41:04 -07:00
Ryan Ernst 97353297dc
Move gradle version check to global build info plugin (#57255)
The gradle version check currently exists in BuildPlugin. However, there
is no reason to check this within every project. Instead, this commit
moves the check to the global build info, which is only applied to the
root project. Additionally, this commit removes the check from buildSrc
because it is not really necessary. The check exists really just for
external plugin authors since we use the gradle wrapper for our own
build.
2020-05-27 17:38:02 -07:00
Ryan Ernst beb1d0c338
Remove compiler java version flag (#57237)
This commit removes the compiler.java setting from the build. It was
originally added when Gradle was far behind support for the latest jdk,
but is no longer applicable as we don't have any need to update the
supported compile version before gradle supports the newer version. Note
that the runtime version changing support still exists here, this only
ensures we use the same jdk to compile as we use to run gradle.
2020-05-27 16:33:38 -07:00
Ryan Ernst 3ead0a183b
Upgrade bundled jdk to 14.0.1 (#57233) 2020-05-27 15:08:12 -07:00
Nhat Nguyen 5b08eaf90c
Fix trimUnsafeCommits for indices created before 6.2 (#57187)
If an upgraded node is restarted multiple times without flushing a new
index commit, then we will wrongly exclude all commits from the starting
commits. This bug is reproducible with these minimal steps: (1) create
an empty index on 6.1.4 with translog retention disabled, (2) upgrade
the cluster to 7.7.0, (3) restart the upgraded the cluster. The problem
is that with the new translog policy can trim translog without having a
new index commit, while the existing commit still refers to the previous
translog generation.

Closes #57091
2020-05-27 15:08:49 -04:00
Jake Landis baca45978c
[7.x] Remove ability to ingore files from spec validation (#56647) (#56691)
An ignore parameter was originally added to the ValidateJsonAgainstSchemaTask
to allow the build to pass for REST specs that did not properly validate
against the schema.

Since the introduction of this task, all schemas that did not validate have
been fixed to now validate properly.

This commit removes the ability to ignore specific files for validation. This
allows any consumers the assurance that all REST specs validate against the schema.
2020-05-27 10:01:14 -05:00
Mark Vieira 3a3efeb9c7
Include 'sourceUrl' column in dependencies report even when empty (#57163)
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-05-26 15:06:06 -07:00
Ryan Ernst a2864191d6
Move common repository configuration to java plugin (#57057)
This commit moves the common maven repository configuration to the ES
java plugin. All java projects need this common set of repos. Note that
the Elastic download and maven repos are removed, as they are not
necessary anymore since distribution download was split into the
DistributionDownloadPlugin.
2020-05-26 14:12:29 -07:00
Jake Landis 41629b7bfd
[7.x] Move license checks back to BuildPlugin (#56975) (#57169) 2020-05-26 15:18:12 -05:00
Rene Groeschke 28920a45f1
Improvement usage of gradle task avoidance api (#56627) (#56981)
Use gradle task avoidance api wherever it is possible as a drop in replacement in the es build
2020-05-25 09:37:33 +02:00
markharwood eb8cb31d46
Update Lucene version to 8.6.0-snapshot-9d6c738ffce (#57024)
Same version as master
2020-05-21 11:28:16 +01:00
Ryan Ernst 94062df77c
Make internalClusterTest run after unit tests (#57016)
This commit adds an ordering rule to ensure unit tests are run first
when running a higher level task like check.
2020-05-20 19:23:34 -07:00
Ryan Ernst 33ca5ba44b
Add individual precommit task plugins (#56926) (#57015)
Each precommit task is currently registered inside the shared
PrecommitTasks class. Having a single class with all precommit tasks
makes individualizing which precommit tasks are needed based on type of
project difficult, where we often just disable somet tasks eg for all qa
projects. This commit creates plugins for each precommit task, and moves
the configuration of the task into each plugin. PrecommitTasks remains
for now, and just delegates to each plugin, but will be removed in a
future change.
2020-05-20 14:12:36 -07:00
Ryan Ernst acfa2e4241
Split dependenciesInfo task to its own plugin (#56957)
The dependenciesInfo task is used to gather information about all
dependencies in elasticsearch. This commit separates it into its own
gradle plugin.
2020-05-20 10:26:05 -07:00
Ryan Ernst 381ed235e9
Re-enable plugin and upgrade bats tests (#51565) (#56999)
While the tests remained in place, the execution of these tests were
lost when creating the DistroTestPlugin. This commit restores executing
them.
2020-05-20 08:34:05 -07:00
Ioannis Kakavas 35bc386ac4
Stop disabling disgnostic trust manager in settings (#56916)
Now that #56526 is merged, we do not need to explicitly disable
the diagnostic trust manager for all of our test clusters - we do
this dynamically in runtime if the combination of java version and
JSSE provider dictates that.
2020-05-18 21:49:00 +03:00
Rene Groeschke b21330b00c Update gradle wrapper to 6.4.1 (#56881) 2020-05-18 17:14:55 +02:00
Ryan Ernst 9fb80d3827
Move publishing configuration to a separate plugin (#56727)
This is another part of the breakup of the massive BuildPlugin. This PR
moves the code for configuring publications to a separate plugin. Most
of the time these publications are jar files, but this also supports the
zip publication we have for integ tests.
2020-05-14 20:23:07 -07:00
Mark Vieira 0fd756d511
Enforce strict license distribution requirements (#56642) 2020-05-14 13:57:56 -07:00
Ignacio Vera b4521d5183
upgrade to Lucene 8.6.0 snapshot (#56661) 2020-05-13 14:25:16 +02:00
Ryan Ernst 902fc546bd
Migrate remaining ESIntegTestCases to internalClusterTest (#56479) (#56563)
This commit migrates the ESIntegTestCase tests in x-pack to the
internalClusterTest source set.
2020-05-11 21:06:04 -07:00
Rene Groeschke c29bc87040
Move bwcVersions extension property to BuildParams (back port) (#56381)
* Move bwcVersions extension property to BuildParams (#56206)
* Fix :qa Task Using Broken BwC Versions Resolution (#56332)

Co-authored-by: Armin Braun <me@obrown.io>
2020-05-11 09:39:13 +02:00
Mark Vieira 0fb9bc5379
Always use archive base name as the pom artifact id (#56447) (#56467) 2020-05-08 16:11:19 -07:00
Ryan Ernst 582145a493
Upgrade forbidden apis to 3.0 (#56368)
This commit upgrades forbidden apis to the latest version, which also
means we now get task configuration avoidance.
2020-05-07 19:05:07 -07:00