Commit Graph

2014 Commits

Author SHA1 Message Date
Nick Knize ef1cbd9e3d
Version bump for 1.1 release (#772)
Bumps the minor to 1.1 for the 1.0 release.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-06-28 09:58:27 -05:00
Nick Knize 0317b604f6
[BUG] Fix versioning issues discovered through version bump (#884)
This commit fixes up the version framework to ensure a smooth version bump
across releases.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-06-25 16:53:06 -05:00
Marc Handalian c7e60b87f9
Add URL for lucene snapshots (#858)
Signed-off-by: Marc Handalian <handalm@amazon.com>
2021-06-22 11:40:53 -05:00
Tianli Feng 110cef7882
Fix resource leak issues suggested by Amazon CodeGuru (#816)
* Address a kind of issue suggested by Amazon CodeGuru Reviewer:

* Add try-with-resources block to automatically close the resources after using to avoid resource leak, in `SymbolicLinkPreservingTarIT`, `LicenseAnalyzer`, `SymbolicLinkPreservingUntarTransform`, `ConcurrentSeqNoVersioningIT` in `VersionProperties`, `GeoFilterIT`, `XContentHelper`,  `Json` and `IndexShard` class

* Add try-finally block to close the resources after using to avoid resource leak, in `ServerChannelContext` class.

* Add try-catch block to close the resources when exception occurs in `FsBlobContainer` class (when XContentFactory.xContentType throws an exception).

* Close resources when assertion error occurs, in `ServerChannelContext` class.
2021-06-15 09:31:36 -07:00
Peter Nied 74f29c7727
Fix arm architecture translation issue (#809)
Found when attempting to build on an `arm64` machine when I recieved an error message below.  Root cause is that string equality in java cannot be done with the `==` sign.

```
unknown architecture [arm64] for jdk [provisioned_runtime], must be one of [aarch64, x64]
```

Signed-off-by: Peter Nied <petern@amazon.com>
2021-06-04 12:26:46 -04:00
Gregor Zurowski 0f9060761c
Fix Javadoc errors in `client/sniffer` (#802)
* Adds a gradle plugin to validate missing javadocs

Use `./gradlew missingJavadoc` to validate missing javadocs.
Currently this task fails because several modules are missing
appropriate javadocs. Once added, this should pass.
Also, precommit PomValidation check currently fails with missing Javadoc
plugin, that needs to be fixed -
https://github.com/opensearch-project/OpenSearch/issues/449
Thus keeping this in a separate feature branch.

Signed-off-by: Himanshu Setia <setiah@amazon.com>

* Fix Javadoc errors in client/sniffer module

Signed-off-by: Gregor Zurowski <gregor@zurowski.org>

* Add package info to client/sniffer module

Signed-off-by: Gregor Zurowski <gregor@zurowski.org>

Co-authored-by: Himanshu Setia <setiah@amazon.com>
2021-06-03 17:08:16 -07:00
Daniel Doubrovkine (dB.) bd9ca7cce5
Version checks are incorrectly returning versions < 1.0.0. (#797)
* Version checks are incorrectly returning versions < 1.0.0.

Signed-off-by: dblock <dblock@amazon.com>

* Removed V_7_10_3 which has not been released as of time of the fork.

Signed-off-by: dblock <dblock@amazon.com>

* Update check for current version to get unreleased versions.

- no unreleased version if the current version is "1.0.0"
- add unit tests for OpenSearch 1.0.0 with legacy ES versions.
- update VersionUtils to include all legacy ES versions as released.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>

Co-authored-by: Rabi Panda <adnapibar@gmail.com>
2021-06-02 16:03:50 -04:00
Rabi Panda 59563fbf72
Support running elasticsearch-oss distribution in test cluster for BWC (#764)
We need to install elasticsearch in the test cluster nodes to be able to run the BWC tests. This commit makes the following changes to enable that,
- updates DistributionDownloadPlugin to download elasticsearch-oss distributions.
- updates OpenSearchNode to be able to run both OpenSearch and Elasticsearch distributions.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-05-25 11:16:12 -07:00
Rabi Panda 9c95cf424b
[Bug] Fix gradle build on Windows failing from a recent change (#758)
A recent change as part of the commit c2e816ec introduced a bug where the build is failing on Windows. The change was made to include the NOTICE.txt file as read-only in the distributions. The code fails on Windows as it's not a POSIX-compliant. This commit adds a check on the current operating system.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-05-24 11:15:06 -07:00
Himanshu Setia 6f893ed1cd
Merging javadoc feature branch changes to main (#715)
* Adds a gradle plugin to validate missing javadocs

Use `./gradlew missingJavadoc` to validate missing javadocs.
Currently this task fails because several modules are missing
appropriate javadocs. Once added, this should pass.
Also, precommit PomValidation check currently fails with missing Javadoc
plugin, that needs to be fixed -
https://github.com/opensearch-project/OpenSearch/issues/449
Thus keeping this in a separate feature branch.

Signed-off-by: Himanshu Setia <setiah@amazon.com>

* Fix Javadoc errors in module `client/rest` (#685)

* Fix Javadoc errors in client/rest module

Signed-off-by: Gregor Zurowski <gregor@zurowski.org>

* Add package info file in client/rest module

Signed-off-by: Gregor Zurowski <gregor@zurowski.org>

* Fix typos

Signed-off-by: Gregor Zurowski <gregor@zurowski.org>

* Add exception documentation to Javadoc

Signed-off-by: Gregor Zurowski <gregor@zurowski.org>

* Fixes precommit task configuration failures due to newly added missin… (#707)

* Fixes precommit task configuration failures due to newly added missingJavadoc task

Signed-off-by: Himanshu Setia <setiah@amazon.com>

* Fixes javadoc task errors due to PR#685

Signed-off-by: Himanshu Setia <setiah@amazon.com>

* Updated CONTRIBUTING.md for info on javadocs

Signed-off-by: Himanshu Setia <setiah@amazon.com>

* Correcting licenses and naming

Signed-off-by: Himanshu Setia <setiah@amazon.com>

* Correcting version info

Signed-off-by: Himanshu Setia <setiah@amazon.com>

Co-authored-by: Gregor Zurowski <gregor@zurowski.org>
2021-05-18 13:21:41 -07:00
Rabi Panda 50abf6d066
[CVE] Upgrade dependencies to mitigate CVEs (#657)
This PR upgrade the following dependencies to fix CVEs.

- commons-codec:1.12 (->1.13) apache/commons-codec@48b6157
- ant:1.10.8 (->1.10.9) https://ant.apache.org/security.html
- jackson-databind:2.10.4 (->2.11.0) FasterXML/jackson-databind#2589
- jackson-dataformat-cbor:2.10.4 (->2.11.0) https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-28491
- apache-httpclient:4.5.10 (->4.5.13) https://bugzilla.redhat.com/show_bug.cgi?id=CVE-2020-13956
- checkstyle:8.20 (->8.29) https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-10782
- junit:4.12 (->4.13.1) https://github.com/junit-team/junit4/security/advisories/GHSA-269g-pwp5-87pp
- netty:4.1.49.Final (->4.1.59) https://github.com/netty/netty/security/advisories/GHSA-5mcr-gq6c-3hq2

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-05-18 11:37:24 -07:00
Rabi Panda c2e816ecf6
[TEST] Fix failing distro tests for linux packages (#569)
Changes to fix the failing OpenSearch distribution tests for packages (linux-archive, linux-archive-aarch64, debian, rpm, docker) on supported linux distros. 

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-05-15 21:45:48 -07:00
Daniel Doubrovkine (dB.) d9cb7daa65
Make -Dtests.output=always actually work. (#648)
Signed-off-by: dblock <dblock@amazon.com>
2021-05-04 19:45:04 -04:00
Sarat Vemulapalli 0e9f74e35f
Renaming CPU architecture to have consistent naming (#612)
* Renaming CPU architecture from x86_64, aarch64 to x64, arm64

Signed-off-by: Sarat Vemulapalli <vemulapallisarat@gmail.com>
2021-04-26 09:40:09 -07:00
Nick Knize c5a3c3cb41
Update lucene version to 8.8.2 (#557)
This commit updates the codebase to the latest released version of Lucene.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-04-23 09:48:41 -05:00
Rabi Panda 3fede8be3c
Rename the distribution used in test clusters. (#603)
For test clusters, we are using the archive(zip, tar), so we rename the distribution accordingly.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-04-22 14:21:32 -07:00
Sarat Vemulapalli 9701b226d8
Updating the Ivy repository to point to real url for Releases (#602)
Signed-off-by: Sarat Vemulapalli <vemulapallisarat@gmail.com>
2021-04-22 10:22:50 -07:00
Rabi Panda 163a51fc5a
Remove the oss string from OpenSearch distributions (#575)
This commit removes the 'oss' string which was a remnant of the predecessor distribution flavors. As OpenSearch has no flavors for distributions, we are removing this tag from all the distribution names.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-04-20 09:27:40 -07:00
Robert Muir 5474e8d094
Speed ups to test suite and precommit tasks. (#580)
Use C1 compiler only for short-lived tasks and unit test execution. Tone
down some of the slowest unit tests.

Signed-off-by: Robert Muir <rmuir@apache.org>
2021-04-20 09:02:45 -05:00
Nick Knize 0ba0e7cc26
[Versioning] Rebase to OpenSearch version 1.0.0 (#555)
This commit rebases the versioning to OpenSearch 1.0.0

Co-authored-by: Rabi Panda <adnapibar@gmail.com>

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-04-15 17:06:47 -05:00
Robert Muir 6aa3675e9d
Prevent setting maxParallelForks=0 on single-cpu machines (#558)
The maxParallelForks is currently numJavaCpus / 2, but this evaluates to
zero (illegal value) if numJavaCpus is 1 and breaks the build.

Remove large chunk of complex dead code above this calculation, as it is
always overwritten by the simple calculation anyway.

Closes #554

Signed-off-by: Robert Muir <rmuir@apache.org>
2021-04-14 19:17:58 -07:00
Nick Knize ee6d15e26a
[License] Add SPDX License Header to security policies (#531)
This commit adds the SPDX license header and modifications copyright to security
policy files.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-04-12 22:59:36 -05:00
Nick Knize 9168f1fb43
[License] Add SPDX and OpenSearch Modification license header (#509)
This commit adds the SPDX Apache-2.0 license header along with an additional
copyright header for all modifications.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-04-09 14:28:18 -05:00
Nick Knize 8f4b2d58d1
[License] Update SPDX License Header (#510)
This commit updates the SPDX License Header for all new files created by
OpenSearch contributors.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-04-09 13:49:56 -05:00
Rabi Panda 2dca3462f2
Fix stragglers from renaming to OpenSearch work. (#483)
This commit fixes more instances where we missed renaming to OpenSearch.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-04-05 11:51:20 -07:00
Rabi Panda 247772033a
Update remote repo for BWC checks. (#482)
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-04-02 13:05:47 -07:00
Rabi Panda 70cad29af3
Perform more renaming to OpenSearch. (#470)
This commit performs more renaming to OpenSearch which were missed in the earlier commits.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-03-30 11:05:50 -07:00
Rabi Panda f7a7b80fba
Fix build-tools integ test failures. (#465)
A few build-tools integTests were failing as a result of the renaming to OpenSearch. This was due to some un-renamed files in the fake archives used for testing. This commit renames those.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-03-26 12:30:13 -07:00
Rabi Panda 3460a8c213
Fix a few more renaming issues. (#464)
This commit fixes some more missed instances where we can perform the renaming to OpenSearch.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-03-26 12:05:16 -07:00
Himanshu Setia f9b0b9fe6d
[GradleCheck] Fixes failing tests for ExportOpenSearchBuildResourcesTaskIT (#442)
Signed-off-by: Himanshu Setia <setiah@amazon.com>
2021-03-23 12:23:11 -07:00
Nick Knize 49907793f3
[Rename] Change license header and copyright notice to SPDX (#437)
This commit changes the oss-license-header.txt file from the legacy ASLv2 header
to the new SPDX format. It also updates the ide gradle file and
LicenseHeaderTask of precommit to accept the new SPDX license format.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-03-22 12:36:34 -05:00
Himanshu Setia f27eb8a98c
Remove lingering instances of Default distribution in favour of Oss (#440)
Signed-off-by: Himanshu Setia <setiah@amazon.com>
2021-03-22 10:32:57 -07:00
Rabi Panda 8bba6603da [Rename] Replace more instances of Elasticsearch with OpenSearch. (#432)
This commit replaces more replaceable instances of Elasticsearch with OpenSearch.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-03-21 20:56:34 -05:00
Nicholas Walter Knize 5ecb607c9d [Rename] OpenSearch in build-tools
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-03-21 20:56:34 -05:00
Nick Knize 90fef184db Update NOTICE.TXT with OpenSearch copyright (#232)
This commit updates the NOTICE.txt file to include the OpenSearch copyright
notice.

Signed-off-by: Nicholas Knize <nknize@amazon.com>
Signed-off-by: Peter Nied <petern@amazon.com>
2021-03-21 20:56:34 -05:00
Nick Knize 7051167c83 [Rename] remaining elasticsearch pass 1 (#416)
This commit refactors instances of 'elasticsearch' with opensearch everywhere
except references to issues, and other places needed to test compatibility with
old elasticsearch clusters.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-03-21 20:56:34 -05:00
Rabi Panda eddfe6760d [Rename] Fix issues for gradle precommit task. (#418)
Fix miscellaneous issues identified during `gradle precommit`. These issues are the side effects of the renaming to OpenSearch work.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-03-21 20:56:34 -05:00
Rabi Panda 127243b83b [Rename] Environment variables in entire repository. (#405)
Rename all environment variables with prefix 'ES_' to 'OPENSEARCH_'.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-03-21 20:56:34 -05:00
Rabi Panda df11cc9de4 [Rename] Fix gradle build as part of the renaming process. (#397)
This commit fixes the currently broken gradle build resulted from the renaming work. It reverts a few dependencies and comments out the `opensearch_distibutions` task which is currently failing for some builds. We will address these separately in the future once we have a working build.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-03-21 20:56:34 -05:00
Rabi Panda 13f6d23e40 [Rename] Property and metadata keys with prefix es. (#389)
Rename all property and metadata keys with prefix 'es.' to 'opensearch.'.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-03-21 20:56:34 -05:00
Nick Knize 5b46a05702 [Rename] remaining packages and resources in test/fixture (#364)
This commit refactors the remaining o.e.index and o.e.test packages in the
test/fixtures module. References throughout the codebase are also refactored.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-03-21 20:56:34 -05:00
Nick Knize 8a6b294e2e Update NOTICE.TXT with OpenSearch copyright (#232)
This commit updates the NOTICE.txt file to include the OpenSearch copyright
notice.

Signed-off-by: Nicholas Knize <nknize@amazon.com>
2021-03-21 20:56:34 -05:00
Rabi Panda ad8ecf08dc [Rename] buildSrc directory, build-tools module (#182)
This PR refactors the build-tools module as part of the Elasticsearch to OpenSearch renaming effort.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-03-21 20:56:34 -05:00
Himanshu Setia 64caa368d6
Fixes process bootstrap failure in IntegTests (#425)
Signed-off-by: Himanshu Setia <setiah@amazon.com>
2021-03-19 18:25:10 -07:00
Nick Knize 807d0e3ba0 Update NOTICE.TXT with OpenSearch copyright (#232)
This commit updates the NOTICE.txt file to include the OpenSearch copyright
notice.

Signed-off-by: Nicholas Knize <nknize@amazon.com>
Signed-off-by: Peter Nied <petern@amazon.com>
2021-03-13 10:36:16 -06:00
Nick Knize 83e87f7e54 Remove any non oss from build, package, and distribution (#102)
This commit changes the building, packaging, and testing framework to only support OSS on different distributions.

Next steps:

completely remove -oss flag dependencies in package and build tests
move 6.x bwc testing to be an explicit option
remove any references to elastic.co download site (or replace with downloads from the OSS website)

Co-authored-by: Himanshu Setia <setiah@amazon.com>
Co-authored-by: Rabi Panda <pandarab@amazon.com>
Co-authored-by: Himanshu Setia <58999915+setiah@users.noreply.github.com>
Co-authored-by: Sarat Vemulapalli <vemsarat@amazon.com>
Signed-off-by: Peter Nied <petern@amazon.com>
2021-03-13 10:36:14 -06:00
Rabi Panda 3b1f107d62 Run precommit and unit tests as part of github actions. (#84)
Signed-off-by: Peter Nied <petern@amazon.com>
2021-03-13 10:36:14 -06:00
Nick Knize 755367f1c9 [PURIFY] Remove x-pack feature flag from yaml test (#68)
This commit removes the xpack no_xpack feature flag from the yaml test suite.

Signed-off-by: Peter Nied <petern@amazon.com>
2021-03-13 10:36:13 -06:00
Rabi Panda eb49365800 Remove x-pack from build, distribution and packaging. (#43)
This PR removes references to x-pack from buildSrc, distribution and qa modules.

Signed-off-by: Peter Nied <petern@amazon.com>
2021-03-13 10:36:10 -06:00
Nick Knize 6f2cbdf0b8 [PURIFY] Cleanup build and unblock HLRC integration tests (#33)
This commit cleans up the following:

* Remove unused imports
* Remove ILM settings in hlrc testCluster formation
* Comment out security users settings in ElasticsearchNode creation for build-tools tests

Signed-off-by: Peter Nied <petern@amazon.com>
2021-03-13 10:36:10 -06:00
Nick Knize 70a07216bb [PURIFY] remove all trace of x-pack watcher (#21)
This commit removes all trace of Elastic licensed watcher

Signed-off-by: Peter Nied <petern@amazon.com>
2021-03-13 10:36:08 -06:00
Nick Knize 4aee8e56db [PURIFY] remove all trace of x-pack index lifecycle management (#20)
This commit removes all trace of Elastic licensed ILM.

Signed-off-by: Peter Nied <petern@amazon.com>
2021-03-13 10:36:08 -06:00
Nick Knize 46f37bfa81 [PURIFY] remove all trace of x-pack migration (#19)
This commit removes all trace of Elastic licensed Migration

Signed-off-by: Peter Nied <petern@amazon.com>
2021-03-13 10:36:08 -06:00
Nick Knize f15b830261 [PURIFY] remove all trace of x-pack sql (#18)
This commit removes all trace of Elastic licensed SQL

Signed-off-by: Peter Nied <petern@amazon.com>
2021-03-13 10:36:08 -06:00
Nick Knize 3769b2c6a4 [PURIFY] remove all trace of x-pack rollups (#17)
This commit removes all trace of Elastic licensed rollups

Signed-off-by: Peter Nied <petern@amazon.com>
2021-03-13 10:36:08 -06:00
Nick Knize fb7ff93c42 [PURIFY] remove all trace of x-pack security (#16)
This commit removes all trace of the security high level rest client and other reference to x-pack security

Co-authored-by: Rabi Panda <rabipanda@icloud.com>
Signed-off-by: Peter Nied <petern@amazon.com>
2021-03-13 10:36:08 -06:00
Nick Knize b5cd4c417c [PURIFY] remove all trace of x-pack ml (#14)
This commit removes all trace of x-pack ml.

Signed-off-by: Peter Nied <petern@amazon.com>
2021-03-13 10:36:07 -06:00
Nick Knize 1cdbc63917 [PURIFY] remove all trace of x-pack graph (#13)
This commit removes all trace of Elastic licensed graph feature

Signed-off-by: Peter Nied <petern@amazon.com>
2021-03-13 10:36:07 -06:00
Nick Knize 125958eb2c Remove the Elastic license file, all checks for this license and the license REST APIs. (#12)
Co-authored-by: Rabi Panda <rabipanda@icloud.com>
Signed-off-by: Peter Nied <petern@amazon.com>
2021-03-13 10:36:07 -06:00
Nick Knize 3a52e9ddc1 [PURIFY] update build.gradle files to ensure build completes; gradle check fails (#7)
Signed-off-by: Peter Nied <petern@amazon.com>
2021-03-13 10:36:06 -06:00
Nick Knize 168f5e825f [PURIFY] remove all trace of x-pack eql (#5)
This commit removes all trace of EQL from the sanitized fork.

Signed-off-by: Peter Nied <petern@amazon.com>
2021-03-13 10:36:06 -06:00
Rene Groeschke f02bd9942b
Version Bumb to 7.10.3 2021-01-14 15:43:41 +01:00
Francisco Fernández Castaño 55246d8d9b [7.10] Bump version after 7.10.1 release 2020-12-09 16:11:29 +01:00
Rory Hunter 410fac5970 Explicitly retry failed docker pulls (#63946)
Closes #63869. Perform `docker pull` explicitly instead of as part of
`docker build`, and wrap it in a retry loop. This is an attempt to make
the build more resilient to transient errors.
2020-12-09 14:30:45 +00:00
Mark Vieira f8f5d27f6b Add option to preserve data in test clusters (#65400)
(cherry picked from commit 1ce323e1368cf5231181f1efaba1c4e425066e37)
2020-11-24 11:56:56 -08:00
Andrei Dan cd35122e48
Bump versions after 7.10 release (#64856) 2020-11-11 13:08:16 +00:00
Ryan Ernst 6be4acd509
Use no-kpi subdomains for tests downloading artifacts (#64502)
This commit converts build code that downloads distributions or other
artifacts to use the new no-kpi subdomain, and removes the formerly used
no-kpi header.
2020-11-07 11:30:52 -08:00
Ignacio Vera 4851bc7bae
Upgrade to Lucene-8.7.0 (#64532) (#64537) 2020-11-03 16:57:04 +01:00
Ryan Ernst b1ba274157
Upgrade bundled jdk to 15.0.1 and switch back to Oracle (#64253)
This commit updates the bundled jdk to 15.0.1, and at the same time once
again switches the bundled jdk back to adoptopenjdk, which has fixed
their build problem and regained support for glibc 2.12.

closes #64026
2020-10-29 10:57:20 -07:00
Mark Vieira 13fb6dbe4b Fix BWC distribution resolution errors (#64310)
(cherry picked from commit e6ffe9e7b73f7f4dd28ec1059bb9651a98d8f5f5)
2020-10-28 12:50:56 -07:00
Ignacio Vera d0f5066310
Upgrade to lucene-8.7.0-snapshot-72d8528c3a6 (#63912) (#63928) (#63933) 2020-10-20 15:08:06 +02:00
Rene Groeschke b5448f07f3
Fix resolveAllDependencies by removing refs to non existing configs (#63409) (#63594) (#63715)
* Fix concurrent modification on task realization
* Use taskprovider instead of relying on tasks in distribution setup
* Port more task references in :distribution to task provider
* Fix nullpointer in distribution setup
2020-10-20 12:43:17 +02:00
Rene Groeschke 26f58c8bdb
Fix build tools git tests on windows 7x (#63456)
* Add error logging for git commands in build-tools integTests
* Explicitly set user and email in git related func tests
2020-10-13 14:12:52 +02:00
Mayya Sharipova e022b78198
Upgrade to lucene-8.7.0-snapshot-5c4168d (#63466)
This disables sort optim on _doc, which may still be unstable.
Backport for #63444
2020-10-08 08:20:43 -04:00
Mayya Sharipova e236ea43e9 Upgrade to lucene-8.7.0-snapshot-e914862 (#63401)
Backport for: #63395
2020-10-07 09:45:14 -04:00
Mayya Sharipova f2ba62b894
Upgrade to lucene- 8.7.0-snapshot-66c49a35402 (#63372)
This includes fixing a bug in doc iteration during sort optimization

Backport for #63349
2020-10-06 22:38:58 -04:00
Ryan Ernst dd5ec83c99
Remove ml cpp notice check from oss distributions (#63370)
The ML cpp notice only exists with default distributions, but the check
task exists on all archive distributions. This commit avoids creating
the task for distributions that don't have ML.

closes #63128
2020-10-06 18:05:25 -07:00
Rene Groeschke 1d0f46cc35
Fix openjdk download on mac (#63309)
- the wrong platform classifier (mac) was used when downloading
openjdk on mac. This fixes it to use osx

closes #63353
2020-10-06 15:04:15 -07:00
Rene Groeschke 09f7cff612
Fix syncing expanded distributions in gradle build (#63285) (#63328) 2020-10-06 16:51:12 +02:00
Rene Groeschke 8144106ace
Build local unreleased bwc versions more efficient for tests (7.x backport) (#63188)
* Wire local unreleased bwc versions more efficient for tests (#62473)

For testing against the local distribution we already avoid the packaging/unpackaging 
cycle of es distributions when setting up test clusters. This PR adopts the usage of the
expanded created distributions for unreleased bwc versions (versions that are checkout 
from a branch and build from source in the :distribution:bwc:minor / :distribution:bwc:bugfix). 
This makes the setup of bwc based cross version tests a bit faster by avoiding 
the unpackaging overhead. We still assemble both in the bwcBuild tasks atm 
which will be addressed in a later issue.

This reworks the :distribution:bwc project:

- Convert all the custom logic from build script logic (groovy) into gradle binary plugins (java)
- Tried to make the bwc setup logic a bit more readable
- Add basic functional test coverage for the bwc logic this PR tweaked.
- Extracted a general internal BWC Git plugin out of the bwc setup plugin to improve maintenance 
and testability
- Changed the InternalDistributionPlugin to resolve the extracted distro instead on relying 
on unpacking the distribution archive

* Fix java8 incompatibility
* Fix extension calculation for 6.8.* distribution
2020-10-06 15:58:13 +02:00
Ryan Ernst 25f8a3ba42
Switch bundled jdk back to Oracle JDK (#63288) (#63290)
We switched to adoptopenjdk from oracle jdk to rely on the notarization
found in adoptopnejdk on MacOS. However, that notarization still had
issues, and we currently do our own notarization of the entire
distribution, including the jdk. The recent bump to jdk 15 has revealed
openjdk to be lax in maintaining support for older systems. Since the
notarization is no longer an issue, this PR moves the bundled jdk back
to Oracle, in order to continue supporting those older systems affected
by adoptopenjdk 15.

relates #62709
2020-10-05 16:31:10 -07:00
Nhat Nguyen 1a6837883a Upgrade to Lucene-8.7.0-snapshot-77396dbf339 (#63222)
Includes LUCENE-9554, which exposes the pendingNumDocs from IndexWriter.
2020-10-05 14:39:30 -04:00
Rene Groeschke f58ebe58ee
Use services for archive and file operations in tasks (#62968) (#63201)
Referencing a project instance during task execution is discouraged by
Gradle and should be avoided. E.g. It is incompatible with Gradles
incubating configuration cache. Instead there are services available to handle
archive and filesystem operations in task actions.

Brings us one step closer to #57918
2020-10-05 15:52:15 +02:00
Jake Landis d864d14832
[7.x] Introduce changes related to yamlRestCompatTest (#62985) (#63105)
Backport for #62985 that includes the related changes, but 
not the actual plugin for yamlRestCompatTest. The plugin 
is not necessary in 7.x, and back porting relevant changes to
help keep 7.x code inline with master.
2020-10-01 15:29:57 -05:00
Ioannis Kakavas 2ea3073a5e
Use BCFIPS 1.0.2 in our CI (#63116)
Bouncy Castle's BC-FJA-1.0.2 has been certified for a while now
but we had noticed that it seems to be rather entropy hungry and
ES would start very slowly ( and tests would take forever )
because of blocking calls to /dev/random.

We verified that this is resolved when enabling hw RNG or a
software one like haveged. While rng-tools should be suggested for
production uses, our ephemeral workers have haveged installed
which should work just fine for CI.

Backport of 63099
2020-10-01 14:48:53 +03:00
Mayya Sharipova 4c8c3c8df6
Upgrade lucene to lucene-8.7.0-snapshot-3b59906 (#62978)
Backport for #62970
2020-09-28 16:52:31 -04:00
Nik Everett c3e07da437
Docs: Support """ in console-response (#62876) (#62962)
We support `"""` in `console` snippets to emulate kibana's CONSOLE.
CONSOLE also spits out `"""` when a json field contains a new line or a
double quote. This adds support for those sorts of responses to the
handling of `console-response` snippets.
2020-09-28 11:51:07 -04:00
Ryan Ernst 7e9b957da5
Handle JAVA_HOME better in packaging tests (#62905)
JAVA_HOME is set as necessary in packaging tests, depending on whether
it is needed for no-jdk distributions or testing override behavior. We
currently rely on gradle finding java through PATH. However, JAVA_HOME
can sometimes be set by the system itself, which then leaks through to
the packaging test. This commit reworks our handling of JAVA_HOME to
pass it through for gradle, and then explicitly clear it whenever
running shell commands in packaging tests.
2020-09-24 17:01:29 -07:00
Ryan Ernst 1c26926dea
Avoid using bundled jdk on unsupported platforms (#62793)
We use the bundled jdk for unit, integ and packaging tests. Since
upgrading to jdk 15, centos-6 and oracle enterprise linux 6 have failed
due to versions of glibc no longer supported by the jdk. This commit
adds detection of the old glibc versions to gradle, and utilizes that
when deciding which jdk to use for tests.

relates #62709
closes #62635
2020-09-23 16:55:47 -07:00
Rory Hunter df2b5dd4d1 Use the Elastic Docker registry for the UBI base image (#62606)
Swap the base Docker image for UBI builds to point at Elastic's registry.
2020-09-22 12:09:07 +01:00
Ryan Ernst ee835ee74a
Copy gradle/system jdk to local dir for packaging tests (#61436)
The distro tests rely on two jdks, pulled in by the jdk download plugin.
The move the artifact transforms result in the path to the extracted
jdks existing under the gradle cache dir, which is outside the vagrant
mount of the elasticsearch project. This commit creates a local copy
within the `qa:os` project that the packaging tests use.

closes #61138
2020-09-21 12:19:09 -07:00
Jake Landis 392f99d695
[7.x] Make yamlRestTest and javaRestTest tasks lazy (#62515) (#62632)
This commit changes the yamlRestTest and javaRestTest tasks to be lazily created.
This change requires pro-actively creating the testClusters container so that the
configuration can be applied without any changes to the build.gradle files.

related: #60261
related: #47804
2020-09-18 12:49:37 -05:00
Adrien Grand 4de8579455
Upgrade to lucene-8.7.0-snapshot-830bd186a8d. (#62596) 2020-09-18 09:51:34 +02:00
Jake Landis 5b7246157f
[7.x] Fix projects that failed to build within Intellij (#62258) (#62408)
This commit address some build failures from the perspective of Intellij.
These changes include:
* changing an order of a dependency definition that seems to can cause Intellij build to fail.
* introduction of an abstract class out of the test source set (seems to be an issue sharing 
  classes cross projects with non-standard source sets. 
* a couple of missing dependency definitions (not sure how the command line worked prior to this)
2020-09-17 17:45:12 -05:00
Ryan Ernst 1b0e6caae6
Upgrade the bundled JDK to JDK 15 (#62580)
This commit upgrades the bundled JDK to JDK 15.
2020-09-17 11:55:35 -07:00
Adrien Grand 9a8225bbc1
Upgrade to lucene-8.7.0-snapshot-9cd3af50f80. (#62450) (#62476)
This new snapshot contains the following JIRAs that we're interested in:
 - [LUCENE-9525](https://issues.apache.org/jira/browse/LUCENE-9525)
Better handling of small documents. This should improve retrieval times
when documents are less than ~1kB.
 - [LUCENE-9510](https://issues.apache.org/jira/browse/LUCENE-9510)
Faster flushes when index sorting is enabled by not compressing the
temporary files that store stored fields and term vectors.
2020-09-17 10:28:20 +02:00
Adrien Grand 6db8afefc2
Upgrade to lucene-8.7.0-snapshot-cdfdc1e0851. (#62376)
Upgrade to a new Lucene snapshot that (at least partially) addresses the
indexing rate regression when index sorting is enabled.

Backport of #62334.
2020-09-15 17:48:07 +02:00
Ignacio Vera c8981ea93d
upgrade to lucene-8.7.0-snapshot-b313618cc1d (#62213) (#62222) 2020-09-10 16:23:18 +02:00
Rene Groeschke 038f7a83df
Avoid packaging / unpacking cycle when using local current distributions (#61592) (#62176)
- Extract distribution archives defaults into plugin
- Added basic test coverage
- Avoid packaging/unpackaging cycle when relying on locally build distributions
- Provide DSL for setting up distribution archives
- Cleanup archives build script
2020-09-10 08:08:50 +02:00
Jake Landis d8dad9ab2c
[7.x] Remove integTest task from PluginBuildPlugin (#61879) (#62135)
This commit removes `integTest` task from all es-plugins.  
Most relevant projects have been converted to use yamlRestTest, javaRestTest, 
or internalClusterTest in prior PRs. 

A few projects needed to be adjusted to allow complete removal of this task
* x-pack/plugin - converted to use yamlRestTest and javaRestTest 
* plugins/repository-hdfs - kept the integTest task, but use `rest-test` plugin to define the task
* qa/die-with-dignity - convert to javaRestTest
* x-pack/qa/security-example-spi-extension - convert to javaRestTest
* multiple projects - remove the integTest.enabled = false (yay!)

related: #61802
related: #60630
related: #59444
related: #59089
related: #56841
related: #59939
related: #55896
2020-09-09 14:25:41 -05:00
Ryan Ernst 6d3b691048
Add snapshot only test modules (#61954)
This commit adds external test modules. These are modules meant for
external systems to test edge cases in elasticsearch, but only within
snapshots. They are not meant to be used in production, so protections
are also added from their accidental inclusion in release builds.

Note that this commit does not actually add any new modules, it only
adds the infrastructure for the new modules, under
`test/external-modules`.
2020-09-04 16:35:18 -07:00
Rene Groeschke 0b6d187932
Fix resolveAllDependencies broken by ArtfactTransforms (#61972) (#61979)
- ignore es extracted configuration for resolveAllDeps
- fixes #61945
2020-09-04 20:21:46 +02:00
Ioannis Kakavas 7b021bf3fb
Run zulu8 fips CI with BCJSSE instead of SunJSSE (#61857)
As we figured out in
https://github.com/elastic/elasticsearch/issues/61316#issuecomment-685482708
Azul brings back a lot of changes from JDK 11 to their Zulu8 build
and this means that we can't run this with SunJSSE in FIPS 140 mode.

This change ensures that we configure Zulu8 JDK JVMs in FIPS 140
mode, using the bouncy castle JSSE FIPS provider, instead of the
SunJSSE one ( as we do for the rest of the java 8 JVMs )

Resolves: #61316
2020-09-04 14:53:43 +03:00
Ignacio Vera 31c026f25c
upgrade to Lucene-8.7.0-snapshot-61ea26a (#61957) (#61974) 2020-09-04 13:46:20 +02:00
Ioannis Kakavas 6d250e0f44
Add runtimeJavaDetails property in BuildParams (#61901) (#61961)
Relates to #61857
2020-09-04 11:47:44 +03:00
Ryan Ernst d6e17170c3
Simplify adding plugins and modules to testclusters (#61886)
There are currently half a dozen ways to add plugins and modules for
test clusters to use. All of them require the calling project to peek
into the plugin or module they want to use to grab its bundlePlugin
task, and then both depend on that task, as well as extract the archive
path the task will produce. This creates cross project dependencies that
are difficult to detect, and if the dependent plugin/module has not yet
been configured, the build will fail because the task does not yet
exist.

This commit makes the plugin and module methods for testclusters
symmetetric, and simply adding a file provider directly, or a project
path that will produce the plugin/module zip. Internally this new
variant uses normal configuration/dependencies across projects to get
the zip artifact. It also has the added benefit of no longer needing the
caller to add to the test task a dependsOn for bundlePlugin task.
2020-09-03 19:37:46 -07:00
Rene Groeschke 9d4e1f1589
Fix duplicate dir handling in untar transform (#61917) (#61933) 2020-09-03 18:52:48 +02:00
Rene Groeschke 41613c16e9
Rework output normalization in build-tools func tests (#61706) (#61721) 2020-09-01 21:00:27 +02:00
Rene Groeschke 77ecea6e3b
Fix jvm options expansion on windows (#61743) (#61789)
* Fix jvm options expansion on windows
* Avoid long paths on windows
* Relativize paths in jvmoptions against working dir
2020-09-01 19:49:19 +02:00
Rene Groeschke 4aaeb1e4e3
Address review feedback
- avoid ! when negating boolean expressions
2020-08-31 17:55:16 +02:00
Jason Tedor 64cd229b35
Upgrade to Lucene 8.6.2 (#61688)
This commit upgrades the Lucene dependencies to 8.6.2.
2020-08-31 09:54:07 -04:00
Rene Groeschke f08981c4da
Simplify distribution download configurations (7.x backport) (#61708)
We leverage artifact transforms now when downloading and unpacking elasticsearch distributions.

This has the benefit of

- handcrafted extract tasks on the root project are not required.
- The general tight coupling to the root project has been removed.
- The overall required configurations required to handle a distribution have been reduced
- ElasticsearchDistribution has been simplified by making Extracted an ordinary Configuration
downloaded and unpacked external distributions are reused in later builds by been cached
in the gradle user home.

DistributionDownloadPlugin functional tests have been extended and ported
to DistributionDownloadPluginFuncTest.

* Fix ElasticsearchNode#getDistributionFiles (#61219)

Fixes #61647
2020-08-31 14:36:42 +02:00
Rene Groeschke d1c196a1b2
Minor cleanup 2020-08-31 11:11:32 +02:00
Rene Groeschke 0d488cfbb9
Fix bwc testcluster setup against elastic < 6.3 2020-08-31 11:07:54 +02:00
Rene Groeschke 4540610b62
Fix jvm options handling in testclusters < 6.2 2020-08-27 16:03:57 +02:00
Rene Groeschke c4773a4949
Remove debug println from build logic 2020-08-27 15:47:14 +02:00
Rene Groeschke fa7f1cc979
Update gradle wrapper to 6.6.1 (#61571) (#61579) 2020-08-27 14:16:53 +02:00
Rene Groeschke ae90ae343a
Fixing jvm options for testcluster for older versions (#61620)
* Fixing jvm options for testcluster for older versions
2020-08-27 14:06:14 +02:00
Rene Groeschke c4a0c603f0
Fix normalizing build output for windows (#61622) (#61625)
- handle windows paths better
Fixes #61612
2020-08-27 14:05:22 +02:00
Ryan Ernst e60c74240a
Add base precommit task to all java projects (#61439)
This commit adds java compilation to the base precommit task, and adds
that to the java plugin. This further reduces dependence on the build
plugin.
2020-08-26 17:21:00 -07:00
Rene Groeschke 3a8cfdc1f5
Extract distribution archive checks into plugin (7.x backport) (#61567)
- Added test coverage
- Removes build script cluttering
- Splits archive building and archive checking logic
- only rely on boost for now for ML licenses(tbd)
- Use Gradle build-in untar and unzip support

* Handle dynamic versions in func tests assertions
2020-08-26 15:04:12 +02:00
Rene Groeschke fac66a7528
Rework test cluster distribution handling (#61407) (#61566)
Driven by this issue 
https://github.com/elastic/elasticsearch/pull/60969#issuecomment-674962158
we apply some rework on how we handle distributions in our test cluster setups:

- If no custom modules, plugins or extra jar files are declared we do not create a cluster
specific distro folder and use the origin distribution folder instead.
- If a custom distribution folder is required, we fallback to file copy when hard linking
is not supported
2020-08-26 15:03:52 +02:00
Ryan Ernst d1031fd928
Move versions convenience property to java plugin (#61438)
Access the common versions map is done in a lot of places. While it can
be access through an import of VersionProperties, the vast majority of
places use it through the provided convenience property added by
BuildPlugin. This commit moves that convenience property to the base
java plugin, so further reduce dependence on the BuildPlugin.
2020-08-21 16:43:02 -07:00
Rory Hunter 890f943d0f
Fix tests that weren't expecting UBI Docker images (#61361)
Closes #61360.
2020-08-20 13:30:21 +01:00
Rory Hunter d8aacbdd31 Add UBI docker builds
Backport of #60742.

This PR resurrects support for building Docker images based on one of
Red Hat's UBI images. It also adds support for running the existing
Docker tests against the image. The image is named
`elasticsearch-ubi8:<version>`.

I also changed the Docker build file uses enums instead strings in a lot
of places, for added rigour.
2020-08-18 09:48:51 +01:00
Mark Vieira d26a3e144e
Revert "Simplify distribution download and extraction (7.x backport) (#61184)"
This reverts commit 8b7a0a1f64.
2020-08-17 08:54:08 -07:00
Rene Groeschke 8b7a0a1f64
Simplify distribution download and extraction (7.x backport) (#61184)
We leverage artifact transforms now when downloading and unpacking elasticsearch distributions.

This has the benefit of

- handcrafted extract tasks on the root project are not required. 
- the general tight coupling to the root project has been removed.
- the overall required configurations required to handle a distribution have been reduced
- ElasticsearchDistribution has been simplified by making Extracted an ordinary Configuration
downloaded and unpacked external distributions are reused in later builds by been cached
in the gradle user home.

DistributionDownloadPlugin functional tests have been extended and ported
to DistributionDownloadPluginFuncTest.

* Fix java8 compliant Path calculation
2020-08-17 10:10:32 +02:00
Rene Groeschke 9ca052d702
Use native Gradle support for --release flag (#61093) (#61151)
With Gradle 6.6 we can now use the native support for the --release compile options. 
As Gradle by default resolves the compatibility version from the release property we needed to workaround this in order to keep our current setup. An issue was raised at gradle to track this at https://github.com/gradle/gradle/issues/14141
2020-08-14 20:01:24 +02:00
Jake Landis cb9f4cdae2
Fix the REST FIPS tests (#61001)
Adds bouncycastle to classpath for tests and testclusters
2020-08-13 16:23:54 -07:00
Ryan Ernst ef796e589c
Fix typo in Taskkill in reaper (#61076)
The reaper is used only when spawned processes by the build like
testclusters or vagrant are left behind in a failed build. When this
occurs on Windows, the Taskkill command is used. However, there was a
typo in the command name. Since the reaper is for edgecases and mostly
protection for local builds (CI uses immutable hosts), this has been
broken since its inception.

closes #60503
2020-08-13 10:21:13 -07:00
Rene Groeschke a5ef38ca40
Update gradle wrapper to 6.6 (#59909) (#60949) 2020-08-11 11:03:19 +02:00
Rene Groeschke e03993d238
Make jdk download repo not consumable (#60875)
- fixes #60860
2020-08-10 14:22:42 +02:00
Ryan Ernst cf39a69be3
Exit gradle run task if any node dies (#60789)
The Elasticsearch `run` task in Gradle is backed by testclusters
spinning up a node and waiting indefinitely on output from that node.
However, this thread currently waits for an interrupt to exit. If
Elasticsearch fails to start, we hang indefinitely. This commit adds a
check after each poll of the process stdout to see if it is still alive,
and fails the build if it has died.

closes #60761
2020-08-06 15:54:43 -07:00
Ryan Ernst 479f32906d
Cleanup vagrant setup (#60697)
Though bats tests were recently removed, there remains a few unnecessary
setups needed for those tests in our vagrant files, as well as CI setup.
Additionally, we no longer rely on the vagrant images setting a
JAVA_HOME, instead relying on DistroTestPlugin to pull the appropriate
jdk when testing with no-jdk distributions.
2020-08-06 15:37:56 -07:00
Rene Groeschke 91e956e96a
Simplify jdk downloads via artifact transforms (#60713) (#60803)
* Simplify jdk downloads via artifact transforms

This reworks the jdk download plugin to use artifact transforms.
That allows us to simplify the jdk download plugin by a lot. The
benefits of using artifact transforms are:

- they transform an artifact to an upacked folder on the fly as part
of the dependenc resolution allowing us to remove all the custom created unpack tasks
and configurations

- Artifact transforms support gradle build cache. Requesting a jdk folder on a clean machine
will likely be resolved from the build cache

- The manual mingling about not extracting jdks multiple times by all jdks channeling through
root project configurations can be removed as they support up-to-date checking and build cache
which will ensure these archives are only resolved once per build at max.

Also the test coverage has been ported to Spock that supports data driven testing. This porting
includes an introduction to a wiremock fixture that can be used later on for mocking repository
urls in other integration tests

* Simplify artifact transform registration

* Change jdk finalization and repository setup

Jdk finalization is now done via `configuration.defaultDependencies`.
This has two benefits:
- we only configure and finalize the JDKs we actually use during the build
- afterEvaluate hooks are problematic in combination with task avoidance api.
Relying on task avoidance api in Gradle moves more configuration logic into the materialization
of tasks which is, with task avoidance api, is done ideally during the task graph calculation.
Anything created (e.g. jdks) created late in this task graph calculation would never be finalized
via afterEvaluate hooks as this has been fired before already with the jdk not there yet.

Furthermore we now only configure repositories in the projects where the jdk is declared
(aka the jdk-download plugin is applied) and only if the jdk is actually requested during the build.

* Fix javadoc

* Fix jdk download repo content filtering

* Minor cleanup

* Apply review feedback and cleanup
2020-08-06 12:34:20 +02:00
Ryan Ernst 7514526b6b
Convert packaging upgrade tests to java (#60560) (#60680)
This commit removes the last of the bats tests, converting the rpm/deb
upgrade tests to java. It adds a new pattern of tasks, similar in nature
but separate from the existing distro tests, named `distroUpgradeTest`.
For each index compatible version, a `distroUpgradeTest.VERSION` task
exxists. Each distribution then has a task, named
`distroUpgradeTest.VERSION.DISTRO`.

One thing to note is these new tests do not cover no-jdk versions of
the rpm/deb packages, since the distribution/bwc project does not
currently build those.

closes #59145
closes #46005
2020-08-05 14:05:37 -07:00
Jake Landis 5db283262e
[7.x] Apply new REST base plugin for the javaRestTest plugin (#60627) (#60672)
This commit applies the new base REST plugin for javaRestTest plugin.

related: #60624
2020-08-04 13:14:08 -05:00
Rene Groeschke bdd7347bbf
Merge test runner task into RestIntegTest (7.x backport) (#60600)
* Merge test runner task into RestIntegTest (#60261)
* Merge test runner task into RestIntegTest
* Reorganizing Standalone runner and RestIntegTest task
* Rework general test task configuration and extension
* Fix merge issues
* use former 7.x common test configuration
2020-08-04 14:46:32 +02:00
Yannick Welsch b0d601fa63 Adjust searchable snapshot license (#60578)
No longer needs Platinum license for testing on staging.
2020-08-03 13:19:53 +02:00
Rene Groeschke ed4b70190b
Replace immediate task creations by using task avoidance api (#60071) (#60504)
- Replace immediate task creations by using task avoidance api
- One step closer to #56610
- Still many tasks are created during configuration phase. Tackled in separate steps
2020-07-31 13:09:04 +02:00
Rene Groeschke 79ec0f2442
Split internal distribution handling into separate internal plugin (7.x backport) (#60270)
* Split internal distribution handling into separate internal plugin (#60295)
* Provide proper failure if unexpected non jdk bundled bwc version is requested
2020-07-31 09:30:21 +02:00
Jake Landis 6ce30bea08
[7.x] Convert most OSS plugins from integTest to [yaml | java]RestTest or internalClusterTest (#59444) (#60343)
For all OSS plugins (except repository-* and discovery-*) integTest
task is now a no-op and all of the tests are now executed via a test,
yamlRestTest, javaRestTest, or internalClusterTest.

This commit does NOT convert the discovery-* and repository-* since they
are bit more complex then the rest of tests and this PR is large enough.
Those plugins will be addressed in a future PR(s).

This commit also fixes a minor issue that did not copy the rest api
for projects that only had YAML TEST tests.

related: #56841
2020-07-29 13:06:13 -05:00
Mark Vieira 39fa1c4df0
Add compatibility testing for JDBC driver (#60409)
This commit adds compatibility testing of our JDBC driver against
different Elasticsearch versions. Although we are really testing the
forwards compatibility nature of the JDBC driver we model the testing
the same as we do existing BWC tests, that is, with the current branch
fetching the earlier versions of the artifact that is to be tested. In
this case, that's the JDBC driver itself.

Because the tests include the JDBC driver jar on it's classpath we had
to change the packaging of the driver jar in order to avoid jarhell and
other conflicting dependency issues when using an old JDBC driver with
later branches. For this we simply relocate all driver dependencies in
the shadow jar under a "shadowed" package. This allows the JDBC driver
to use the correct version of Elasticsearch libs classes, while the
tests themselves use their versions. Since this required a change to the
driver jar compatibility testing can only go back as far as that version
which at the time of this commit is 7.8.1.
2020-07-29 10:45:11 -07:00
Jake Landis 92ce41cfaf
[7.x] Introduce javaRestTest source set/task and convert modules (#59939) (#60026)
Introduce a javaRestTest source set and task to compliment the yamlRestTest.
javaRestTest differs such that the code is sourced from Java and may have
different dependencies and setup requirements for the test clusters. This also
allows the tests to run in parallel in different cluster instances to prevent any
cross test contamination between the two types of tests.

Included in this PR is all :modules no longer use the integTest task. The tests
are now driven by test, yamlRestTest, javaRestTest, and internalClusterTest.
Since only :modules (and :rest-api-spec) have been converted to yamlRestTest
we can now disable the integTest task if either yamlRestTest or javaRestTest have
been applied. Once all projects are converted, we can delete the integTest task.

related: #56841
related: #59444
2020-07-28 08:39:11 -05:00
Rene Groeschke 3fe6635b92
Remove stale gradle plugin descriptor 2020-07-22 09:10:01 +02:00
Rene Groeschke c6d3af35b9
Simplify gradle test task error reporting (#59760) (#59964)
* Simplify test error reporting

- avoid using extra plugin
- avoid extra task listener (should be avoided related to #57918 )
- keep all logic in the listener
2020-07-21 14:13:35 +02:00
Rene Groeschke 60d46f1d13
Remove superflous enforce deprecation failure plugin (#59770) (#59898)
- With enforcing the build to fail on all gradle deprecated api usage we do not need
this tailored plugin anymore
2020-07-21 12:54:17 +02:00
Rene Groeschke e31ebc96f9
Enforce fail on deprecated gradle usage (7.x backport) (#59758)
* Enforce fail on deprecated gradle usage (#59598)
* Fix branch specific deprecated gradle api usages
* Fix archiveVersion property usage
2020-07-20 08:52:30 +02:00
Ryan Ernst e963918830
Ensure precommit runs as part of check (#59476)
Precommit is setup to run as a dependency of the check task, but
unfortunately this wiring was only happening when the java plugin (but
not java-base plugin) was applied. This commit moves the wiring to occur
whenever the check task exists, which is with the lifecycle-base plugin.
2020-07-17 15:41:37 -07:00
Rene Groeschke afbc4678ae
Remove deprecated compileOnly resolution (#58857) (#59709)
- Ideally this should not be required but current runtime classpath
seems to require this. We need to revisit the classpath declarations in detail
to be able to remove this type of (in theory) superflous compile only references

Related to #57920
2020-07-17 12:06:57 +02:00