Commit Graph

62 Commits

Author SHA1 Message Date
Nick Knize 1babf43777
[Rename] Fix env variables and old es maven repo (#439)
This commit fixes some straggling es. environment variables along with a
reference to old elasticsearch versions from an opensearch maven repo that
doesn't exist.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-03-22 10:17:54 -07:00
Rabi Panda df11cc9de4 [Rename] Fix gradle build as part of the renaming process. (#397)
This commit fixes the currently broken gradle build resulted from the renaming work. It reverts a few dependencies and comments out the `opensearch_distibutions` task which is currently failing for some builds. We will address these separately in the future once we have a working build.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-03-21 20:56:34 -05:00
Rabi Panda da8a35f464 [Rename] Fix imports in the modules directory. (#386)
Fix import issues in the modules directory resulting from the refactoring.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-03-21 20:56:34 -05:00
Rabi Panda 13f6d23e40 [Rename] Property and metadata keys with prefix es. (#389)
Rename all property and metadata keys with prefix 'es.' to 'opensearch.'.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-03-21 20:56:34 -05:00
Rabi Panda 046d877358 [Rename] modules/reindex (#230)
Refactor reindex module as part of the Elasticsearch to OpenSearch renaming effort.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-03-21 20:56:34 -05:00
Ryan Ernst d6e17170c3
Simplify adding plugins and modules to testclusters (#61886)
There are currently half a dozen ways to add plugins and modules for
test clusters to use. All of them require the calling project to peek
into the plugin or module they want to use to grab its bundlePlugin
task, and then both depend on that task, as well as extract the archive
path the task will produce. This creates cross project dependencies that
are difficult to detect, and if the dependent plugin/module has not yet
been configured, the build will fail because the task does not yet
exist.

This commit makes the plugin and module methods for testclusters
symmetetric, and simply adding a file provider directly, or a project
path that will produce the plugin/module zip. Internally this new
variant uses normal configuration/dependencies across projects to get
the zip artifact. It also has the added benefit of no longer needing the
caller to add to the test task a dependsOn for bundlePlugin task.
2020-09-03 19:37:46 -07:00
Jake Landis f3752ba1d5
7.x suport new path for re-index java-api doc (#60319)
This commit uses the new location for the reindex java-api documentation.
Temporary files have been left behind to pacify the docs build.

related #60339
2020-08-05 09:05:07 -05:00
Rene Groeschke bdd7347bbf
Merge test runner task into RestIntegTest (7.x backport) (#60600)
* Merge test runner task into RestIntegTest (#60261)
* Merge test runner task into RestIntegTest
* Reorganizing Standalone runner and RestIntegTest task
* Rework general test task configuration and extension
* Fix merge issues
* use former 7.x common test configuration
2020-08-04 14:46:32 +02:00
Jake Landis 92ce41cfaf
[7.x] Introduce javaRestTest source set/task and convert modules (#59939) (#60026)
Introduce a javaRestTest source set and task to compliment the yamlRestTest.
javaRestTest differs such that the code is sourced from Java and may have
different dependencies and setup requirements for the test clusters. This also
allows the tests to run in parallel in different cluster instances to prevent any
cross test contamination between the two types of tests.

Included in this PR is all :modules no longer use the integTest task. The tests
are now driven by test, yamlRestTest, javaRestTest, and internalClusterTest.
Since only :modules (and :rest-api-spec) have been converted to yamlRestTest
we can now disable the integTest task if either yamlRestTest or javaRestTest have
been applied. Once all projects are converted, we can delete the integTest task.

related: #56841
related: #59444
2020-07-28 08:39:11 -05:00
Jake Landis 665b7b7bd8
Convert modules to use yamlRestTest (#59089) (#59446)
This commit moves the modules REST tests to the
newly introduced yamlRestTest source set. A few
tests have also been re-named to include the correct
IT suffix. Without changing the names, the testing
conventions task would fail since now that the YAML
tests are no longer present pacify the convention.
These tests have moved to the internalClusterTest
source set.

related: #56841
2020-07-13 13:53:05 -05:00
Jake Landis 604c6dd528
7.x - Create plugin for yamlTest task (#56841) (#59090)
This commit creates a new Gradle plugin to provide a separate task name
and source set for running YAML based REST tests. The only project
converted to use the new plugin in this PR is distribution/archives/integ-test-zip.
For which the testing has been moved to :rest-api-spec since it makes the most
sense and it avoids a small but awkward change to the distribution plugin.

The remaining cases in modules, plugins, and x-pack will be handled in followups.

This plugin is distinctly different from the plugin introduced in #55896 since
the YAML REST tests are intended to be black box tests over HTTP. As such they
should not (by default) have access to the classpath for that which they are testing.

The YAML based REST tests will be moved to separate source sets (yamlRestTest).
The which source is the target for the test resources is dependent on if this
new plugin is applied. If it is not applied, it will default to the test source
set.

Further, this introduces a breaking change for plugin developers that
use the YAML testing framework. They will now need to either use the new source set
and matching task, or configure the rest resources to use the old "test" source set that
matches the old integTest task. (The former should be preferred).

As part of this change (which is also breaking for plugin developers) the
rest resources plugin has been removed from the build plugin and now requires
either explicit application or application via the new YAML REST test plugin.

Plugin developers should be able to fix the breaking changes to the YAML tests
by adding apply plugin: 'elasticsearch.yaml-rest-test' and moving the YAML tests
under a yamlRestTest folder (instead of test)
2020-07-06 14:16:26 -05:00
Rene Groeschke d952b101e6
Replace compile configuration usage with api (7.x backport) (#58721)
* Replace compile configuration usage with api (#58451)

- Use java-library instead of plugin to allow api configuration usage
- Remove explicit references to runtime configurations in dependency declarations
- Make test runtime classpath input for testing convention
  - required as java library will by default not have build jar file
  - jar file is now explicit input of the task and gradle will ensure its properly build

* Fix compile usages in 7.x branch
2020-06-30 15:57:41 +02:00
Rene Groeschke 01e9126588
Remove deprecated usage of testCompile configuration (#57921) (#58083)
* Remove usage of deprecated testCompile configuration
* Replace testCompile usage by testImplementation
* Make testImplementation non transitive by default (as we did for testCompile)
* Update CONTRIBUTING about using testImplementation for test dependencies
* Fail on testCompile configuration usage
2020-06-14 22:30:44 +02:00
Ryan Ernst 70b499b7aa
Simplify java home verification (#55635)
* Simplify java home verification

At one time, all uses of java home were found through the getJavaHome
utility method on BuildPlugin. However, that was changed many
refactorings ago, but the complex support for registering a java home
version needed that fails at configuration time still exists. The only
remaining use of grabbing java home is within bwc tests, and must be at
runtime since that is when we have the checkout and know what version is
needed.

This commit consolidates the java home finding method into a utility
unassociated with BuildPlugin.

* fix checkstyle

* address feedback
2020-04-27 12:43:32 -07:00
Jake Landis a2fafa6af4
[7.x] Lazy test cluster module and plugins (#54852) (#55087)
This change converts the module and plugin parameters
for testClusters to be lazy. Meaning that the values
are not resolved until they are actually used. This
removes the requirement to use project.afterEvaluate to
be able to resolve the bundle artifact.

Note - this does not completely remove the need for afterEvaluate
since it is still needed for the custom resource extension.
2020-04-13 10:53:35 -05:00
Mark Vieira dd73a14d11
Improve total build configuration time (#54611) (#54994)
This commit includes a number of changes to reduce overall build
configuration time. These optimizations include:

- Removing the usage of the 'nebula.info-scm' plugin. This plugin
   leverages jgit to load read various pieces of VCS information. This
   is mostly overkill and we have our own minimal implementation for
   determining the current commit id.
- Removing unnecessary build dependencies such as perforce and jgit
   now that we don't need them. This reduces our classpath considerably.
- Expanding the usage lazy task creation, particularly in our
   distribution projects. The archives and packages projects create
   lots of tasks with very complex configuration. Avoiding the creation
   of these tasks at configuration time gives us a nice boost.
2020-04-08 16:47:02 -07:00
Jake Landis db3420d757
[7.x] Optimize which Rest resources are used by the Rest tests… (#53766)
This should help with Gradle's incremental compile such that projects
only depend upon the resources they use.

related #52114
2020-03-19 12:28:59 -05:00
Henning Andersen 9085024e1d Disable reindex against 0.90 on mac (#51449)
We still test remote reindex against version 0.90. This failed on mac a
few times and rather than spend time investigating this, we no longer
test remote reindex against 0.90 on mac.

Closes #51202
2020-01-27 12:42:12 +01:00
Ryan Ernst c6a8913c38 Fix java home validation usage by tasks (#49204)
Tasks intending to use a particular java home provided by JAVA<N>_HOME
use the getJavaHome method, which verifies the given java home is
available, or will be if the task will run. However, the verification
logic was broken, in addition to unnecessarily delaying retrieving the
java home until runtime. This commit fixes the verification logic to run
at either config time, delaying verification, or at runtime which
immediately checks if java home is available.

closes #49153
2019-11-19 10:30:19 -08:00
Rory Hunter c46a0e8708
Apply 2-space indent to all gradle scripts (#49071)
Backport of #48849. Update `.editorconfig` to make the Java settings the
default for all files, and then apply a 2-space indent to all `*.gradle`
files. Then reformat all the files.
2019-11-14 11:01:23 +00:00
Mark Vieira 6ab4645f4e
[7.x] Introduce type-safe and consistent pattern for handling build globals (#48818)
This commit introduces a consistent, and type-safe manner for handling
global build parameters through out our build logic. Primarily this
replaces the existing usages of extra properties with static accessors.
It also introduces and explicit API for initialization and mutation of
any such parameters, as well as better error handling for uninitialized
or eager access of parameter values.

Closes #42042
2019-11-01 11:33:11 -07:00
Alpar Torok 36d018c909 Convert RunTask to use testclusers, remove ClusterFormationTasks (#47572)
* Convert RunTask to use testclusers, remove ClusterFormationTasks

This PR adds a new RunTask and a way for it to start a
testclusters cluster out of band and block on it to replace
the old RunTask that used ClusterFormationTasks.

With this we can now remove ClusterFormationTasks.
2019-10-08 14:43:29 +03:00
Tim Brooks d18f511327
Propogate version in reindex from remote search (#42958)
This is related to #31908. In order to use the external version in a
reindex from remote request, the search request must be configured to
request the version (as it is not returned by default). This commit
modifies the search request to request the version. Additionally, it
modifies our current reindex from remote tests to randomly use the
external version_type.
2019-06-06 14:50:06 -04:00
Mark Vieira e44b8b1e2e
[Backport] Remove dependency substitutions 7.x (#42866)
* Remove unnecessary usage of Gradle dependency substitution rules (#42773)

(cherry picked from commit 12d583dbf6f7d44f00aa365e34fc7e937c3c61f7)
2019-06-04 13:50:23 -07:00
Jason Tedor 6362ac14ad
Lazily compute Java 8 home in reindex configuration (#42630)
In the reindex from old tests we require Java 8. Today when configuring
the reindex from old tests, we eagerly evalulate Java 8 home, which
means that we require JAVA8_HOME to be set even if the reindex from old
test tasks are not in the task graph. This is an onerous requirement if,
for example, all that you want to do is build a distribution. This
commit addresses this by making evaluation of Java 8 home lazy, so that
it is only done and required if the reindex from old test tasks would be
executed.
2019-05-28 13:06:48 -04:00
Alpar Torok e1e2568fa3 Add FIPS specific testclusters configuration (#41199)
ClusterFormationTasks auto configured these properties for clusters.
This PR adds FIPS specific configuration across all test clusters from
the main build script to prevent coupling betwwen testclusters and the
build plugin.

Closes #40904
2019-04-19 10:36:54 +03:00
Armin Braun 9cfe194131
Disable TestingConventions Check in Reindex Module (#41100) (#41110)
* This is currently failing with:

```
Test classes are not included in any enabled task (:modules:reindex:test):
  * org.elasticsearch.client.documentation.ReindexDocumentationIT
  * org.elasticsearch.index.reindex.ManyDocumentsIT
  * org.elasticsearch.index.reindex.ReindexClientYamlTestSuiteIT
  * org.elasticsearch.index.reindex.ReindexWithoutContentIT
  * org.elasticsearch.index.reindex.remote.ReindexFromOldRemoteIT
```

* Same fix as in #38546
2019-04-11 13:47:14 +02:00
Mark Vieira 1287c7d91f
[Backport] Replace usages RandomizedTestingTask with built-in Gradle Test (#40978) (#40993)
* Replace usages RandomizedTestingTask with built-in Gradle Test (#40978)

This commit replaces the existing RandomizedTestingTask and supporting code with Gradle's built-in JUnit support via the Test task type. Additionally, the previous workaround to disable all tasks named "test" and create new unit testing tasks named "unitTest" has been removed such that the "test" task now runs unit tests as per the normal Gradle Java plugin conventions.

(cherry picked from commit 323f312bbc829a63056a79ebe45adced5099f6e6)

* Fix forking JVM runner

* Don't bump shadow plugin version
2019-04-09 11:52:50 -07:00
Ioannis Kakavas c9ff630a12 Mute reindex integTest in FIPS (#40941)
Relates: #40904
2019-04-08 13:12:13 +03:00
Alpar Torok 25944c4317 convert modules to use testclusters (#40804)
* convert modules to use testclusters
* Eliminate PluginPropertiesTask and move logic in plugin where it belongs
2019-04-04 11:45:40 +03:00
Tim Vernum a8596de31f
Introduce ssl settings to reindex from remote (#37527)
Adds reindex.ssl.* settings for reindex from remote.

This uses the ssl-config/ internal library to parse and load SSL
configuration and files. This is applied when using the low level
rest client to connect to a remote ES node

Relates: #37287
Resolves: #29755
2019-01-31 18:06:05 +11:00
Alpar Torok a7c3d5842a
Split third party audit exclusions by type (#36763) 2019-01-07 17:24:19 +02:00
Alpar Torok e9ef5bdce8
Converting randomized testing to create a separate unitTest task instead of replacing the builtin test task (#36311)
- Create a separate unitTest task instead of Gradle's built in 
- convert all configuration to use the new task 
- the  built in task is now disabled
2018-12-19 08:25:20 +02:00
Jason Tedor 48ab6d7c92
Switch old reindex tests to use use JDK 8 (#34646)
This commit switches the old reindex tests to use JDK 8. This enables us
to not require that JDK 7 be installed, easing the burden on developers.
2018-10-19 15:11:48 -04:00
Tanguy Leroux 079d130d8c
[Test] Remove duplicate method in TestShardRouting (#32815) 2018-09-10 18:29:00 +02:00
Tanguy Leroux 80c4661d0c
Fix typos (#33499) 2018-09-10 14:20:04 +02:00
Tanguy Leroux 8b4d80ad09
Fix AntFixture waiting condition (#31272)
The AntFixture waiting condition is evaluated to false 
but it should be true.
2018-06-13 12:40:22 +02:00
Nik Everett 51fa8739ea
Reindex: Fold "with all deps" project into reindex (#30154)
This folds the `:qa:smoke-test-reindex-with-all-modules` project into
`:modules:reindex` by declaring the reindex's integration testing
cluster requires the `parent-join` and `lang-painless` plugins and then
moving all of the integration tests that depended on parent-join and
painless into reindex.

It saves us one cluster start up during the build at the cost of a
little of the reindex module's "purity". Since the reindex module *does*
have unit tests that test scripting without painless I'm fairly ok with
that.
2018-05-10 08:02:23 -04:00
Nik Everett d12e644206
Build: Log a warning if disabling reindex-from-old (#30304)
We disable the reindex-from-old tests if we're running on windows or in
a directory that contains a space. This adds a warning to the logs when
we do that so that you can tell that it happened. This will be nice to
have when looking at CI and will be a hint to anyone developing locally.
2018-05-01 11:23:18 -04:00
David Turner d2ca16b4c7 Suppress reindex-from-old tests if there are spaces in the path 2018-05-01 14:32:13 +01:00
Nik Everett 912fbb2211
Reindex: Fold "from old" tests into reindex module (#30142)
This folds the `:qa:reindex-from-old` project into the `:modules:reindex`
project. This should speed up the build marginally by removing a single
clsuter start up at the cost of having to wait for old versions of
Elasticsearch to start up when checking reindex's integration tests.
Those don't take that long so this feels worth it.
2018-04-27 14:04:37 -04:00
Ryan Ernst 19a2b01e43
Build: Omit dependency licenses check for elasticsearch deps (#28304)
Sometimes modules/plugins depend on locally built elasticsearch jars.
This means not only that the jar is constantly changing (so no need for
a sha check), but also that the license falls under the Elasticsearch
license, and there is no need to keep another copy. This commit updates
the dependencies checked by dependencyLicenses to exclude those that are
built by elasticsearch.
2018-01-18 14:15:44 -08:00
Jason Tedor 9aae2f593a Avoid stack overflow on search phases
When a search is executing locally over many shards, we can stack
overflow during query phase execution. This happens due to callbacks
that occur after a phase completes for a shard and we move to the same
phase on another shard. If all the shards for the query are local to the
local node then we will never go async and these callbacks will end up
as recursive calls. With sufficiently many shards, this will end up as a
stack overflow. This commit addresses this by truncating the stack by
forking to another thread on the executor for the phase.

Relates #27069
2017-10-25 22:05:46 -04:00
Nik Everett eb754a71be Fix update_by_query's default size parameter (#26784)
We were accidentally defaulting it to the scroll size.
Untwists some of the tricks that we play with parsing
so that the size is no longer scrambled.

Closes #26761
2017-09-25 16:25:27 -04:00
Michael Basnight cfd14cd2b8 Revert shading for the low level rest client (#26367)
At current, we do not feel there is enough of a reason to shade the low
level rest client. It caused problems with commons logging and IDE's
during the brief time it was used. We did not know exactly how many
users will need this, and decided that leaving shading out until we
gather more information is best. Users can still shade the jar
themselves. For information and feeback, see issue #26366.

Closes #26328

This reverts commit 3a20922046.
This reverts commit 2c271f0f22.
This reverts commit 9d10dbea39.
This reverts commit e816ef89a2.
2017-08-25 14:13:12 -05:00
Michael Basnight e816ef89a2 Shade external dependencies in the rest client jar
This commit removes all external dependencies from the rest client jar
and shades them in an 'org.elasticsearch.client' package within the jar
using shadowJar gradle plugin. All projects that depended on the
existing jar have been converted to using the 'org.elasticsearch.client'
package prefixes to interact with the rest client.

Closes #25208
2017-07-24 12:55:43 -05:00
Luca Cavanna ec66d655b5 Rename client artifacts (#25693)
It was brought up that our current client artifacts have generic names like 'rest' that may cause conflicts with other artifacts.

This commit renames:

- rest -> elasticsearch-rest-client
- sniffer -> elasticsearch-rest-client-sniffer
- rest-high-level -> elasticsearch-rest-high-level-client

A couple of small changes are also preparing the high level client for its first release.

Closes #20248
2017-07-13 09:44:25 +02:00
Jim Ferenczi 279a18a527 Add parent-join module (#24638)
* Add parent-join module

This change adds a new module named `parent-join`.
The goal of this module is to provide a replacement for the `_parent` field but as a first step this change only moves the `has_child`, `has_parent` queries and the `children` aggregation to this module.
These queries and aggregations are no longer in core but they are deployed by default as a module.

Relates #20257
2017-05-12 15:58:06 +02:00
Ryan Ernst 175bda64a0 Build: Rework integ test setup and shutdown to ensure stop runs when desired (#23304)
Gradle's finalizedBy on tasks only ensures one task runs after another,
but not immediately after. This is problematic for our integration tests
since it allows multiple project's integ test clusters to be
simultaneously. While this has not been a problem thus far (gradle 2.13
happened to keep the finalizedBy tasks close enough that no clusters
were running in parallel), with gradle 3.3 the task graph generation has
changed, and numerous clusters may be running simultaneously, causing
memory pressure, and thus generally slower tests, or even failure if the
system has a limited amount of memory (eg in a vagrant host).

This commit reworks how integ tests are configured. It adds an
`integTestCluster` extension to gradle which is equivalent to the current
`integTest.cluster` and moves the rest test runner task to
`integTestRunner`.  The `integTest` task is then just a dummy task,
which depends on the cluster runner task, as well as the cluster stop
task. This means running `integTest` in one project will both run the
rest tests, and shut down the cluster, before running `integTest` in
another project.
2017-02-22 12:43:15 -08:00
Tim Brooks a10aa8aade Add TestWithDependenciesPlugin to build (#22646)
This commit adds a MessyRestTestPlugin to the gradle build. It extends 
StandaloneRestTestPlugin. The main piece of functionality that it adds 
is to copy plugin-metadata from dependencies into the 
generated-resources for the current test source. This is necessary to 
ensure that permissions for dependencies are applied when running the 
tests.

A current limitation is that the permissions are applied differently 
than in the distribution sources. When permissions are granted to all 
depedencies for a module or plugin, the permissions are granted to all 
dependencies on the classpath for tests besides a few hardcoded 
exclusions:
- es core
- es test framework
- lucene test framework
- randomized runner
- junit library
2017-01-19 09:43:53 -06:00