Commit Graph

1371 Commits

Author SHA1 Message Date
Ryan Ernst 381ed235e9
Re-enable plugin and upgrade bats tests () ()
While the tests remained in place, the execution of these tests were
lost when creating the DistroTestPlugin. This commit restores executing
them.
2020-05-20 08:34:05 -07:00
Ryan Ernst 79224a9a4a
Do not daemonize when testing error output in packaging tests () ()
The packaging tests start elasticsearch in various ways. All of these
currently expect it is started asynchronously, yet some tests expect it
will fail to start and want to check the error output. This commit adds
a daemonize flag to the utility methods to start elasticsearch for such
cases, so that when the start method returns, all the error output
should already be available since the process will have exited.

relates 
2020-05-20 07:56:48 -07:00
Andrei Dan c8278e333a
Enable decompression of response within LowLevelRestClient () ()
Added support for decompression at LLRC and added integration test

(cherry picked from commit 2621452473e0c236aa28db749f782a24eca6c974)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>

Co-authored-by: Hakky54 <hakangoudberg@hotmail.com>
2020-05-15 16:50:45 +01:00
David Roberts ab40466bfb
Prevent unexpected native controller output hanging the process ()
In normal operation native controllers are not expected to write
anything to stdout or stderr.  However, if due to an error or
something unexpected with the environment a native controller
does write something to stdout or stderr then it will block if
nothing is reading that output.

This change makes the stdout and stderr of native controllers
reuse the same stdout and stderr as the Elasticsearch JVM (which
are by default redirected to es.stdout.log and es.stderr.log) so
that if something unexpected is written to native controller
output then:

1. The native controller process does not block, waiting for
   something to read the output
2. We can see what the output was, making it easier to debug
   obscure environmental problems

Backport of 

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-05-13 22:57:00 +01:00
Ignacio Vera b4521d5183
upgrade to Lucene 8.6.0 snapshot () 2020-05-13 14:25:16 +02:00
Rene Groeschke c29bc87040
Move bwcVersions extension property to BuildParams (back port) ()
* Move bwcVersions extension property to BuildParams ()
* Fix :qa Task Using Broken BwC Versions Resolution ()

Co-authored-by: Armin Braun <me@obrown.io>
2020-05-11 09:39:13 +02:00
Hendrik Muhs ad54c51467
[7.x] add a basic get index rolling upgrade test () ()
add a very basic rolling upgrade test for get index, post mortem action of 
2020-05-11 07:51:38 +02:00
Hendrik Muhs e2e4c3179c Revert "add a basic get index rolling upgrade test ()"
This reverts commit 0b40886db3.
2020-05-08 12:41:15 +02:00
Hendrik Muhs 406d58f28c Revert "dont use mappings for the test, due to different output in 6.9 (type removal)"
This reverts commit b2c70a95da.
2020-05-08 12:41:11 +02:00
Hendrik Muhs b2c70a95da dont use mappings for the test, due to different output in 6.9 (type removal) 2020-05-08 11:53:47 +02:00
Hendrik Muhs 0b40886db3 add a basic get index rolling upgrade test ()
add a very basic rolling upgrade test for get index, post mortem action of 
2020-05-08 10:53:01 +02:00
Igor Motov d8f9df771d
Expose agg usage in Feature Usage API () ()
Counts usage of the aggs and exposes them on the _nodes/usage/.

Closes 
2020-04-30 12:53:36 -04:00
Rory Hunter c2c14e8875
Use official checksums to verify Tini ()
Firstly, backport the use of tini as the Docker entrypoint. This was supposed
to have been done following , but was missed. It isn't a direct backport
as this branch will continue using root as the initial Docker user.

Secondly, backport  to use the official checksums when downloading tini.
2020-04-24 16:47:24 +01:00
Rory Hunter d66af46724
Always use deprecateAndMaybeLog for deprecation warnings ()
Backport of .

Replace calls to deprecate(String,Object...) with deprecateAndMaybeLog(...),
with an appropriate key, so that all messages can potentially be deduplicated.
2020-04-23 09:20:54 +01:00
Ryan Ernst 742b69a7dc
Make windows JAVA_HOME handling consistent with linux () ()
In bash, checking for whether an env variable exists uses the -z test,
against a stringified env var, so that the test is actually whether the
env var is empty, but not necessarily undefined. We use this to test
whether JAVA_HOME is set, to determine whether the bundled jdk should be
used. In windows, this test is an actual "undefined" check. This commit
brings the behavior on two systems in sync, opting to allow for an empty
JAVA_HOME in windows to indicate the bundled jdk should be used.

closes 
2020-04-16 16:17:41 -07:00
Mark Tozzi 22c55180c1
[7.x] Backport ValuesSourceRegistry and related work ()
* Add ValuesSource Registry and associated logic ()

* Remove ValuesSourceType argument to ValuesSourceAggregationBuilder ()

* ValuesSourceRegistry Prototype ()

* Remove generics from ValuesSource related classes ()

* fix percentile aggregation tests ()

* Basic thread safety for ValuesSourceRegistry ()

* Remove target value type from ValuesSourceAggregationBuilder ()

* Cleanup default values source type ()

* CoreValuesSourceType no longer implements Writable ()

* Remove genereics & hard coded ValuesSource references from Matrix Stats ()

* Put values source types on fields ()

* Remove VST Any ()

* Rewire terms agg to use new VS registry ()

Also adds some basic AggTestCases for untested code
paths (and boilerplate for future tests once the IT are
converted over)

* Wire Cardinality aggregation to work with the ValuesSourceRegistry ()

* Wire Percentiles aggregator into new VS framework ()

This required a bit of a refactor to percentiles itself.  Before,
the Builder would switch on the chosen algo to generate an
algo-specific factory.  This doesn't work (or at least, would be
difficult) in the new VS framework.

This refactor consolidates both factories together and introduces
a PercentilesConfig object to act as a standardized way to pass
algo-specific parameters through the factory.  This object
is then used when deciding which kind of aggregator to create

Note: CoreValuesSourceType.HISTOGRAM still lives in core, and will
be moved in a subsequent PR.

* Remove generics and target value type from MultiVSAB ()

* fix checkstyle after merge ()

* Plumb ValuesSourceRegistry through to QuerySearchContext ()

* Convert RareTerms to new VS registry ()

* Wire up Value Count ()

* Wire up Max & Min aggregations ()

* ValuesSource refactoring: Wire up Sum aggregation ()

* ValuesSource refactoring: Wire up SigTerms aggregation ()

* Soft immutability for VSConfig ()

* Unmute testSupportedFieldTypes, fix Percentiles/Ranks/Terms tests ()

Also fixes Percentiles which was incorrectly specified to only accept
numeric, but in fact also accepts Boolean and Date (because those are
numeric on master - thanks `testSupportedFieldTypes` for catching it!)

* VS refactoring: Wire up stats aggregation ()

* ValuesSource refactoring: Wire up string_stats aggregation ()

* VS refactoring: Wire up median (MAD) aggregation ()

* fix valuesourcetype issue with constant_keyword field ()x-pack/plugin/rollup/src/main/java/org/elasticsearch/xpack/rollup/job/RollupIndexer.java

this commit implements `getValuesSourceType` for
the ConstantKeyword field type.

master was merged into feature/extensible-values-source
introducing a new field type that was not implementing
`getValuesSourceType`.

* ValuesSource refactoring: Wire up Avg aggregation ()

* Wire PercentileRanks aggregator into new VS framework  ()

* Add a VSConfig resolver for aggregations not using the registry ()

* Vs refactor wire up ranges and date ranges ()

* Wire up geo_bounds aggregation to ValuesSourceRegistry ()

This commit updates the geo_bounds aggregation to depend
on registering itself in the ValuesSourceRegistry

relates .

* VS refactoring: convert Boxplot to new registry ()

* Wire-up geotile_grid and geohash_grid to ValuesSourceRegistry ()

This commit updates the geo*_grid aggregations to depend
on registering itself in the ValuesSourceRegistry

relates to the values-source refactoring meta issue .

* Wire-up geo_centroid agg to ValuesSourceRegistry ()

This commit updates the geo_centroid aggregation to depend
on registering itself in the ValuesSourceRegistry.

relates to the values-source refactoring meta issue .

* Fix type tests for Missing aggregation ()

* ValuesSource Refactor: move histo VSType into XPack module ()

- Introduces a new API (`getBareAggregatorRegistrar()`) which allows plugins to register aggregations against existing agg definitions defined in Core.
- This moves the histogram VSType over to XPack where it belongs. `getHistogramValues()` still remains as a Core concept
- Moves the histo-specific bits over to xpack (e.g. the actual aggregator logic). This requires extra boilerplate since we need to create a new "Analytics" Percentile/Rank aggregators to deal with the histo field. Doubly-so since percentiles/ranks are extra boiler-plate'y... should be much lighter for other aggs

* Wire up DateHistogram to the ValuesSourceRegistry ()

* Vs refactor parser cleanup ()

Co-authored-by: Zachary Tong <polyfractal@elastic.co>
Co-authored-by: Zachary Tong <zach@elastic.co>
Co-authored-by: Christos Soulios <1561376+csoulios@users.noreply.github.com>
Co-authored-by: Tal Levy <JubBoy333@gmail.com>

* First batch of easy fixes

* Remove List.of from ValuesSourceRegistry

Note that we intend to have a follow up PR dealing with the mutability
of the registry, so I didn't even try to address that here.

* More compiler fixes

* More compiler fixes

* More compiler fixes

* Precommit is happy and so am I

* Add new Core VSTs to tests

* Disabled supported type test on SigTerms until we can backport it's fix

* fix checkstyle

* Fix test failure from semantic merge issue

* Fix some metaData->metadata replacements that got lost

* Fix list of supported types for MinAggregator

* Fix list of supported types for Avg

* remove unused import

Co-authored-by: Zachary Tong <polyfractal@elastic.co>
Co-authored-by: Zachary Tong <zach@elastic.co>
Co-authored-by: Christos Soulios <1561376+csoulios@users.noreply.github.com>
Co-authored-by: Tal Levy <JubBoy333@gmail.com>
2020-04-16 16:54:46 -04:00
David Turner 7941f4a47e Add RepositoriesService to createComponents() args ()
Today we pass the `RepositoriesService` to the searchable snapshots plugin
during the initialization of the `RepositoryModule`, forcing the plugin to be a
`RepositoryPlugin` even though it does not implement any repositories.

After discussion we decided it best for now to pass this in via
`Plugin#createComponents` instead, pending some future work in which plugins
can depend on services more dynamically.
2020-04-16 16:27:36 +01:00
William Brafford 2ba3be9db6
Remove deprecated third-party methods from tests () ()
I've noticed that a lot of our tests are using deprecated static methods
from the Hamcrest matchers. While this is not a big deal in any
objective sense, it seems like a small good thing to reduce compilation
warnings and be ready for a new release of the matcher library if we
need to upgrade. I've also switched a few other methods in tests that
have drop-in replacements.
2020-04-15 17:54:47 -04:00
Ryan Ernst 29b70733ae
Use task avoidance with forbidden apis ()
Currently forbidden apis accounts for 800+ tasks in the build. These
tasks are aggressively created by the plugin. In forbidden apis 3.0, we
will get task avoidance
(https://github.com/policeman-tools/forbidden-apis/pull/162), but we
need to ourselves use the same task avoidance mechanisms to not trigger
these task creations. This commit does that for our foribdden apis
usages, in preparation for upgrading to 3.0 when it is released.
2020-04-15 13:27:53 -07:00
Rory Hunter 70616cd76a
Define aarch64 packaging test tasks ()
Backport of .

We added tasks to build an ARM distribution and Docker image, but didn't
provide any way to run packaging tests against them. Add extra loops on
the possible Architecture values, and skip tasks that can't be run on
the current Architecture.
2020-04-15 13:39:18 +01:00
Rory Hunter 18dc2f7330 Rename some Docker projects for consistency ()
Apply the :distribution:archives naming convention to some of the Docker
sub-projects, so that we have a more consistent naming scheme.

Also, we've seen some examples of Docker packaging tests failing sporadically
when they try to clean up the temp directory, citing a not-empty
directory. Ensure that any running container is removed before cleaning
up the temp dir, in an effort to avoid this problem.
2020-04-14 22:09:05 +01:00
Jake Landis a2fafa6af4
[7.x] Lazy test cluster module and plugins () ()
This change converts the module and plugin parameters
for testClusters to be lazy. Meaning that the values
are not resolved until they are actually used. This
removes the requirement to use project.afterEvaluate to
be able to resolve the bundle artifact.

Note - this does not completely remove the need for afterEvaluate
since it is still needed for the custom resource extension.
2020-04-13 10:53:35 -05:00
Mark Vieira dd73a14d11
Improve total build configuration time () ()
This commit includes a number of changes to reduce overall build
configuration time. These optimizations include:

- Removing the usage of the 'nebula.info-scm' plugin. This plugin
   leverages jgit to load read various pieces of VCS information. This
   is mostly overkill and we have our own minimal implementation for
   determining the current commit id.
- Removing unnecessary build dependencies such as perforce and jgit
   now that we don't need them. This reduces our classpath considerably.
- Expanding the usage lazy task creation, particularly in our
   distribution projects. The archives and packages projects create
   lots of tasks with very complex configuration. Avoiding the creation
   of these tasks at configuration time gives us a nice boost.
2020-04-08 16:47:02 -07:00
Nik Everett 3c56e0de42
Fix scripted metric in ccs (backport of ) ()
`scripted_metric` did not work with cross cluster search because it
assumed that you'd never perform a partial reduction, serialize the
results, and then perform a final reduction. That
serialized-after-partial-reduction step was broken.

This is also required to support .
2020-04-07 10:43:00 -04:00
Jason Tedor 5fcda57b37
Rename MetaData to Metadata in all of the places ()
This is a simple naming change PR, to fix the fact that "metadata" is a
single English word, and for too long we have not followed general
naming conventions for it. We are also not consistent about it, for
example, METADATA instead of META_DATA if we were trying to be
consistent with MetaData (although METADATA is correct when considered
in the context of "metadata"). This was a simple find and replace across
the code base, only taking a few minutes to fix this naming issue
forever.
2020-03-31 17:24:38 -04:00
Nik Everett 8f40f1435a
Save a little space in agg tree (backport of ) ()
This drop the "top level" pipeline aggregators from the aggregation
result tree which should save a little memory and a few serialization
bytes. Perhaps more imporantly, this provides a mechanism by which we
can remove *all* pipelines from the aggregation result tree. This will
save quite a bit of space when pipelines are deep in the tree.

Sadly, doing this isn't simple because of backwards compatibility. Nodes
before 7.7.0 *need* those pipelines. We provide them by setting passing
a `Supplier<PipelineTree>` into the root of the aggregation tree that we
only call if we need to serialize to a version before 7.7.0.

This solution works for cross cluster search because we always reduce
the aggregations in each remote cluster and then forward them back to
the coordinating node. Its quite possible that the coordinating node
needs the pipeline (say it is version 7.1.0) and the gateway node in the
remote cluster doesn't (version 7.7.0). In that case the data nodes
won't send the pipeline aggregations back to the gateway node.
Critically, the gateway node *will* send the pipeline aggregations back
to the coordinating node. This is all managed with that
`Supplier<PipelineTree>`, but *how* it is managed is a bit tricky.
2020-03-25 15:51:16 -04:00
Rory Hunter b066cc587d Cleanup after DockerTests suite completes ()
Closes . The Docker tests weren't cleanup up the created container
after the last test completed. Now they are.
2020-03-25 11:58:44 +00:00
Jason Tedor 381d7586e4
Introduce formal role for remote cluster client ()
This commit introduce a formal role for identifying nodes that are
capable of making connections to remote clusters.

Relates 
2020-03-24 21:59:43 -04:00
Tim Brooks caefa78513
Align remote info api with new settings ()
Currently the remote info api has added a number of possible fields
(proxy, num_socket_connections, etc) that are available in proxy mode.
These fields are not aligned with what the settings are named. This
commit modifies this API to align with the settings.
2020-03-24 10:27:24 -06:00
Jim Ferenczi 9e3f7f4575
Add heuristics to compute pre_filter_shard_size when unspecified () ()
This commit changes the pre_filter_shard_size default from 128 to unspecified.
This allows to apply heuristics based on the request and the target indices when deciding
whether the can match phase should run or not. When unspecified, this pr runs the can match phase
automatically if one of these conditions is met:
  * The request targets more than 128 shards.
  * The request contains read-only indices.
  * The primary sort of the query targets an indexed field.
Users can opt-out from this behavior by setting the `pre_filter_shard_size` to a static value.

Closes 
2020-03-24 02:05:15 +01:00
Lee Hinman 1f3de2fa7e
Set feature flags for IndexTemplatesV2 in top-level gradle file ()
Resolves 
2020-03-20 14:52:22 -06:00
Rory Hunter 17de77fd0f
Convert Wildfly tests to run in Docker ()
Backport of .

Convert the Wildfly tests to run using Docker Compose. This drastically
simplifies the build setup, and will hopefully make the tests more
resilient.
2020-03-20 16:47:21 +00:00
Alan Woodward d23112f441 Report parser name and location in XContent deprecation warnings ()
It's simple to deprecate a field used in an ObjectParser just by adding deprecation
markers to the relevant ParseField objects. The warnings themselves don't currently
have any context - they simply say that a deprecated field has been used, but not
where in the input xcontent it appears. This commit adds the parent object parser
name and XContentLocation to these deprecation messages.

Note that the context is automatically stripped from warning messages when they
are asserted on by integration tests and REST tests, because randomization of
xcontent type during these tests means that the XContentLocation is not constant
2020-03-20 11:52:55 +00:00
Lee Hinman 40181eb200
[7.x] Fix feature flag setting for ComponentTemplate APIs (#53… ()
* Fix feature flag setting for ComponentTemplate APIs ()

The feature flag was set for *most* of the builds, but there are a couple where it was missing.

Resolves 

* Add skip for older versions of ES
2020-03-19 09:35:07 -06:00
Przemyslaw Gomulka 2438b899eb
Support joda style date patterns in 7.x ()
If an index was created in version 6 and contain a date field with a joda-style pattern it should still be allowed to search and insert document into it.
Those created in 6 but date pattern starts with 8, should be considered as java style.
2020-03-12 08:57:03 +01:00
Armin Braun 7189c57b6c
Record Force Merges in Live Commit Data () ()
* Record Force Merges in live commit data

Prerequisite of . Record force merges in the live commit data
so two shard states with the same sequence number that differ only in whether
or not they have been force merged can be distinguished when creating snapshots.
2020-03-11 06:30:36 +01:00
Mark Vieira 4b528d97ad
Consolidate duplication of BWC testing task setup in script plugin ()
(cherry picked from commit 33fc8e7ebfac8d47a5f9f026b3836bb47bea141a)
2020-03-03 14:43:02 -08:00
William Brafford d3a8ac66c6
Handle special chars in JAVA_HOME in elasticsearch-service.bat () ()
* Handle special chars in JAVA_HOME in elasticsearch-service.bat ()

* Test case for windows service where JAVA_HOME path contains spaces ()

Co-authored-by: Muhammad Shaheer Akram <41253927+shaheerakr@users.noreply.github.com>
2020-03-03 12:01:54 -05:00
Rory Hunter d863c510da
Autoformat :qa:os and :benchmarks ()
Add `:qa:os` and `:benchmarks` to the list of automatically formatted
projects, and apply some manual fix-ups to polish it up.

In particular, I noticed that `Files.write(...)` when passed a list will
automaticaly apply a UTF-8 encoding and write a newline after each line,
making it easier to use than FileUtils.append. It's even available from
1.8.

Also, in the Allocators class, a number of methods declared thrown exceptions that IntelliJ reported were never thrown, and as far as I could see this is true, so I removed the exceptions.
2020-02-28 14:48:04 +00:00
Nhat Nguyen fa701e4c1f Fix testRetentionLeasesEstablishedWhenRelocatingPrimary ()
Replace the current assertion with a more robust assertion.

Closes 
2020-02-26 22:06:01 -05:00
Tim Brooks be8d704e2b
Remove seeds depedency for remote cluster settings ()
Currently 3 remote cluster settings (ping interval, skip unavailable,
and compression) have a dependency on the seeds setting being
comfigured. With proxy mode, it is now possible that these settings the
seeds setting has not been configured. This commit removes this
dependency and adds new validation for these settings.
2020-02-26 10:17:25 -07:00
Jake Landis 8d311297ca
[7.x] Smarter copying of the rest specs and tests () ()
* Smarter copying of the rest specs and tests ()

This PR addresses the unnecessary copying of the rest specs and allows
for better semantics for which specs and tests are copied. By default
the rest specs will get copied if the project applies
`elasticsearch.standalone-rest-test` or `esplugin` and the project
has rest tests or you configure the custom extension `restResources`.

This PR also removes the need for dozens of places where the x-pack
specs were copied by supporting copying of the x-pack rest specs too.

The plugin/task introduced here can also copy the rest tests to the
local project through a similar configuration.

The new plugin/task allows a user to minimize the surface area of
which rest specs are copied. Per project can be configured to include
only a subset of the specs (or tests). Configuring a project to only
copy the specs when actually needed should help with build cache hit
rates since we can better define what is actually in use.
However, project level optimizations for build cache hit rates are
not included with this PR.

Also, with this PR you can no longer use the includePackaged flag on
integTest task.

The following items are included in this PR:
* new plugin: `elasticsearch.rest-resources`
* new tasks: CopyRestApiTask and CopyRestTestsTask - performs the copy
* new extension 'restResources'
```
restResources {
  restApi {
    includeCore 'foo' , 'bar' //will include the core specs that start with foo and bar
    includeXpack 'baz' //will include x-pack specs that start with baz
  }
  restTests {
    includeCore 'foo', 'bar' //will include the core tests that start with foo and bar
    includeXpack 'baz' //will include the x-pack tests that start with baz
  }
}

```
2020-02-26 08:13:41 -06:00
David Roberts 4cae4ded4b
[TEST] Unmute DebMetadataTests.test05CheckLintian ()
The underlying problem was fixed in elastic/ml-cpp#1019

Backport of 
2020-02-24 21:35:32 +00:00
Mark Vieira f06d692706
[Backport] Consolidate docker availability logic () 2020-02-21 15:24:05 -08:00
Rory Hunter ce7ebb2d39
Limit _FILE env var support to specific vars ()
Backport of .

Closes . Implement a list of `_FILE` env vars that will be used to
populate env vars with file content, instead of processing all `_FILE`
vars in the environment.
2020-02-21 19:36:15 +00:00
Jay Modi f3f6ff97ee
Single instance of the IndexNameExpressionResolver ()
This commit modifies the codebase so that our production code uses a
single instance of the IndexNameExpressionResolver class. This change
is being made in preparation for allowing name expression resolution
to be augmented by a plugin.

In order to remove some instances of IndexNameExpressionResolver, the
single instance is added as a parameter of Plugin#createComponents and
PersistentTaskPlugin#getPersistentTasksExecutor.

Backport of 
2020-02-21 07:50:02 -07:00
Armin Braun 4bb780bc37
Refactor Inflexible Snapshot Repository BwC () ()
* Refactor Inflexible Snapshot Repository BwC ()

Transport the version to use for  a snapshot instead of whether to use shard generations in the snapshots in progress entry. This allows making upcoming repository metadata changes in a flexible manner in an analogous way to how we handle serialization BwC elsewhere.
Also, exposing the version at the repository API level will make it easier to do BwC relevant changes in derived repositories like source only or encrypted.
2020-02-21 09:14:34 +01:00
Mark Vieira a8725cde53
Mute DebMetadataTests.test05CheckLintian
Signed-off-by: Mark Vieira <portugee@gmail.com>
2020-02-20 12:50:08 -08:00
Ryan Ernst 0656a33d62 Remove reference to removed bats test tasks
The oss and default bats tests were removed, but these references to
them remained, causing gradle failures when trying to run packaging
tests. While the upgrade and plugins bats tests should still be tested,
that is being handled in . This commit removes the outdated
references.

closes 
2020-02-15 10:21:07 -08:00
William Brafford 2311bc62f7
Filter empty lines from "docker ls" response ()
* Filter empty lines from docker ls response

In order to cut down on test time, our docker/vagrant tests build the
docker image outside of the vagrant VM. When we get around to launching
the Vagrant VM, we mount that already-built docker image to a known
location. At that point, we need to load the docker image. But we only
want to load it once. As we're running tests, we use "docker ls" to
check whether the local image is loaded for use. Empty output from the
particular ls invocation means no image is loaded.

There was a bug in how we checked this. In Java, splitting an empty
string will yield an array containing one empty string. So when we're
counting the output from the docker ls command, we need to filter out
empty lines in order to proceed to loading the image for docker tests.
2020-02-14 11:39:13 -05:00