Commit Graph

760 Commits

Author SHA1 Message Date
Martijn van Groningen 3ed60df59d
Re-enable resolve index multi cluster test (#62365)
Backport of #62361 to 7.x branch.

This test was fine and shouldn't have been muted.
The test case class should have preserved data streams as part of #62205

Closes #62210
2020-09-15 13:26:52 +02:00
David Kyle 717259a049 Revert "[ML] Add debug logging of notification messages to upgrade test (#62342)"
This reverts commit c50899dd8f.
2020-09-15 11:07:23 +01:00
David Kyle c50899dd8f
[ML] Add debug logging of notification messages to upgrade test (#62342)
For #61908
2020-09-15 08:24:13 +01:00
David Roberts 3d5c13f559
[ML] Add an assertion on annotations mappings to upgrade test (#62331)
The annotations index is not covered by the comparison between
mappings and templates, as it does not use an index template.

This commit adds an assertion on annotations index mappings
that will fail if the mappings are not upgraded as expected.

Backport of #62325
2020-09-14 18:46:35 +01:00
Nhat Nguyen aafb2cb812 Support point in time cross cluster search (#61827)
This commit integrates point in time into cross cluster search.

Relates #61062
Closes #61790
2020-09-10 19:25:48 -04:00
Martijn van Groningen 81b89fe3ba
Change yaml test suite testcase to automatically delete all data streams after each yaml test (#62214)
Backporting #62205 to 7.x branch.

This is similar to what happens for indices. Initially we decided to let each test cleanup the
data streams it created.

The reason behind this was that client yaml test runners would need to be modified to do this too and
because data steams were new, we waited with that and let each test cleanup the data stream it created.
However we sometimes have very hard to debug test failures, because many tests fail because another test
failed mid way and didn't clean up the data streams it created. Given that and data streams exist in
the code base for a while now, we should automatically delete all data streams after each yaml test.

Relates to #62190

* preserve data streams for rolling upgrade yaml tests
2020-09-10 15:10:57 +02:00
Martijn van Groningen f87fc67592
Mute resolve index data stream tests. (#62211)
Relates to #62190 and #62210
2020-09-10 13:13:50 +02:00
Hendrik Muhs ab259d37a7 mute mapping upgrade check for the .ml-notifications, as a global AwaitsFix
would mute everything I allow myself to use a comment
2020-09-10 12:24:57 +02:00
Jake Landis d8dad9ab2c
[7.x] Remove integTest task from PluginBuildPlugin (#61879) (#62135)
This commit removes `integTest` task from all es-plugins.  
Most relevant projects have been converted to use yamlRestTest, javaRestTest, 
or internalClusterTest in prior PRs. 

A few projects needed to be adjusted to allow complete removal of this task
* x-pack/plugin - converted to use yamlRestTest and javaRestTest 
* plugins/repository-hdfs - kept the integTest task, but use `rest-test` plugin to define the task
* qa/die-with-dignity - convert to javaRestTest
* x-pack/qa/security-example-spi-extension - convert to javaRestTest
* multiple projects - remove the integTest.enabled = false (yay!)

related: #61802
related: #60630
related: #59444
related: #59089
related: #56841
related: #59939
related: #55896
2020-09-09 14:25:41 -05:00
David Kyle a4fb501a33
[ML] Extra exceptions for renamed fields in mapping tests (#62151)
Add renamed 'maximum_number_trees' fields to exceptions
2020-09-09 12:03:42 +01:00
Dimitris Athanasiou 14547b14fb
[ML] Add changes stats fields as exception in mappings upgrade test (#62121)
Following #61980 we need to exclude the replaced fields in the test
in the 7.x branch.
2020-09-09 10:09:35 +03:00
Yang Wang 18a08c0cf2
Mute MlMappingsUpgradeIT testMappingsUpgrade (#61909) (#62136)
For #61908

Co-authored-by: David Kyle <david.kyle@elastic.co>
2020-09-09 13:23:36 +10:00
David Kyle fb6ee5b36d
[7.x] [ML] Assert mappings match templates in Upgrade tests (#61905)
At the end of the rolling upgrade tests check the mappings of the concrete
.ml and .transform-internal indices match the mappings in the templates.
When the templates change, the tests should prove that the mappings have
been updated in the new cluster.
2020-09-08 12:21:19 +01:00
Przemko Robakowski bb357f6aae
[7.x] Move internal index templates to composable templates (#61457) (#61661)
This change moves watcher, ILM history and SLM history templates to composable templates.
Versions are updated to reflect the switch. Only change to the templates themselves is added `_meta` to mark them as managed
2020-09-08 11:26:06 +02:00
Ryan Ernst d6e17170c3
Simplify adding plugins and modules to testclusters (#61886)
There are currently half a dozen ways to add plugins and modules for
test clusters to use. All of them require the calling project to peek
into the plugin or module they want to use to grab its bundlePlugin
task, and then both depend on that task, as well as extract the archive
path the task will produce. This creates cross project dependencies that
are difficult to detect, and if the dependent plugin/module has not yet
been configured, the build will fail because the task does not yet
exist.

This commit makes the plugin and module methods for testclusters
symmetetric, and simply adding a file provider directly, or a project
path that will produce the plugin/module zip. Internally this new
variant uses normal configuration/dependencies across projects to get
the zip artifact. It also has the added benefit of no longer needing the
caller to add to the test task a dependsOn for bundlePlugin task.
2020-09-03 19:37:46 -07:00
David Roberts 89599ba0a3
[ML] Update ML mappings upgrade test and extend to config index (#61830)
The ML mappings upgrade test had become useless as it was
checking a field that has been the same since 6.5. This
commit switches to a field that was changed in 7.9.

Additionally, the test only used to check the results index
mappings.  This commit also adds checking for the config
index.

Backport of #61340
2020-09-02 12:23:59 +01:00
Ioannis Kakavas 3640ff1ff2
Add SAML AuthN request signing tests (#61582)
- Add a unit test for our signing code
- Change SAML IT to use signed authentication requests for Shibboleth to consume

Backport of #48444
2020-08-27 10:41:56 +03:00
Ioannis Kakavas 283eaabc71
[7.x] Refactor SamlAuthenticationIT (#57162) (#61568)
Refactor the tests to not require a mock HTTP Server. This has been
the cause of flakiness and removing it doesn't affect the logical
coverage of this suite. The "fake UI" is now simulated by an
http client that makes the necessary requests to Elasticsearch APIs.
2020-08-26 15:34:56 +03:00
David Roberts d1b60269f4
[ML] Ensure annotations index mappings are up to date (#61142)
When the ML annotations index was first added, only the
ML UI wrote to it, so the code to create it was designed
with this in mind.  Now the ML backend also creates
annotations, and those mappings can change between
versions.

In this change:

1. The code that runs on the master node to create the
   annotations index if it doesn't exist but another ML
   index does also now ensures the mappings are up-to-date.
   This is good enough for the ML UI's use of the
   annotations index, because the upgrade order rules say
   that the whole Elasticsearch cluster must be upgraded
   prior to Kibana, so the master node should be on the
   newer version before Kibana tries to write an
   annotation with the new fields.
2. We now also check whether the annotations index exists
   with the correct mappings before starting an autodetect
   process on a node.  This is necessary because ML nodes
   can be upgraded before the master node, so could write
   an annotation with the new fields before the master node
   knows about the new fields.

Backport of #61107
2020-08-14 13:51:04 +01:00
Dan Hermann c17839c255
Add warning handler to resolve test failure (#60427) (#61102) 2020-08-13 10:37:08 -05:00
Yang Wang c7b0290256
Mute kerberos tests for jdk 8u[262,271) (#60995)
The Kerberos bug (JDK-8246193) is introduced in JDK 8u262 and fixed in 8u271.
This PR mute for any possible releases between these two versions.
2020-08-12 11:50:48 +10:00
Rene Groeschke bdd7347bbf
Merge test runner task into RestIntegTest (7.x backport) (#60600)
* Merge test runner task into RestIntegTest (#60261)
* Merge test runner task into RestIntegTest
* Reorganizing Standalone runner and RestIntegTest task
* Rework general test task configuration and extension
* Fix merge issues
* use former 7.x common test configuration
2020-08-04 14:46:32 +02:00
Jake Landis bcb9d06bb6
[7.x] Cleanup xpack build.gradle (#60554) (#60603)
This commit does three things:
* Removes all Copyright/license headers for the build.gradle files under x-pack. (implicit Apache license)
* Removes evaluationDependsOn(xpackModule('core')) from build.gradle files under x-pack
* Removes a place holder test in favor of disabling the test task (in the async plugin)
2020-08-03 13:11:43 -05:00
Rene Groeschke ed4b70190b
Replace immediate task creations by using task avoidance api (#60071) (#60504)
- Replace immediate task creations by using task avoidance api
- One step closer to #56610
- Still many tasks are created during configuration phase. Tackled in separate steps
2020-07-31 13:09:04 +02:00
Yang Wang 3a0e7f4294
Unmute kerberos tests for jdk 15 and mute for jdk 8u262 (#60279)
The JDK bug (https://bugs.openjdk.java.net/browse/JDK-8246193) is fixed since b26.
The tests can be unmuted since we are already using b33. However the same bug is now
affecting jdk 8u262, which is the base for current Zulu jdk 8.48. This PR mute the tests
for this specific jdk version.

Relates: #56507
2020-07-29 12:57:00 +10:00
Zachary Tong 46f9c38c33 Mute tests while waiting on 58807
Bugurl: https://github.com/elastic/elasticsearch/issues/58807
2020-07-28 12:45:49 -04:00
Dan Hermann fe12217c7f
[7.x] Move REST specs for data streams (#60111) 2020-07-23 08:10:54 -05:00
Rene Groeschke e31ebc96f9
Enforce fail on deprecated gradle usage (7.x backport) (#59758)
* Enforce fail on deprecated gradle usage (#59598)
* Fix branch specific deprecated gradle api usages
* Fix archiveVersion property usage
2020-07-20 08:52:30 +02:00
Martijn van Groningen 74c9402912
Re-enable data stream bwc tests (#59734)
after backporting #59503
Backport of #59732 yo 7.x
2020-07-16 23:59:52 +02:00
Martijn van Groningen 0096238df1
Replaced _data_stream_timestamp meta field's 'path' option with 'enabled' option (#59727)
Backport #59503 to 7.x

and adjusted exception messages.

Relates to #59076
2020-07-16 22:29:40 +02:00
Martijn van Groningen 4089cbd767
Ignore multiple matching templates warning in specific tests. (#59692) (#59715)
Closes #59679
2020-07-16 20:07:38 +02:00
Yang Wang 067db1fc3b
Fix test of API key creation in a mixed cluster (#59680)
RoleDescriptors are mandatory prior to v7.3

Relates: #59425
2020-07-16 12:44:17 +10:00
Lee Hinman 74372df824
Mute {p0=mixed_cluster/120_api_key_auth/Test API key authentication will work in a mixed cluster} (#59663)
Relates to #59425
2020-07-15 14:14:33 -06:00
Nhat Nguyen 93d419b9c8 Mute CcrRollingUpgradeIT
Tracked at #59625
2020-07-15 14:43:32 -04:00
Martijn van Groningen 53249dcca8
No need to select only < 7.9 nodes in 7.x branch. (#59609) 2020-07-15 15:23:16 +02:00
Albert Zaharovits 6d6d565eeb
Fix auditing of nameless API Keys (#59531)
API keys can be created nameless using the grant endpoint (it is a bug, see #59484).
This change ensures auditing doesn't throw when such an API Key is used for authn.
2020-07-14 23:46:25 +03:00
Albert Zaharovits 4eb310c777
Disallow mapping updates for doc ingestion privileges (#58784)
The `create_doc`, `create`, `write` and `index` privileges do not grant
the PutMapping action anymore. Apart from the `write` privilege, the other
three privileges also do NOT grant (auto) updating the mapping when ingesting
a document with unmapped fields, according to the templates.

In order to maintain the BWC in the 7.x releases, the above privileges will still grant
the Put and AutoPutMapping actions, but only when the "index" entity is an alias
or a concrete index, but not a data stream or a backing index of a data stream.
2020-07-14 23:39:41 +03:00
Andrei Dan d477aa14ef
Data Streams: fix bwc test (#59528) (#59534)
(cherry picked from commit ed1a5c00abed8c63ad395ea93df7a303da7b7a65)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
2020-07-14 15:17:20 +01:00
Andrei Dan 7dcdaeae49
Default to @timestamp in composable template datastream definition (#59317) (#59516)
This makes the data_stream timestamp field specification optional when
defining a composable template.
When there isn't one specified it will default to `@timestamp`.

(cherry picked from commit 5609353c5d164e15a636c22019c9c17fa98aac30)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
2020-07-14 12:36:54 +01:00
Rene Groeschke d5c11479da
Remove remaining deprecated api usages (#59231) (#59498)
- Fix duplicate path deprecation by removing duplicate test resources
- fix deprecated non annotated input property in LazyPropertyList
- fix deprecated usage of AbstractArchiveTask.version
- Resolve correct test resources
2020-07-14 10:25:00 +02:00
Martijn van Groningen b1b7bf3912
Make data streams a basic licensed feature. (#59392)
Backport of #59293 to 7.x branch.

* Create new data-stream xpack module.
* Move TimestampFieldMapper to the new module,
  this results in storing a composable index template
  with data stream definition only to work with default
  distribution. This way data streams can only be used
  with default distribution, since a data stream can
  currently only be created if a matching composable index
  template exists with a data stream definition.
* Renamed `_timestamp` meta field mapper
   to `_data_stream_timestamp` meta field mapper.
* Add logic to put composable index template api
  to fail if `_data_stream_timestamp` meta field mapper
  isn't registered. So that a more understandable
  error is returned when attempting to store a template
  with data stream definition via the oss distribution.

In a follow up the data stream transport and
rest actions can be moved to the xpack data-stream module.
2020-07-13 17:26:46 +02:00
Yang Wang cc9166a5ea Mute failed 120_api_key_auth test till #59425 is addressed. 2020-07-14 01:10:36 +10:00
Yang Wang edf27cd765 Adjust BWC versions for API key auth test.
API key realm name is not available in authentication metadata prior to
v7.5. The issue is tracked at #59425
2020-07-14 00:38:42 +10:00
Yang Wang a84469742c
Improve role cache efficiency for API key roles (#58156) (#59397)
This PR ensure that same roles are cached only once even when they are from different API keys.
API key role descriptors and limited role descriptors are now saved in Authentication#metadata
as raw bytes instead of deserialised Map<String, Object>.
Hashes of these bytes are used as keys for API key roles. Only when the required role is not found
in the cache, they will be deserialised to build the RoleDescriptors. The deserialisation is directly
from raw bytes to RoleDescriptors without going through the current detour of
"bytes -> Map -> bytes -> RoleDescriptors".
2020-07-13 22:58:11 +10:00
Lee Hinman bb1c53a0f5
Allow warnings about 'global' template in upgrade tests (#59242)
These tests sometimes install a template so they can be compatible with older versions, but they run
amok of the occasionally installed "global" template which changes the default number of shards.

This commit adds `allowedWarnings` and allows these warnings to be present, but doesn't fail if they
are not (since the global template is only randomly installed).

Resolves #58807
Resolves #58258
2020-07-08 13:40:55 -06:00
Jake Landis 604c6dd528
7.x - Create plugin for yamlTest task (#56841) (#59090)
This commit creates a new Gradle plugin to provide a separate task name
and source set for running YAML based REST tests. The only project
converted to use the new plugin in this PR is distribution/archives/integ-test-zip.
For which the testing has been moved to :rest-api-spec since it makes the most
sense and it avoids a small but awkward change to the distribution plugin.

The remaining cases in modules, plugins, and x-pack will be handled in followups.

This plugin is distinctly different from the plugin introduced in #55896 since
the YAML REST tests are intended to be black box tests over HTTP. As such they
should not (by default) have access to the classpath for that which they are testing.

The YAML based REST tests will be moved to separate source sets (yamlRestTest).
The which source is the target for the test resources is dependent on if this
new plugin is applied. If it is not applied, it will default to the test source
set.

Further, this introduces a breaking change for plugin developers that
use the YAML testing framework. They will now need to either use the new source set
and matching task, or configure the rest resources to use the old "test" source set that
matches the old integTest task. (The former should be preferred).

As part of this change (which is also breaking for plugin developers) the
rest resources plugin has been removed from the build plugin and now requires
either explicit application or application via the new YAML REST test plugin.

Plugin developers should be able to fix the breaking changes to the YAML tests
by adding apply plugin: 'elasticsearch.yaml-rest-test' and moving the YAML tests
under a yamlRestTest folder (instead of test)
2020-07-06 14:16:26 -05:00
Luca Cavanna 4f86f6fb38 Submit async search to not require read privilege (#58942)
When we execute search against remote indices, the remote indices are authorized on the remote cluster and not on the CCS cluster. When we introduced submit async search we added a check that requires that the user running it has the privilege to execute it on some index. That prevents users from executing async searches against remote indices unless they also have read access on the CCS cluster, which is common when the CCS cluster holds no data.

The solution is to let the submit async search go through as we already do for get and delete async search. Note that the inner search action will still check that the user can access local indices, and remote indices on the remote cluster, like search always does.
2020-07-03 12:18:07 +02:00
Lee Hinman d3d03fc1c6
[7.x] Add default composable templates for new indexing strategy (#57629) (#58757)
Backports the following commits to 7.x:

    Add default composable templates for new indexing strategy (#57629)
2020-07-01 09:32:32 -06:00
Tim Vernum 9e49af03b7
Reenable test after backport (#58717)
This commit re-enables CCR rolling upgrade tests following the
backport of #58217 to 7.8 branch (7.8.1)
2020-07-01 11:50:30 +10:00
Rene Groeschke d952b101e6
Replace compile configuration usage with api (7.x backport) (#58721)
* Replace compile configuration usage with api (#58451)

- Use java-library instead of plugin to allow api configuration usage
- Remove explicit references to runtime configurations in dependency declarations
- Make test runtime classpath input for testing convention
  - required as java library will by default not have build jar file
  - jar file is now explicit input of the task and gradle will ensure its properly build

* Fix compile usages in 7.x branch
2020-06-30 15:57:41 +02:00