Commit Graph

773 Commits

Author SHA1 Message Date
David Roberts 1b32daf37b Mute FullClusterRestartIT.testWatcherWithApiKey (#63241)
Due to https://github.com/elastic/elasticsearch/issues/63088
2020-10-05 15:03:42 +01:00
Rene Groeschke f58ebe58ee
Use services for archive and file operations in tasks (#62968) (#63201)
Referencing a project instance during task execution is discouraged by
Gradle and should be avoided. E.g. It is incompatible with Gradles
incubating configuration cache. Instead there are services available to handle
archive and filesystem operations in task actions.

Brings us one step closer to #57918
2020-10-05 15:52:15 +02:00
David Roberts ccaec70a84
[ML] Muting mappings upgrade test for .ml-stats (#63234)
Due to https://github.com/elastic/elasticsearch/issues/61908
2020-10-05 13:22:13 +01:00
Joe Gallo d172a18c95 Tidy up some ILM and SLM packages (#63146)
Very minor refactoring, just moving some ILM and SLM classes around to decrease
the total number of packages.
2020-10-02 09:30:24 -04:00
Ioannis Kakavas e91f66e22f
Ensure domain_name setting for AD realm is present (#61983) (#63159)
We would only check for a null value and not for an empty string so
that meant that we were not actually enforcing this mandatory
setting. This commits ensures we check for both and fail 
accordingly if necessary, on startup
2020-10-02 12:16:08 +03:00
Yang Wang e31bef4032
Fix API key role descriptors rewrite bug for upgraded clusters (#62917) (#63042)
This PR ensures that API key role descriptors are always rewritten to a target node
compatible format before a request is sent.
2020-09-30 22:16:39 +10:00
Benjamin Trent b7c47b1717
[ML] Add data frame analytics bwc testing (#63012)
This commit adds bwc testing for data frame analytics.

The bwc tests only go back to the 7.9.0.
Meaning, initially only rolling upgrades from 7.9.x -> 7.10.0 are tested.

Since the feature was experimental in < 7.9.0, this is acceptable.
2020-09-30 07:13:40 -04:00
David Kyle bc34ecc581
[ML] Mute annotations index upgrade mapping test (#62814)
For #61908
2020-09-23 09:37:04 +01:00
David Kyle 31fbc6800f
[7.x] [ML] Add upgrade mappings assertions to full cluster restart tests (#62293) (#62305)
Refactors the index mapping checks in the rolling upgrade tests
and use that shared code in the full cluster restart tests.
2020-09-22 13:09:51 +01:00
Yang Wang 28503f04f7
Fix privilege requirement for CCS with Point In Time reader (#62261) (#62696)
When target indices are remote only, CCS does not require user to have privileges on the local cluster. This PR ensure Point-In-Time reader follows the same pattern.

Relates: #61827
2020-09-22 12:51:51 +10:00
Yang Wang 897d2e8a02
Fix ccs permission for search with a scroll id (#62053) (#62695)
CCS with remote indices only does not require any privileges on the local cluster.
This PR ensures that search with scroll follow the permission model.
2020-09-22 11:49:40 +10:00
Jake Landis 5b7246157f
[7.x] Fix projects that failed to build within Intellij (#62258) (#62408)
This commit address some build failures from the perspective of Intellij.
These changes include:
* changing an order of a dependency definition that seems to can cause Intellij build to fail.
* introduction of an abstract class out of the test source set (seems to be an issue sharing 
  classes cross projects with non-standard source sets. 
* a couple of missing dependency definitions (not sure how the command line worked prior to this)
2020-09-17 17:45:12 -05:00
Yang Wang a11dfbe031
Oidc additional client auth types (#58708) (#62289)
The OpenID Connect specification defines a number of ways for a
client (RP) to authenticate itself to the OP when accessing the
Token Endpoint. We currently only support `client_secret_basic`.

This change introduces support for 2 additional authentication
methods, namely `client_secret_post` (where the client credentials
are passed in the body of the POST request to the OP) and
`client_secret_jwt` where the client constructs a JWT and signs
it using the the client secret as a key.

Support for the above, and especially `client_secret_jwt` in our
integration tests meant that the OP we use ( Connect2id server )
should be able to validate the JWT that we send it from the RP.
Since we run the OP in docker and it listens on an ephemeral port
we would have no way of knowing the port so that we can configure
the ES running via the testcluster to know the "correct" Token
Endpoint, and even if we did, this would not be the Token Endpoint
URL that the OP would think it listens on. To alleviate this, we
run an ES single node cluster in docker, alongside the OP so that
we can configured it with the correct hostname and port within
the docker network.

Co-authored-by: Ioannis Kakavas <ioannis@elastic.co>
2020-09-16 14:29:09 +10:00
Martijn van Groningen 3ed60df59d
Re-enable resolve index multi cluster test (#62365)
Backport of #62361 to 7.x branch.

This test was fine and shouldn't have been muted.
The test case class should have preserved data streams as part of #62205

Closes #62210
2020-09-15 13:26:52 +02:00
David Kyle 717259a049 Revert "[ML] Add debug logging of notification messages to upgrade test (#62342)"
This reverts commit c50899dd8f.
2020-09-15 11:07:23 +01:00
David Kyle c50899dd8f
[ML] Add debug logging of notification messages to upgrade test (#62342)
For #61908
2020-09-15 08:24:13 +01:00
David Roberts 3d5c13f559
[ML] Add an assertion on annotations mappings to upgrade test (#62331)
The annotations index is not covered by the comparison between
mappings and templates, as it does not use an index template.

This commit adds an assertion on annotations index mappings
that will fail if the mappings are not upgraded as expected.

Backport of #62325
2020-09-14 18:46:35 +01:00
Nhat Nguyen aafb2cb812 Support point in time cross cluster search (#61827)
This commit integrates point in time into cross cluster search.

Relates #61062
Closes #61790
2020-09-10 19:25:48 -04:00
Martijn van Groningen 81b89fe3ba
Change yaml test suite testcase to automatically delete all data streams after each yaml test (#62214)
Backporting #62205 to 7.x branch.

This is similar to what happens for indices. Initially we decided to let each test cleanup the
data streams it created.

The reason behind this was that client yaml test runners would need to be modified to do this too and
because data steams were new, we waited with that and let each test cleanup the data stream it created.
However we sometimes have very hard to debug test failures, because many tests fail because another test
failed mid way and didn't clean up the data streams it created. Given that and data streams exist in
the code base for a while now, we should automatically delete all data streams after each yaml test.

Relates to #62190

* preserve data streams for rolling upgrade yaml tests
2020-09-10 15:10:57 +02:00
Martijn van Groningen f87fc67592
Mute resolve index data stream tests. (#62211)
Relates to #62190 and #62210
2020-09-10 13:13:50 +02:00
Hendrik Muhs ab259d37a7 mute mapping upgrade check for the .ml-notifications, as a global AwaitsFix
would mute everything I allow myself to use a comment
2020-09-10 12:24:57 +02:00
Jake Landis d8dad9ab2c
[7.x] Remove integTest task from PluginBuildPlugin (#61879) (#62135)
This commit removes `integTest` task from all es-plugins.  
Most relevant projects have been converted to use yamlRestTest, javaRestTest, 
or internalClusterTest in prior PRs. 

A few projects needed to be adjusted to allow complete removal of this task
* x-pack/plugin - converted to use yamlRestTest and javaRestTest 
* plugins/repository-hdfs - kept the integTest task, but use `rest-test` plugin to define the task
* qa/die-with-dignity - convert to javaRestTest
* x-pack/qa/security-example-spi-extension - convert to javaRestTest
* multiple projects - remove the integTest.enabled = false (yay!)

related: #61802
related: #60630
related: #59444
related: #59089
related: #56841
related: #59939
related: #55896
2020-09-09 14:25:41 -05:00
David Kyle a4fb501a33
[ML] Extra exceptions for renamed fields in mapping tests (#62151)
Add renamed 'maximum_number_trees' fields to exceptions
2020-09-09 12:03:42 +01:00
Dimitris Athanasiou 14547b14fb
[ML] Add changes stats fields as exception in mappings upgrade test (#62121)
Following #61980 we need to exclude the replaced fields in the test
in the 7.x branch.
2020-09-09 10:09:35 +03:00
Yang Wang 18a08c0cf2
Mute MlMappingsUpgradeIT testMappingsUpgrade (#61909) (#62136)
For #61908

Co-authored-by: David Kyle <david.kyle@elastic.co>
2020-09-09 13:23:36 +10:00
David Kyle fb6ee5b36d
[7.x] [ML] Assert mappings match templates in Upgrade tests (#61905)
At the end of the rolling upgrade tests check the mappings of the concrete
.ml and .transform-internal indices match the mappings in the templates.
When the templates change, the tests should prove that the mappings have
been updated in the new cluster.
2020-09-08 12:21:19 +01:00
Przemko Robakowski bb357f6aae
[7.x] Move internal index templates to composable templates (#61457) (#61661)
This change moves watcher, ILM history and SLM history templates to composable templates.
Versions are updated to reflect the switch. Only change to the templates themselves is added `_meta` to mark them as managed
2020-09-08 11:26:06 +02:00
Ryan Ernst d6e17170c3
Simplify adding plugins and modules to testclusters (#61886)
There are currently half a dozen ways to add plugins and modules for
test clusters to use. All of them require the calling project to peek
into the plugin or module they want to use to grab its bundlePlugin
task, and then both depend on that task, as well as extract the archive
path the task will produce. This creates cross project dependencies that
are difficult to detect, and if the dependent plugin/module has not yet
been configured, the build will fail because the task does not yet
exist.

This commit makes the plugin and module methods for testclusters
symmetetric, and simply adding a file provider directly, or a project
path that will produce the plugin/module zip. Internally this new
variant uses normal configuration/dependencies across projects to get
the zip artifact. It also has the added benefit of no longer needing the
caller to add to the test task a dependsOn for bundlePlugin task.
2020-09-03 19:37:46 -07:00
David Roberts 89599ba0a3
[ML] Update ML mappings upgrade test and extend to config index (#61830)
The ML mappings upgrade test had become useless as it was
checking a field that has been the same since 6.5. This
commit switches to a field that was changed in 7.9.

Additionally, the test only used to check the results index
mappings.  This commit also adds checking for the config
index.

Backport of #61340
2020-09-02 12:23:59 +01:00
Ioannis Kakavas 3640ff1ff2
Add SAML AuthN request signing tests (#61582)
- Add a unit test for our signing code
- Change SAML IT to use signed authentication requests for Shibboleth to consume

Backport of #48444
2020-08-27 10:41:56 +03:00
Ioannis Kakavas 283eaabc71
[7.x] Refactor SamlAuthenticationIT (#57162) (#61568)
Refactor the tests to not require a mock HTTP Server. This has been
the cause of flakiness and removing it doesn't affect the logical
coverage of this suite. The "fake UI" is now simulated by an
http client that makes the necessary requests to Elasticsearch APIs.
2020-08-26 15:34:56 +03:00
David Roberts d1b60269f4
[ML] Ensure annotations index mappings are up to date (#61142)
When the ML annotations index was first added, only the
ML UI wrote to it, so the code to create it was designed
with this in mind.  Now the ML backend also creates
annotations, and those mappings can change between
versions.

In this change:

1. The code that runs on the master node to create the
   annotations index if it doesn't exist but another ML
   index does also now ensures the mappings are up-to-date.
   This is good enough for the ML UI's use of the
   annotations index, because the upgrade order rules say
   that the whole Elasticsearch cluster must be upgraded
   prior to Kibana, so the master node should be on the
   newer version before Kibana tries to write an
   annotation with the new fields.
2. We now also check whether the annotations index exists
   with the correct mappings before starting an autodetect
   process on a node.  This is necessary because ML nodes
   can be upgraded before the master node, so could write
   an annotation with the new fields before the master node
   knows about the new fields.

Backport of #61107
2020-08-14 13:51:04 +01:00
Dan Hermann c17839c255
Add warning handler to resolve test failure (#60427) (#61102) 2020-08-13 10:37:08 -05:00
Yang Wang c7b0290256
Mute kerberos tests for jdk 8u[262,271) (#60995)
The Kerberos bug (JDK-8246193) is introduced in JDK 8u262 and fixed in 8u271.
This PR mute for any possible releases between these two versions.
2020-08-12 11:50:48 +10:00
Rene Groeschke bdd7347bbf
Merge test runner task into RestIntegTest (7.x backport) (#60600)
* Merge test runner task into RestIntegTest (#60261)
* Merge test runner task into RestIntegTest
* Reorganizing Standalone runner and RestIntegTest task
* Rework general test task configuration and extension
* Fix merge issues
* use former 7.x common test configuration
2020-08-04 14:46:32 +02:00
Jake Landis bcb9d06bb6
[7.x] Cleanup xpack build.gradle (#60554) (#60603)
This commit does three things:
* Removes all Copyright/license headers for the build.gradle files under x-pack. (implicit Apache license)
* Removes evaluationDependsOn(xpackModule('core')) from build.gradle files under x-pack
* Removes a place holder test in favor of disabling the test task (in the async plugin)
2020-08-03 13:11:43 -05:00
Rene Groeschke ed4b70190b
Replace immediate task creations by using task avoidance api (#60071) (#60504)
- Replace immediate task creations by using task avoidance api
- One step closer to #56610
- Still many tasks are created during configuration phase. Tackled in separate steps
2020-07-31 13:09:04 +02:00
Yang Wang 3a0e7f4294
Unmute kerberos tests for jdk 15 and mute for jdk 8u262 (#60279)
The JDK bug (https://bugs.openjdk.java.net/browse/JDK-8246193) is fixed since b26.
The tests can be unmuted since we are already using b33. However the same bug is now
affecting jdk 8u262, which is the base for current Zulu jdk 8.48. This PR mute the tests
for this specific jdk version.

Relates: #56507
2020-07-29 12:57:00 +10:00
Zachary Tong 46f9c38c33 Mute tests while waiting on 58807
Bugurl: https://github.com/elastic/elasticsearch/issues/58807
2020-07-28 12:45:49 -04:00
Dan Hermann fe12217c7f
[7.x] Move REST specs for data streams (#60111) 2020-07-23 08:10:54 -05:00
Rene Groeschke e31ebc96f9
Enforce fail on deprecated gradle usage (7.x backport) (#59758)
* Enforce fail on deprecated gradle usage (#59598)
* Fix branch specific deprecated gradle api usages
* Fix archiveVersion property usage
2020-07-20 08:52:30 +02:00
Martijn van Groningen 74c9402912
Re-enable data stream bwc tests (#59734)
after backporting #59503
Backport of #59732 yo 7.x
2020-07-16 23:59:52 +02:00
Martijn van Groningen 0096238df1
Replaced _data_stream_timestamp meta field's 'path' option with 'enabled' option (#59727)
Backport #59503 to 7.x

and adjusted exception messages.

Relates to #59076
2020-07-16 22:29:40 +02:00
Martijn van Groningen 4089cbd767
Ignore multiple matching templates warning in specific tests. (#59692) (#59715)
Closes #59679
2020-07-16 20:07:38 +02:00
Yang Wang 067db1fc3b
Fix test of API key creation in a mixed cluster (#59680)
RoleDescriptors are mandatory prior to v7.3

Relates: #59425
2020-07-16 12:44:17 +10:00
Lee Hinman 74372df824
Mute {p0=mixed_cluster/120_api_key_auth/Test API key authentication will work in a mixed cluster} (#59663)
Relates to #59425
2020-07-15 14:14:33 -06:00
Nhat Nguyen 93d419b9c8 Mute CcrRollingUpgradeIT
Tracked at #59625
2020-07-15 14:43:32 -04:00
Martijn van Groningen 53249dcca8
No need to select only < 7.9 nodes in 7.x branch. (#59609) 2020-07-15 15:23:16 +02:00
Albert Zaharovits 6d6d565eeb
Fix auditing of nameless API Keys (#59531)
API keys can be created nameless using the grant endpoint (it is a bug, see #59484).
This change ensures auditing doesn't throw when such an API Key is used for authn.
2020-07-14 23:46:25 +03:00
Albert Zaharovits 4eb310c777
Disallow mapping updates for doc ingestion privileges (#58784)
The `create_doc`, `create`, `write` and `index` privileges do not grant
the PutMapping action anymore. Apart from the `write` privilege, the other
three privileges also do NOT grant (auto) updating the mapping when ingesting
a document with unmapped fields, according to the templates.

In order to maintain the BWC in the 7.x releases, the above privileges will still grant
the Put and AutoPutMapping actions, but only when the "index" entity is an alias
or a concrete index, but not a data stream or a backing index of a data stream.
2020-07-14 23:39:41 +03:00