Commit Graph

543 Commits

Author SHA1 Message Date
Albert Zaharovits 026729e911 TOKEN_SERVICE_ENABLED_SETTING enabled if HTTP_SSL_ENABLED (elastic/x-pack-elasticsearch#2321)
`authc.token.enabled` is true unless `http.ssl.enabled` is `false` and `http.enabled` is `true`.

* TokenService default enabled if HTTP_ENABLED == false

* Fixed tests that need TokenService explicitly enabled

* [DOC] Default value for `xpack.security.authc.token.enabled`

Original commit: elastic/x-pack-elasticsearch@bd154d16eb
2017-08-23 13:21:30 +03:00
Alexander Reelsen 3f541fa556 Tests: Ensure watcher is started via awaitBusy in bwc test
Original commit: elastic/x-pack-elasticsearch@a8c0cf04c9
2017-08-22 00:39:11 +02:00
Alexander Reelsen 17980ab360 Tests: Remove randomized unsupported code upgrading two indices at once
Original commit: elastic/x-pack-elasticsearch@c86b87927d
2017-08-22 00:09:01 +02:00
Alexander Reelsen 27f39c615b Watcher: Create two index ugprade checks for watcher upgrade (elastic/x-pack-elasticsearch#2298)
As there are two indices to upgrade for watcher, it makes a lot of sense
to also have two upgrade checks.

There is one upgrader for the watches index, which deletes
old templates, adds the new one before and then does the reindexing.
Same for the triggered watches index.

This also means, that there will be two entries popping up in the kibana
UI.

Note: Each upgrade check checks if the other index (for the .watches
upgrade check the triggered watches index and vice versa) is already
upgraded and only if that is true, watcher is restarted.

relates elastic/x-pack-elasticsearch#2238

Original commit: elastic/x-pack-elasticsearch@2c92040ed6
2017-08-21 17:36:16 +02:00
Igor Motov a27c726f72 Merge remote-tracking branch 'elastic/master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@ecbfa82bcf
2017-08-18 15:40:49 -04:00
Simon Willnauer 71827b70a0 Bump token service BWC version to 6.0.0-beta2
Original commit: elastic/x-pack-elasticsearch@ef688f02cb
2017-08-18 17:06:45 +02:00
Nik Everett a68d839edc Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@1f74db9cde
2017-08-18 09:56:29 -04:00
Simon Willnauer ac9ab974f4 Ensure token service can boostrap itself without a pre-shared key (elastic/x-pack-elasticsearch#2240)
Today we require a pre-shared key to use the token service. Beside the
additional setup step it doesn't allow for key-rotation which is a major downside.

This change adds a TokenService private ClusterState.Custom that is used to distribute
the keys used to encrypt tokens. It also has the infrastructur to add automatic key
rotation which is not in use yet but included here to illustrate how it can work down
the road.

This is considered a prototype and requires additioanl integration testing. Yet, it's fully
BWC with a rolling / full cluster restart from a previous version (also from 5.6 to 6.x)
since if the password is set it will just use it instead of generating a new one.
Once we implement the automatic key rotation via the clusterstate we need to ensure that we are
fully upgraded before we do that.
Also note that the ClusterState.Custom is fully transient and will never be serialized to disk.

Original commit: elastic/x-pack-elasticsearch@1ae22f5d41
2017-08-18 14:23:43 +02:00
Igor Motov 09579eb630 Merge remote-tracking branch 'elastic/master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@87d023325e
2017-08-17 15:47:52 -04:00
Simon Willnauer 724325f161 Fallback to `keystore.seed` as a bootstrap password if actual password is not present (elastic/x-pack-elasticsearch#2295)
Today we require the `bootstrap.password` to be present in the keystore in order to
bootstrap xpack. With the addition of `keystore.seed` we have a randomly generated password
per node to do the bootstrapping. This will improve the initial user experience significantly
since the user doesn't need to create a keystore and add a password, they keystore is created
automatically unless already present and is always created with this random seed.

Relates to elastic/elasticsearch#26253

Original commit: elastic/x-pack-elasticsearch@5a984b4fd8
2017-08-17 16:42:32 +02:00
Alexander Reelsen 6d30806996 Watcher: Improvements on the rolling restart tests (elastic/x-pack-elasticsearch#2286)
This improves the rolling restart tests (tests different paths in
different ways) and aligns the upgrade code with the 5.6 branch from

Relates elastic/x-pack-elasticsearch#2238

Original commit: elastic/x-pack-elasticsearch@01b0954558
2017-08-17 11:41:11 +02:00
Alexander Reelsen 0a86f00d7e Tests: Ensure responses are closed in watch backwards compat tests
The HTTP client has to have it's response entities closed, otherwise
it might block further requests because the underlying connection cannot
be reused.

Relates elastic/x-pack-elasticsearch#2004

Original commit: elastic/x-pack-elasticsearch@a24ecb9764
2017-08-16 16:29:52 +02:00
Nik Everett 335838db08 Audit logging for SQL (elastic/x-pack-elasticsearch#2210)
Adapts audit logging to actions that delay getting index access control until the action is started. The audit log will contain an entry for the action itself starting without any associated indices because the indices are not yet known. The audit log will also contain an entry for every time the action resolved security for a set of indices. Since sql resolves indices one at a time it will contain an entry per index.

All of this customization is entirely in the security code. The only SQL change in this PR is to add audit logging support to the integration test.

Original commit: elastic/x-pack-elasticsearch@539bb3c2a8
2017-08-15 14:19:28 -04:00
Nik Everett 02cc23cf21 Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@aaad327bd0
2017-08-15 13:05:24 -04:00
Alexander Reelsen 69b3ffa40a Tests: Remove useless wait time in watcher REST test
The rest test waited for the watch to run in the background, but there
were no guarantees that this really happened. Also it waited for five
seconds, instead of just executing the watch manually.

relates elastic/x-pack-elasticsearch#2255

Original commit: elastic/x-pack-elasticsearch@56765a649e
2017-08-14 11:52:26 +02:00
Nik Everett 2207c19afa Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@f9c0bc95d4
2017-08-11 15:41:22 -04:00
Alexander Reelsen 3ad2f5e9f5 Tests: Increase logging for watcher smoke tests
These tests have repeating but not reproducible failures,
where the stash is filled with a second PUT operation and the
watcher stats response does not match. Setting the log to trace
should shed some light on this.

As the smoke tests are only four tests this will not lead to a
log explosion.

Relates elastic/x-pack-elasticsearch#1513, elastic/x-pack-elasticsearch#1874

Original commit: elastic/x-pack-elasticsearch@5832dc7990
2017-08-11 15:29:52 +02:00
Alexander Reelsen 26c5766a0d Tests: Do not delete index templates on bwc tests
In order to run the bwc tests there is no need to delete the
index template at the end of a test run which results
in recreation of those due to all the cluster state listener.

Relates elastic/x-pack-elasticsearch#2228

Original commit: elastic/x-pack-elasticsearch@702d1c61ed
2017-08-11 14:11:50 +02:00
Nik Everett 2b2f831116 Fix broke bwc test in sql
The test was enabled for the wrong versions....

Original commit: elastic/x-pack-elasticsearch@2662a11e0c
2017-08-09 15:58:47 -04:00
Nik Everett c47c66362e Fix content type in test
I'd left it off. Thanks Jenkins!

Original commit: elastic/x-pack-elasticsearch@57b708e414
2017-08-08 15:19:20 -04:00
Nik Everett 570b66638e Add test for index with two types (elastic/x-pack-elasticsearch#2194)
Adds a test that shows *how* SQL fails to address an index with two types
to the full cluster restart tests. Because we're writing this code
against 7.0 don't actually execute the test, but we will execute it when
we merge to 6.x and it *should* work.

Original commit: elastic/x-pack-elasticsearch@b536e9a142
2017-08-08 13:32:13 -04:00
Nik Everett bcd9934050 Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@e5809f0785
2017-08-07 11:09:17 -04:00
Alexander Reelsen 2d08477093 Tests: Fix upgrade tests by not running stats against arbitrary hosts
Original commit: elastic/x-pack-elasticsearch@1b858aad52
2017-08-07 13:27:36 +02:00
Alexander Reelsen 8f6874abf9 Tests: Increase logging in watcher upgrade tests to debug test failures
Original commit: elastic/x-pack-elasticsearch@380a8541dc
2017-08-07 11:31:16 +02:00
David Roberts 05cbe8dc0c [ML] Disallow creating a job against a closed results or state index (elastic/x-pack-elasticsearch#2186)
Previously if this was attempted you'd get an NPE (5.x) or hang (6.x).
Following this change you get an error message telling you what the
problem is.

relates elastic/x-pack-elasticsearch#2170

Original commit: elastic/x-pack-elasticsearch@ea12a9ff46
2017-08-07 08:53:12 +01:00
Nik Everett de9adfde81 Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@5464f9839f
2017-08-04 16:59:32 -04:00
Nik Everett f241512e33 SQL support for field level security (elastic/x-pack-elasticsearch#2162)
This adds support for field level security to SQL by creating a new type of flow for securing requests that look like sql requests. `AuthorizationService` verifies that the user can execute the request but doesn't check the indices in the request because they are not yet ready. Instead, it adds a `BiFunction` to the context that can be used to check permissions for an index while servicing the request. This allows requests to cooperatively secure themselves. SQL does this by implementing filtering on top of its `Catalog` abstraction and backing that filtering with security's filters. This minimizes the touch points between security and SQL.

Stuff I'd like to do in followups:

What doesn't work at all still:
1. `SHOW TABLES` is still totally unsecured
2. `DESCRIBE TABLE` is still totally unsecured
3. JDBC's metadata APIs are still totally unsecured

What kind of works but not well:
1. The audit trail doesn't show the index being authorized for SQL.

Original commit: elastic/x-pack-elasticsearch@86f88ba2f5
2017-08-04 15:27:27 -04:00
Jay Modi 8b0fb5eae8 Re-enable OpenLDAP tests and run against vagrant instance (elastic/x-pack-elasticsearch#2121)
This commit re-enables the OpenLDAP tests that were previously running against a one-off instance
in AWS but now run against a vagrant fixture. There were some IntegTests that would run against the
OpenLDAP instance randomly but with this change they no longer run against OpenLDAP. This is ok as
the functionality that is tested by these has coverage elsewhere.

relates elastic/x-pack-elasticsearch#1823

Original commit: elastic/x-pack-elasticsearch@ac9bc82297
2017-08-04 09:44:08 -06:00
Nik Everett 35389d3be0 Fix one more test after dropping type awareness
Original commit: elastic/x-pack-elasticsearch@ab6949f353
2017-08-03 17:39:24 -04:00
Nik Everett 4f42de6b1a Fix as many busted tests as I can
Original commit: elastic/x-pack-elasticsearch@5ec24f6818
2017-08-03 17:27:56 -04:00
Jay Modi a7d6138f83 Fix the building of the default URL for the setup password tool (elastic/x-pack-elasticsearch#2176)
This commit fixes the building of the default URL for the setup password tool so that a default
elasticsearch.yml file will still result in a succesful run of the tool.

relates elastic/x-pack-elasticsearch#2174

Original commit: elastic/x-pack-elasticsearch@2291b14875
2017-08-03 15:14:24 -06:00
Nik Everett 0605802d22 Add a multi-node test to sql (elastic/x-pack-elasticsearch#2136)
SQL relies on being able to fetch information about fields from
the cluster state and it'd be disasterous if that information
wasn't available. This should catch that.

Original commit: elastic/x-pack-elasticsearch@1a62747332
2017-08-02 14:39:25 -04:00
Nik Everett 767a43ca44 Move sql rest test into qa (elastic/x-pack-elasticsearch#2149)
Running the sql rest action test inside the server caused a dependency
loop which was failing the build.

Original commit: elastic/x-pack-elasticsearch@43283671d8
2017-08-01 17:23:07 -04:00
Nik Everett a9b72019ad Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@b45f682d72
2017-08-01 16:17:17 -04:00
Ryan Ernst 45a55d16cf Bump master version to 7.0.0-alpha1 (elastic/x-pack-elasticsearch#2135)
This is the xpack side of
https://github.com/elastic/elasticsearch/pull/25876

Original commit: elastic/x-pack-elasticsearch@c86ea25009
2017-08-01 15:48:04 -04:00
Alexander Reelsen 4bf5d9536a Tests: Remove @ClusterScope tests, create REST tests (elastic/x-pack-elasticsearch#2131)
Replacing integration tests with rest tests and unit tests, thus removing integration tests that require start of a new cluster. Removing unused testing methods

Original commit: elastic/x-pack-elasticsearch@265966d80c
2017-08-01 14:15:36 +02:00
Jason Tedor 5b1bf9a31e Remove max script compilation settings in tests
Some standalone tests set the max script compilation limit. However,
this setting is now set in the main cluster setup in core so it is no
longer needed here. This commit removes these obviated settings in
standalone tests.

Original commit: elastic/x-pack-elasticsearch@089328c8d7
2017-08-01 14:57:57 +09:00
Jason Tedor 50b8a56d34 Add packaging assertions for new scripts
This commit adds packaging assertions for recent script additions:
setup-passwords and x-pack-env.

Original commit: elastic/x-pack-elasticsearch@d83624f419
2017-07-30 09:32:39 +09:00
Jay Modi db4c00b565 Update the full cluster restart tests to be more generic (elastic/x-pack-elasticsearch#2107)
The full cluster restart tests are currently geared towards the 5.6 -> 6.0 upgrade and have some
issues when the versions are changed to 6.x -> 7.0. One issue is a real code issue in that the
security code always expects the mappings to have the same version as the version of the node, but
we no longer update the mappings on the security index during a rolling upgrade. We know look at
the index format to determine if the index is up to date.

Original commit: elastic/x-pack-elasticsearch@14c1c72ff6
2017-07-28 10:31:44 -06:00
Nik Everett b755b3e543 Merge branch 'master' into feature/sql
Original commit: elastic/x-pack-elasticsearch@b80d0cba1d
2017-07-28 12:06:32 -04:00
Jason Tedor 826df4eb72 Fix migrate tool integration test
The command line flag --path.conf was removed yet this test was still
using it. This commit replaces the usage of this flag in this test with
the new mechanism.

Original commit: elastic/x-pack-elasticsearch@c37050894c
2017-07-28 21:33:54 +09:00
David Kyle bb360c1ff3 Mute MigrateToolIT::testRunMigrateTool
Original commit: elastic/x-pack-elasticsearch@48cb30169a
2017-07-28 11:19:08 +01:00
Nik Everett ec36875872 Sql security take 1 (elastic/x-pack-elasticsearch#2106)
Add some basic security testing/integration.

The good news:
1. Basic security now works. Users without access to an index can't run sql queries against it. Without this change they could.
2. Document level security works! At least so far as I can tell.

The work left to do:
1. Field level security doesn't work properly. I mean, it kind of works in that the field's values don't leak but it just looks like they all have null values.
2. We will need to test scrolling.
3. I've only added tests for the rest sql action. I'll need to add tests for jdbc and the CLI as well.
4. I've only added tests for `SELECT` and have ignored stuff like `DESCRIBE` and `SHOW TABLES`.

Original commit: elastic/x-pack-elasticsearch@b9909bbda0
2017-07-27 17:40:14 -04:00
Michael Basnight 6a7e51d9c0 Use shaded rest client dependencies
This commit modifies all org.apache.http to use the shaded rest clients
org.elasticsearch.client.http packages. It also removes a few unused
licenses due to the change.

Relates elastic/elasticsearch#25780

Original commit: elastic/x-pack-elasticsearch@8605560232
2017-07-24 12:56:17 -05:00
Ali Beyad 5190a05b75 Improves error message on non-upgraded security index (elastic/x-pack-elasticsearch#2061)
This commit improves the error message in 6x if the security index has
not been upgraded, and warns the user that the native realm will not be
functional until the upgrade API is run.

Original commit: elastic/x-pack-elasticsearch@710b7634b4
2017-07-21 15:56:15 -04:00
jaymode 9e14bff948 Test: increase number of script compilations per minute to 20
This commit increases the number of script compilations per minute to 20 to account for security
and watcher upgrade scripts in addition to the scripts contained within cluster alerts.

Relates elastic/x-pack-elasticsearch#2041

Original commit: elastic/x-pack-elasticsearch@6c15cb9b89
2017-07-21 13:49:22 -06:00
Alexander Reelsen 139513fdd3 Tests: Fix WatchBackwardsCompatibilityIT temporarily
the upgrade API is lacking some functionality in a special case,
where triggered_watches exists, but .watches does not. This
deletes the triggered watches index manually until we integrated
this properly in the upgrade API to fix the tests

Original commit: elastic/x-pack-elasticsearch@e9d1b0d35d
2017-07-18 18:00:55 +02:00
Ali Beyad 37cc602aef Adds upgrade API functionality for security (elastic/x-pack-elasticsearch#2012)
This commit adds the upgrade API functionality and script for security.
It also enables previously muted tests that would fail due to the lack
of security upgrade features in testing cluster restarts and old
security index backward compatibility.

Original commit: elastic/x-pack-elasticsearch@4abe9f1263
2017-07-18 11:44:28 -04:00
Alexander Reelsen 8200b18e9f Tests: Allow to run BWC in isolation
The current tests were only able to finish successfully, if the earlier
tests were run as well, you could not run the restart test in isolation.

This commit ensures an upgrade is executed if needed.

Original commit: elastic/x-pack-elasticsearch@616ebbd6eb
2017-07-18 15:29:07 +02:00
Alexander Reelsen 443cfb94be Tests: Ensure waiting time between stopping and starting watcher (elastic/x-pack-elasticsearch#2008)
Otherwise we might run into race conditions that prevent a useful
start up again.

Those tests can be massively improved (no need to run against the real
master node), once the watcher BWC compatible stats are in.

relates elastic/x-pack-elasticsearch#2004

Original commit: elastic/x-pack-elasticsearch@52ca77809c
2017-07-18 10:19:33 +02:00
Tim Brooks 6d04eacdec Require elastic password be bootstrapped (elastic/x-pack-elasticsearch#1962)
This is related to elastic/x-pack-elasticsearch#1217. This commit requires that the elastic password
be bootstrapped for the user to be authenticated. As a result it removes
the special "setup" mode that allowed the user to be authenticated from
localhost.

Additionally, this commit updates the tests to work with this
functionality.

Original commit: elastic/x-pack-elasticsearch@d0d5d697a7
2017-07-13 19:59:50 -05:00
Igor Motov 4de6d9ebe5 Upgrade API: upgrade assistance shouldn't throw 404 on an empty cluster (elastic/x-pack-elasticsearch#1997)
When a user asks for upgrade information for all indices and there are no indices in the cluster, upgrade assistance should just return an empty response indicating that no indices require upgrade or reindexing. This commit also reverts the temporary fix in WatchBackwardsCompatibilityIT tests that was added as a workaround for this issue.

Original commit: elastic/x-pack-elasticsearch@2ea9707867
2017-07-13 17:01:50 -04:00
Alexander Reelsen eb118b365c Tests: Ignore 404 errors in watcher bwc tests
Original commit: elastic/x-pack-elasticsearch@8e583cf293
2017-07-13 13:41:51 +02:00
Alexander Reelsen 32bc0cd5d5 Tests: Add logging and error tracing to watcher bwc tests
Original commit: elastic/x-pack-elasticsearch@4a4b65bcb7
2017-07-13 11:08:48 +02:00
Ryan Ernst df41b8342d Convert put_template uses to put_script in rest tests
Original commit: elastic/x-pack-elasticsearch@5f5f52b955
2017-07-13 00:01:56 -07:00
Tim Vernum e4c8851a24 Convert Realm.authenticate to provide a richer result (elastic/x-pack-elasticsearch#1932)
This allows for messages to be returned, and distinguishes between 4 different results:
- I have authenticated the user
- I don't know how to authenticate that user. Try another realm.
- I tried to authenticate the user, but failed. Try another realm.
- I tried to authenticate the user, but failed. Fail the authentication attempt.

Original commit: elastic/x-pack-elasticsearch@f796949cfb
2017-07-13 14:24:08 +10:00
Simon Willnauer 84ee21ed26 Followup for elastic/elasticsearch#25658 (elastic/x-pack-elasticsearch#1984)
This is the xpack side fo elastic/elasticsearch#25658 which is mainly refactorings
of a ctor and added tests.

Original commit: elastic/x-pack-elasticsearch@d8e2a2a057
2017-07-12 22:19:34 +02:00
Alexander Reelsen e64cf23b13 Watcher: Start watcher on master node only with mixed versions (elastic/x-pack-elasticsearch#1983)
When there are data or master nodes in the cluster, that are older
than ES 6.0 alpha3, then watcher will only start on the master node.

Changed all transport actions to be master node actions, as there is
already a method to decide to run locally, which we can piggyback on.

Original commit: elastic/x-pack-elasticsearch@65cecb6d69
2017-07-12 20:58:47 +02:00
Jack Conradson 32dbfba0c2 Disallow lang to specified in requests where a stored script is used. (elastic/x-pack-elasticsearch#1949)
Requests that execute a stored script will no longer be allowed to specify the lang of the script. This information is stored in the cluster state making only an id necessary to execute against. Putting a stored script will still require a lang.

Original commit: elastic/x-pack-elasticsearch@926a7b2d86
2017-07-12 07:56:08 -07:00
Alexander Reelsen 14c12cfcde Revert "Tests: Disable all watcher BWC tests until fixed correctly"
This reverts commit elastic/x-pack-elasticsearch@8043ec1858.

Original commit: elastic/x-pack-elasticsearch@39180b095c
2017-07-11 00:40:05 +02:00
Alexander Reelsen 4dc13c3698 Tests: Disable all watcher BWC tests until fixed correctly
Original commit: elastic/x-pack-elasticsearch@8043ec1858
2017-07-10 23:12:19 +02:00
Igor Motov 37075bd201 Upgrade API: Add end-to-end bwc test for watcher upgrade (elastic/x-pack-elasticsearch#1939)
This test creates watches in old versions of elasticsearch, upgrades them after upgrading cluster to the latest version and then tests that they were upgraded correctly.

Original commit: elastic/x-pack-elasticsearch@b9d45eb2a5
2017-07-09 11:21:57 -04:00
Tim Brooks d95c365e64 Loosen setup mode restrictions for upgrade tests (elastic/x-pack-elasticsearch#1927)
This commit is related to elastic/x-pack-elasticsearch#1896. Currently setup mode means that the
password must be set post 6.0 for using x-pack. This interferes with
upgrade tests as setting the password fails without a properly
upgraded security index.

This commit loosens two aspects of the security.

1. The old default password will be accept in setup mode (requests
from localhost).
2. All request types can be submitted in setup mode.

Original commit: elastic/x-pack-elasticsearch@8a2a577038
2017-07-06 10:37:48 -05:00
Dimitrios Athanasiou 2e0560528f [TEST] Fix MlBasicMultiNodeIT after changing flush response
Relates elastic/x-pack-elasticsearch#1914

Original commit: elastic/x-pack-elasticsearch@5175bf64d9
2017-07-05 13:30:25 +01:00
Jay Modi a9707a461d Use a secure setting for the watcher encryption key (elastic/x-pack-elasticsearch#1831)
This commit removes the system key from master and changes watcher to use a secure setting instead
for the encryption key.

Original commit: elastic/x-pack-elasticsearch@5ac95c60ef
2017-06-29 14:58:35 -06:00
Tim Brooks f2cbe20ea0 Remove default passwords from reserved users (elastic/x-pack-elasticsearch#1665)
This is related to elastic/x-pack-elasticsearch#1217. This PR removes the default password of
"changeme" from the reserved users.

This PR adds special behavior for authenticating the reserved users. No
ReservedRealm user can be authenticated until its password is set. The
one exception to this is the elastic user. The elastic user can be
authenticated with an empty password if the action is a rest request
originating from localhost. In this scenario where an elastic user is
authenticated with a default password, it will have metadata indicating
that it is in setup mode. An elastic user in setup mode is only
authorized to execute a change password request.

Original commit: elastic/x-pack-elasticsearch@e1e101a237
2017-06-29 15:27:57 -05:00
Jay Modi f60c0f893c Test: add a basic rest test for CCS with non-matching remote index patterns (elastic/x-pack-elasticsearch#1866)
This commit adds a basic rest test to verify that security works with cross cluster search when a
remote pattern is provided and no remote indices match.

Relates elastic/elasticsearch#25436
relates elastic/x-pack-elasticsearch#1854

Original commit: elastic/x-pack-elasticsearch@e804d0bb12
2017-06-29 07:14:11 -06:00
Ali Beyad a68fb27a23 Upgrade security index to use only one (the default) index type (elastic/x-pack-elasticsearch#1780)
The .security index used several different types to differentiate the
documents added to the index (users, reserved-users, roles, etc).  Since
types are deprecated in 6.x, this commit changes the .security index
access layer to only use a single type and have all documents in the
index be of that single type.  To differentiate documents that may have
the same id (e.g. the same user name and role name), the appropriate
type of the document is prepended to the id.  For example, a user named
"jdoe" will now have the document id "user-jdoe".  

This commit also ensures that any native realm security index operations
that lead to auto creation of the security index first go through the process
of creating the internal security index (.security-v6) and creating the alias
.security to point to the internal index. 

Lastly, anytime the security index is accessed without having been
upgraded, an exception is thrown notifying the user to use the
upgrade API to upgrade the security index.

Original commit: elastic/x-pack-elasticsearch@cc0a474aed
2017-06-27 17:53:58 -04:00
Ryan Ernst 9b3fb66394 Settings: Add secure versions of SSL passphrases (elastic/x-pack-elasticsearch#1852)
This commit adds new settings for the ssl keystore (not the ES keystore)
passphrase settings. New setting names are used, instead of trying to
support the existing names in both yml and the ES keystore, so that
there does not need to be complicated logic between the two. Note that
the old settings remain the only way to set the ssl passphrases for the
transport client, but the Settings object for transport clients are
created in memory by users, so they are already as "secure" as having a
loaded ES keystore. Also note that in the long term future (6.x
timeframe?) these settings should be deprecated and the keys/certs
themselves should be moved into the ES keystore, so there will be no
need for separate keystores/passphrases.

relates elastic/elasticsearch#22475

Original commit: elastic/x-pack-elasticsearch@be5275fa3d
2017-06-27 10:15:12 -07:00
Jason Tedor c22494bcb7 Remove path.conf setting
This commit is a response to a change in core removing path.conf as a
valid setting.

Relates elastic/x-pack-elasticsearch#1844

Original commit: elastic/x-pack-elasticsearch@477a7eab71
2017-06-26 15:18:49 -04:00
Alexander Reelsen efc93c7246 Tests: Increase loglevel, deactivate ML/monitoring to unclutter logs
Relates elastic/x-pack-elasticsearch#1807

Original commit: elastic/x-pack-elasticsearch@a958f32a3c
2017-06-23 10:40:31 +02:00
Alexander Reelsen 1e7f61b4c8 Tests: Ensure watcher index templates are installed in REST tests (elastic/x-pack-elasticsearch#1784)
The current testing setup only checked if watcher was started, but it
also needs to check for the index template in order to be sure that
everything is set up correctly, before trying to put a watch.

relates elastic/x-pack-elasticsearch#1762

Original commit: elastic/x-pack-elasticsearch@3ed78b15a1
2017-06-20 14:17:36 +02:00
Nik Everett 1559f85c73 Remove assemble from build task when assemble removed
Removes the `assemble` task from the `build` task when we have
removed `assemble` from the project. We removed `assemble` from
projects that aren't published so our releases will be faster. But
That broke CI because CI builds with `gradle precommit build` and,
it turns out, that `build` includes `check` and `assemble`. With
this change CI will only run `check` for projects without an
`assemble`.

Original commit: elastic/x-pack-elasticsearch@d01b0df1d9
2017-06-16 17:19:47 -04:00
Nik Everett d526461bd2 Add basic full cluster restart tests for x-pack (elastic/x-pack-elasticsearch#1743)
Adds tests similar to `:qa:full-cluster-restart` for x-pack. You
run them with `gradle :x-pack:qa:full-cluster-restart:check`.

The actual tests are as basic as it gets: create a doc and load it,
shut down, upgrade to master, startup, and load it. Create a user
and load it, shut down, upgrade to master, startup, and load it.

Relates to elastic/x-pack-elasticsearch#1629

Original commit: elastic/x-pack-elasticsearch@8994bec8e7
2017-06-16 11:44:51 -04:00
Martijn van Groningen 8cc4f29f33 test: make sure analysis-common module is also available on the cluster 1 and cluster2 nodes.
Original commit: elastic/x-pack-elasticsearch@eef5d2b566
2017-06-15 23:25:45 +02:00
jaymode ee3d17adfd Test: fix security rolling upgrade tests that were failing
This commit fixes the default password migration tests that had been failing reproducibly. The
first fix skips tests using the set enabled api when running against a version prior to 5.1.2 as
this api would otherwise trip an assertion that the xcontent builder was not closed. The second
fix is to ensure we set the password field in the user object.

relates elastic/x-pack-elasticsearch#1529
relates elastic/x-pack-elasticsearch#1516

Original commit: elastic/x-pack-elasticsearch@2f9c804309
2017-06-15 14:50:26 -06:00
Martijn van Groningen 863755d2da Make sure that the module are installed too.
Original commit: elastic/x-pack-elasticsearch@f581d0902c
2017-06-15 22:03:18 +02:00
David Roberts b748da1880 [ML] Prevent time_field and control field name in analysis_config (elastic/x-pack-elasticsearch#1729)
In does not make sense for the time_field in the data_description to
be used as a by/over/partition field name, nor the summary_count_field,
categorization_field or as an influencer.  Therefore, configurations
where the time_field in the data_description is used in the
analysis_config are now rejected.

Additionally, it causes a problem communicating with the C++ code if
the control field name (which is '.') is used in the analysis_config,
so this is also rejected at the validation stage.

Relates elastic/x-pack-elasticsearch#1684

Original commit: elastic/x-pack-elasticsearch@e6750a2cda
2017-06-15 13:04:25 +01:00
Jason Tedor 8c5e7b589c Use master flag for disabling BWC tests
This commit skips the the BWC tests if the master BWC flag
bwc_tests_enabled in core is set to false.

Relates elastic/x-pack-elasticsearch#1725

Original commit: elastic/x-pack-elasticsearch@7b924066a9
2017-06-15 07:45:20 -04:00
Nik Everett 0970c509bc Remove the assemble task from projects not published (elastic/x-pack-elasticsearch#1721)
Removes the `assemble` task from projects that aren't published
to speed up `gradle assemble` so the unified release can call it.

Original commit: elastic/x-pack-elasticsearch@43dfcc15f3
2017-06-14 19:57:26 -04:00
Simon Willnauer 97693b9357 Add scroll support for cross cluster search (elastic/x-pack-elasticsearch#1706)
Original commit: elastic/x-pack-elasticsearch@eadffa396b
2017-06-14 20:38:58 +02:00
Jay Modi 9c8e12280b Test: increase the wait for green cluster health calls (elastic/x-pack-elasticsearch#1703)
This commit increases the amount of time to wait for green cluster health during a rolling upgrade
to account for the time that may be needed in the case of delayed shards. Additionally some old
timeout values were removed as they were used due to the default timeout of 30s.

Relates elastic/x-pack-elasticsearch#1683

Original commit: elastic/x-pack-elasticsearch@9996673db0
2017-06-14 10:25:40 -06:00
James Baiera 1d3921f581 Fixing vagrant build file issues with new vagrant support plugin
Original commit: elastic/x-pack-elasticsearch@dbb5d4a215
2017-06-12 10:03:30 -04:00
Dimitris Athanasiou 8eb62eac27 [ML] Automate detection of way to extract fields (elastic/x-pack-elasticsearch#1686)
In 5.4.x, the datafeed attempts to get all fields from
doc_values by default. It has a `_source` parameter which
when enabled changes the strategy to instead try to get
all fields from the source.

This has been the most common issue users have been
reporting as it means the datafeed will fail to fetch
any text fields by default.

This change uses the field capabilities API in order
to automatically detect whether a field is aggregatable.
It then extracts such fields from doc_values while the
rest are taken from source. The change also adds
validation to the start datafeed action so that if
fields are missing mappings or the time field is not
aggregatable we respond with an appropriate error.

relates elastic/x-pack-elasticsearch#1649

Original commit: elastic/x-pack-elasticsearch@76e2cc6cb2
2017-06-12 14:56:31 +01:00
Alexander Reelsen 27b5142de6 Watcher: Fix croneval tool for packaging (elastic/x-pack-elasticsearch#1689)
The croneval script used an old parameter to start up.
This commit removes this parameter, that is used, when a
package is used.

In addition a concrete vagrant test has been added.

relates elastic/x-pack-elasticsearch#1635

Original commit: elastic/x-pack-elasticsearch@ea7b8a08f4
2017-06-12 13:56:26 +02:00
Boaz Leskes b5ab68fac8 qa/upgrade_cluster/10_basic.yaml add shard level info on health failure
Original commit: elastic/x-pack-elasticsearch@46847ca262
2017-06-09 21:59:46 +02:00
Chris Earle c356074606 [Test] Rolling Upgrade should preserve templates (elastic/x-pack-elasticsearch#1687)
Real upgrades will have preserved their templates, so rolling upgrade tests should preserve them as well and internal services should be expected to replace them as needed.

Original commit: elastic/x-pack-elasticsearch@93a155951e
2017-06-09 13:41:22 -04:00
jaymode 9b3ee9f96c Remove security trace logging from rolling uprade tests
Original commit: elastic/x-pack-elasticsearch@b0ecb80816
2017-06-09 10:02:44 -06:00
Ryan Ernst c1a3f50e19 Convert script uses to use source/id keys (elastic/x-pack-elasticsearch#1670)
This is the xpack side of
https://github.com/elastic/elasticsearch/pull/25127

Original commit: elastic/x-pack-elasticsearch@e25bd90825
2017-06-09 08:29:36 -07:00
David Roberts 048ff24b79 [TEST] Wait for correct template version in rolling upgrade tests (elastic/x-pack-elasticsearch#1682)
We wait for index templates to be installed before running tests, but
these can get upgraded when the master node is upgraded.  If we don't
wait again in this case then tests can fail due to the cleanup code
being overwhelmed by pending cluster state updates that weren't waited
for before the test.

Original commit: elastic/x-pack-elasticsearch@eff0b24c11
2017-06-09 15:23:15 +01:00
Nik Everett b8d86682ef Rework rolling restart tests (elastic/x-pack-elasticsearch#1590)
Reworks the rolling restart tests so that all configuration
options share code. Now there is a project per configuration:
* `qa:rolling-upgrade:with-ssl-with-system-key`
* `qa:rolling-upgrade:with-ssl-without-system-key`

Original commit: elastic/x-pack-elasticsearch@5672b4a4f2
2017-06-08 12:48:41 -06:00
David Roberts 64330ff14d Move bracket
(Matches part of a change made to 5.5/5.x that doesn't need fully forward
porting.)

Original commit: elastic/x-pack-elasticsearch@01f0837e6c
2017-06-08 18:01:58 +01:00
David Roberts 7cb1c8bd35 [TEST] Fix security test blacklist for new ML test
Original commit: elastic/x-pack-elasticsearch@b6a054a2a7
2017-06-08 10:12:33 +01:00
David Roberts f865755259 [ML] Tolerate InternalAssumptionViolatedException in overridden test (elastic/x-pack-elasticsearch#1657)
In order for elastic/elasticsearch#25100 to work, overridden test()
methods that call ESClientYamlSuiteTestCase.test() must not consume
any InternalAssumptionViolatedException that
ESClientYamlSuiteTestCase.test() throws.

Relates elastic/x-pack-elasticsearch#1650

Original commit: elastic/x-pack-elasticsearch@081ccaa0a6
2017-06-07 14:22:05 +01:00
Alexander Reelsen 887538d6bc Watcher: Fix BWC tests, disable test requiring upgrade API
Original commit: elastic/x-pack-elasticsearch@e8de71cdf7
2017-06-07 11:09:31 +02:00
David Kyle ae299f633e [ML] Validate initial job settings (elastic/x-pack-elasticsearch#1646)
* [ML] Validate initial job settings

* Add same job creation checks to the validate endpoint

Original commit: elastic/x-pack-elasticsearch@ab76cf9ea2
2017-06-07 09:34:58 +01:00
Chris Earle b7c9fd7c93 Remove monitoring index checks until they exist without timing related issues elastic/x-pack-elasticsearch#1574
Original commit: elastic/x-pack-elasticsearch@97e44e4679
2017-06-07 00:32:10 -04:00
Chris Earle a357c97aaf Wait explicitly for .monitoring-es-* instead of the more generic wait
Original commit: elastic/x-pack-elasticsearch@4e1e4e379a
2017-06-06 18:24:17 -04:00
Chris Earle 3436ba6ecb With .monitoring-data-N index gone, we no longer want to wait for it
Original commit: elastic/x-pack-elasticsearch@312f1341ae
2017-06-06 18:22:00 -04:00
Jay Modi 2d893df7e9 Add better authorization for scroll requests and remove signing (elastic/x-pack-elasticsearch#1416)
This commit adds better security for scroll requests in that they are now tied to a single user as
we only authorize the request that creates the scroll. This is accomplished by adding a
SearchOperationListener that listens for new scroll contexts and stores the authentication on the
ScrollContext. Then upon
retrieval of the search context for a query or fetch, the current authentication is compared to the
authentication that was present when the scroll context was created. If the current authentication
belongs to a different user, then a SearchContextMissingException will be thrown to prevent leaking
a valid vs invalid scroll id.

Additionally, signing of a scroll id is only performed when there is a older node in the cluster
that would expect the scroll id to be signed. Once this is backported to 5.x, we can remove this
bwc layer for 6.0/master.

Original commit: elastic/x-pack-elasticsearch@0e5dcafd32
2017-06-06 10:23:18 -06:00
David Roberts b55d301a22 [TEST] Add more exclusions to ML security tests
Original commit: elastic/x-pack-elasticsearch@af9acc61ee
2017-06-05 11:37:12 +01:00
Ryan Ernst 7ee8eccf95 Script: Convert uses of CompiledTemplate to TemplateScript (elastic/x-pack-elasticsearch#1630)
This is the xpack side of elastic/elasticsearch#25032

Original commit: elastic/x-pack-elasticsearch@ba7df4f6ce
2017-06-02 13:41:33 -07:00
jaymode ff6fa6790e Test: fix kibana user role works in mixed cluster on older versions
The kibana user role in mixed cluster test expects a old cluster test to have run first that would
disable the kibana user, but that test is not executed on 5.1.1 and prior versions. This change
makes a update request an upsert to account for this.

relates elastic/x-pack-elasticsearch#1527

Original commit: elastic/x-pack-elasticsearch@df2945610e
2017-06-01 14:26:57 -06:00
David Kyle e16787ae7b Fix ML upgrade tests
Original commit: elastic/x-pack-elasticsearch@50e114e7eb
2017-06-01 20:26:57 +01:00
jaymode 68ad44c5c2 Test: enable default password migration test for versions > 5.1.1
Versions below 5.1.1 had a bug where the rest set enabled action would trip an assertion and cause
this test to fail so we skip those versions.

relates elastic/x-pack-elasticsearch#1523

Original commit: elastic/x-pack-elasticsearch@fba8ddd7e7
2017-06-01 08:37:39 -06:00
David Kyle 34f526b60b [ML] Yml rolling upgrade tests (elastic/x-pack-elasticsearch#1555)
* ML bwc jobs tests

* Test for starting old data feeds

Original commit: elastic/x-pack-elasticsearch@131c7b15eb
2017-06-01 14:29:10 +01:00
Nik Everett 592af606f8 Build: fix the rolling restarts for real
I had the numbers backwards. 🤦

Original commit: elastic/x-pack-elasticsearch@8f2d21db52
2017-05-26 17:31:50 -04:00
Nik Everett 928589bd89 Build: Use correct data node in test
We were getting lock errors because I hand ported something and made
a copy and paste error.

Original commit: elastic/x-pack-elasticsearch@9896ba4cc7
2017-05-26 15:20:44 -04:00
Nik Everett 43928708fa Build: handle core's change to data directory
Core changed how the data directory is configured so we have to
conform to the new way.

Original commit: elastic/x-pack-elasticsearch@360e2fea1a
2017-05-26 13:32:39 -04:00
Nik Everett b215c66778 Add tests for wire compatibility when system key is enabled (elastic/x-pack-elasticsearch#1557)
Reworks the rolling restart tests so they'd have caugh an
incompatibility in the wire protocol that we say between 5.4.0 and
5.4.1.

Original commit: elastic/x-pack-elasticsearch@f5e69cf58e
2017-05-26 12:17:00 -04:00
Alexander Reelsen e9fed20d96 Watcher: Use execute watch API in tests to prevent wrong watch history query
This test failed, because there could have been other results in the
watch history. However by switching to the execute watch API, there
is no need to query the watch history at all. This also removes a 5
second wait time in the test.

relates elastic/x-pack-elasticsearch#1549

Original commit: elastic/x-pack-elasticsearch@692779521e
2017-05-26 14:27:32 +02:00
Ryan Ernst 3b58334efb Refactor script context generic types (elastic/x-pack-elasticsearch#1547)
This is the xpack side of
https://github.com/elastic/elasticsearch/pull/24877

Original commit: elastic/x-pack-elasticsearch@4f3a1a27e6
2017-05-24 19:20:59 -07:00
Ryan Ernst 9a7c28786a Scripting: Use type aware script contexts (elastic/x-pack-elasticsearch#1538)
This creates two different script contexts for watcher, one which may be
used for SearchScript, and another for ExecutableScript.
This is the xpack side of elastic/elasticsearch#24868.

Original commit: elastic/x-pack-elasticsearch@9ae3d45fed
2017-05-24 14:29:25 -07:00
Alexander Reelsen 3d057991e0 Tests: Various watcher test improvements
* Reduced a longish timeout to a shorter one, as a watch should be
  executed in a HTTP test.
* Ensured that the TimeThrottleIntegration tests only query for own
  watches in the watch history, also use random names for watch ids
* HipChatServiceTests configured deprecated logging package, so it was
  not possible to follow the HTTP calls to the hipchat service endpoint.

relates elastic/x-pack-elasticsearch#1514
Relates elastic/x-pack-elasticsearch#1515

Original commit: elastic/x-pack-elasticsearch@adb492e4e9
2017-05-24 14:08:02 +02:00
Jason Tedor 284bf2512a Fix rolling upgrade test dependency
The rolling upgrade nodes need a keystore for SSL configuration but
there was no dependency on the task that copies the keystore into the
output directory for the nodes to pick up as an extra configuration
file. This commit addresses this by adding such a dependency. To do
this, we need to break the dependency of the keystore copy task on the
REST spec copy task; this is not an issue since the dependency was for
convenience of ordering the task and not actually needed.

Original commit: elastic/x-pack-elasticsearch@fddbc06e9f
2017-05-23 21:44:59 -04:00
Jason Tedor ed827970f1 Mark failing password migration test as skipped
A previous commit marked the wrong test as skipped, this commit marks
the correct issue.

Original commit: elastic/x-pack-elasticsearch@0723964206
2017-05-23 14:25:02 -04:00
Jason Tedor 3a151ac320 Mark failing password migration test as skipped
This test is failing reliably, so this commit marks it as skipped.

Original commit: elastic/x-pack-elasticsearch@15afd615d3
2017-05-23 14:13:57 -04:00
Jason Tedor dde2600181 Mark verify Kibana user works as skipped
This test fails reliably, so we mark it as skipped.

Original commit: elastic/x-pack-elasticsearch@4fc8a6883e
2017-05-23 13:35:44 -04:00
Jason Tedor 6ce8bd69ef Mark default password migration test as skipped
This test fails reliably, so this commit marks the test as awaits fix.

Original commit: elastic/x-pack-elasticsearch@8eb8b732ba
2017-05-23 13:35:44 -04:00
Jason Tedor 87ff7f0b52 Mark default password migration test as awaits fix
This commit marks the default password migration test as awaits fix as
the test is reliabily failling.

Original commit: elastic/x-pack-elasticsearch@4377480d98
2017-05-23 12:02:11 -04:00
David Kyle ea0f3fe4a0 [ML] Wait for a stopping datafeed (elastic/x-pack-elasticsearch#1461)
* [ML] Wait for a stopping datafeed

* Fix compilation after rebase

* Address review comments

Original commit: elastic/x-pack-elasticsearch@2baed641e9
2017-05-23 14:31:19 +01:00
Ryan Ernst 68e0bc419a Use simplified ScriptContext (elastic/x-pack-elasticsearch#1502)
This is the xpack side of elastic/elasticsearch#24818

Original commit: elastic/x-pack-elasticsearch@0edbbab431
2017-05-22 13:11:23 -07:00
Ryan Ernst 883f5d8a7a Remove overrides of isInlineScriptEnabled (elastic/x-pack-elasticsearch#1500)
This is the xpack side of elastic/elasticsearch#24815

Original commit: elastic/x-pack-elasticsearch@5aa314a49d
2017-05-20 12:01:34 -07:00
Jack Conradson b92a2f6582 Remove deprecated script settings. (elastic/x-pack-elasticsearch#1469)
Original commit: elastic/x-pack-elasticsearch@069e5a1087
2017-05-18 13:32:57 -07:00
Jay Modi 667f842f92 Fix authentication forward compatibility (elastic/x-pack-elasticsearch#1481)
The authentication object was changed in 5.4.0 in that it was conditionally signed depending on
the version and other factors. A bug was introduced however that causes the authentication to
actually get written with the version of the node it is being sent to even if that version is
greater than the version of the current node, which causes rolling upgrades to fail.

Original commit: elastic/x-pack-elasticsearch@a718ff8a52
2017-05-18 15:30:53 -04:00
Dimitris Athanasiou 1bb7651dba [ML] Refactor filters API to not use _type (elastic/x-pack-elasticsearch#1483)
- Removes dependence on _type for filters.
- Changes the put filter API to take the id in the URI
- Prepares .ml-meta index to be able to host more types in future

Relates elastic/x-pack-elasticsearch#668

Original commit: elastic/x-pack-elasticsearch@d4cffa9382
2017-05-18 18:09:20 +01:00
jaymode 1cc4ec95f3 Test: fix kibana write rolling upgrade test
This commit fixes the kibana write rolling upgrade test. The test needs to enable the kibana user
as another test explicitly disables it. After the test runs it disables the kibana user again.

relates elastic/x-pack-elasticsearch#1460

Original commit: elastic/x-pack-elasticsearch@437495432d
2017-05-18 10:12:34 -04:00
Ryan Ernst d6a9185857 Build: Convert rolling upgrade test to create tasks per bwc version (elastic/x-pack-elasticsearch#1477)
This is the xpack equivalent of elastic/elasticsearch#24758

Original commit: elastic/x-pack-elasticsearch@73e1a9c6b4
2017-05-18 02:25:54 -07:00
Ryan Ernst 58bd30e64f Fix put script calls in script tests with watcher
Original commit: elastic/x-pack-elasticsearch@2eac2e5192
2017-05-18 00:52:55 -07:00
Ryan Ernst d475010d1b Remove leftover file template references
Original commit: elastic/x-pack-elasticsearch@04e98c6261
2017-05-17 23:22:27 -07:00
Tim Vernum 7052194094 [TESTS] Fix filename pattern for templating in SSL smoke test
Relates to elastic/x-pack-elasticsearch@4c1d82c

Original commit: elastic/x-pack-elasticsearch@f5cbc1c6b3
2017-05-18 11:07:42 +10:00
Ryan Ernst f7705eac86 Remove file scripts (elastic/x-pack-elasticsearch#1399)
This is the xpack side of elastic/elasticsearch#24627


Original commit: elastic/x-pack-elasticsearch@4d1c745d74
2017-05-17 14:42:46 -07:00
Ryan Ernst 573da95e26 Use new wireCompatVersions property instead of bwcVersion (elastic/x-pack-elasticsearch#1466)
This is the xpack side of elastic/elasticsearch#24748

Original commit: elastic/x-pack-elasticsearch@8b7dd5cdbe
2017-05-17 12:58:51 -07:00
jaymode cc167467aa Test: mute kibana_write rest test that never ran or worked
See elastic/x-pack-elasticsearch#1460

Original commit: elastic/x-pack-elasticsearch@505c8b8832
2017-05-17 11:45:22 -04:00
Simon Willnauer bfc3779f51 Fix indentation in 30_kibana_write.yml
Original commit: elastic/x-pack-elasticsearch@6321493b03
2017-05-17 17:13:40 +02:00
Simon Willnauer 4f94624277 Rename test from `.yaml` to `yml`
We cut over to run tests only for `.yml` files in  elastic/elasticsearch#24659
The corresponding x-pack change was missing, this commit fixes it.

relates elastic/x-pack-elasticsearch#1457

Original commit: elastic/x-pack-elasticsearch@4c1d82c209
2017-05-17 15:39:57 +02:00
David Kyle abbdf232aa [ML] Test ML with the Transport Client (elastic/x-pack-elasticsearch#1440)
* Hide ML actions for tribe node client
* Remove unused parameters
* Enable ML actions and rest endpoints for the transport client
* Create the ML components for the transport client
* Add ml transport client tests

Original commit: elastic/x-pack-elasticsearch@509007ca29
2017-05-16 14:34:44 +01:00
Tim Vernum 463133b7de [Security] Cross cluster wildcard security (elastic/x-pack-elasticsearch#1290)
Support the resolution of remote index names, including those that contain wildcards in the cluster name or index part)

Specifically these work:
- `GET /remote*:foo/_search`
- `GET /*:foo/_search`
- `GET /*:foo,*/_search`
- `GET /remote:*/_search`
- `GET /*:*/_search`

This change assumes that every user is allowed to attempt a cross-cluster search against any remote index, and the actual authorisation of indices happens on the remote nodes. Thus ` GET /*:foo/_search` will expand to search the `foo` index on every registered remote without consideration of the roles and privileges that the user has on the source cluster.

Original commit: elastic/x-pack-elasticsearch@b45041aaa3
2017-05-15 15:02:13 +10:00
Dimitrios Athanasiou 7293bcdc1c [TEST] Set data_description in MlTransportClientIT
Original commit: elastic/x-pack-elasticsearch@0d392d26d8
2017-05-12 18:51:38 +01:00
Alexander Reelsen 50e9e413da Watcher: Make watch history use doc type instead of watch_record (elastic/x-pack-elasticsearch#1311)
As this does not require any reindexing this is easy to fix by just
changing the watch history template.

In addition the old templates are deleted on start up and the new ones
are instantiated.

Original commit: elastic/x-pack-elasticsearch@7e1ad495ad
2017-05-12 16:52:57 +02:00
Simon Willnauer 891c2a6c3f Replace XPacks delete_by_query impl with the core impl (elastic/x-pack-elasticsearch#1378)
This can now be shared with core once elastic/elasticsearch#24578

Original commit: elastic/x-pack-elasticsearch@42bbd75aee
2017-05-11 20:23:55 +02:00
David Roberts cd99024599 [ML] Validate job configs before transport (elastic/x-pack-elasticsearch#1375)
If invalid job configs are transported to the master node then the root
cause of the validation exception gets reported as a remote_transport_exception,
which is extremely confusing.

This commit moves the validation of job configurations to the first node that
handles the action.

Fixes elastic/x-pack-kibana#1172

Original commit: elastic/x-pack-elasticsearch@5ed59d2a6f
2017-05-10 10:55:16 +01:00
Ryan Ernst 85deb1f8f7 Rename ScriptEngineService to ScriptEngine (elastic/x-pack-elasticsearch#1374)
This is the xpack side of elastic/elasticsearch#24574

Original commit: elastic/x-pack-elasticsearch@4d37021f6d
2017-05-10 00:47:39 -07:00
Ryan Ernst 1c3d907748 Reverse runAs user setup to store authenticated user inside runAs user (elastic/x-pack-elasticsearch#1371)
Original commit: elastic/x-pack-elasticsearch@8276662298
2017-05-09 13:49:14 -07:00
David Kyle e5b11d0222 [ML] Not an error to close a job twice (elastic/x-pack-elasticsearch#1340)
* [ML] Not an error to close a job twice

* Error if job is opening

* Address review comments

* Test closed job isn’t resolved

Original commit: elastic/x-pack-elasticsearch@7da7b24c08
2017-05-08 16:34:46 +01:00
Simon Willnauer 5a4b7ee98b [TEST] Add REST test to search via unfiltered alias in CrossClusterSearch
Original commit: elastic/x-pack-elasticsearch@4e1b99824c
2017-05-05 10:25:06 +02:00
Simon Willnauer eddc7a2147 Add tests for cross cluster `_field_caps` API (elastic/x-pack-elasticsearch#1294)
This commit adds REST tests to ensure `_field_caps` works with cross cluster
search and security.

Relates to elastic/elasticsearch#24463

Original commit: elastic/x-pack-elasticsearch@2af0a8c179
2017-05-04 15:04:39 +02:00
Alexander Reelsen 4078b2f1b2 Watcher: Replace _status field with status (elastic/x-pack-elasticsearch#1285)
As fields with underscores will be disallowed in master, and we have to
prepare the upgrade, this commit renames the _status field to status.

When the 5.x upgrade logic is in place in the 5.x we can remove all the
old style _status handling from the master branch.

Note: All the BWC compatibility tests, that load 5.x indices are now
faking a finished upgrade by adding the `status` field to the mapping
of the watches index.

Original commit: elastic/x-pack-elasticsearch@9d5cc9aaec
2017-05-04 10:08:34 +02:00
Alexander Reelsen 50241717d3 Tests: Use execute watch API instead of persisting watch
To speed up the test and ensure no other failures occur,
simply use the execute watch API instead of persisting the watch
and executing then.

relates elastic/x-pack-elasticsearch#1350

Original commit: elastic/x-pack-elasticsearch@2888fb0e93
2017-05-02 14:04:04 +02:00
Alexander Reelsen c62f6f8177 Watcher: Distributed watch execution (elastic/x-pack-elasticsearch#544)
The distribution of watches now happens on the node which holds the
watches index, instead of on the master node. This requires several
changes to the current implementation.

1. Running on shards and replicas
   In order to run watches on the nodes with the watches index on its
   primaries and replicas. To ensure that watches do not run twice, there is
   a logic which checks the local shards, runs a murmurhash on the id and
   runs modulo against the number of shards and replicas, this is the way to
   find out, if a watch should run local. Reloading happens
2. Several master node actions moved to a HandledTransportAction, as they
   are basically just aliases for indexing actions, among them the
   put/delete/get watch actions, the acknowledgement action, the de/activate
   actions
3. Stats action moved to a broadcast node action, because we potentially
   have to query every node to get watcher statistics
4. Starting/Stopping watcher now is a master node action, which updates
   the cluster state and then listeners acts on those. Because of this watches
   can be running on two systems, if you those have different cluster state
   versions, until the new watcher state is propagated
5. Watcher is started on all nodes now. With the exception of the ticker
   schedule engine most classes do not need a lot of resources while running.
   However they have to run, because of the execute watch API, which can hit
   any node - it does not make sense to find the right shard for this watch
   and only then execute (as this also has to work with a watch, that has not
   been stored before)
6. By using a indexing operation listener, each storing of a watch now
   parses the watch first and only stores on successful parsing
7. Execute watch API now uses the watcher threadpool for execution
8. Getting the number of watches for the stats now simply queries the
   different execution engines, how many watches are scheduled, so this is
   not doing a search anymore

There will be follow up commits on this one, mainly to ensure BWC compatibility.

Original commit: elastic/x-pack-elasticsearch@0adb46e658
2017-05-02 10:12:46 +02:00
Tim Vernum ddf5fd68c2 Add ClusterSearchShardsAction to "read_cross_cluster" privilege (elastic/x-pack-elasticsearch#1231)
Cross cluster search uses ClusterSearchShardsAction under the covers.
Without this change, you would need both "read_cross_cluster" and "view_index_metadata" privilegs in order to have permission to execute searches from a remote cluster.

Original commit: elastic/x-pack-elasticsearch@65a6aff329
2017-04-27 22:39:13 +10:00
Yannick Welsch c6b8542758 [TEST] Adapt wait_for_nodes condition for ClusterFormationTasks with tribes
Companion commit for elastic/elasticsearch#24351

Original commit: elastic/x-pack-elasticsearch@0f57eecbdb
2017-04-27 09:57:54 +02:00
Luca Cavanna 2a1fd02dcf [TEST] Cross Cluster Search: specify index name when creating role (elastic/x-pack-elasticsearch#1208)
This makes sure that the user needs to have access only to the specific index that needs to be accessed on the remote clusters.

relates to elastic/x-pack-elasticsearch#1153

Original commit: elastic/x-pack-elasticsearch@7966416bd7
2017-04-26 21:46:31 +02:00
Jay Modi 295051ee8c Add a short-lived token based access mechanism (elastic/x-pack-elasticsearch#1029)
This commit adds a token based access mechanism that is a subset of the OAuth 2.0 protocol. The
token mechanism takes the same values as a OAuth 2 standard (defined in RFC 6749 and RFC 6750),
but differs in that we use XContent for the body instead of form encoded values. Additionally, this
PR provides a mechanism for expiration of a token; this can be used to implement logout
functionality that prevents the token from being used again.

The actual tokens are encrypted using AES-GCM, which also provides authentication. The key for
encryption is derived from a salt value and a passphrase that is stored on each node in the
secure settings store. By default, the tokens have an expiration time of 20 minutes and is
configurable up to a maximum of one hour.

Relates elastic/x-pack-elasticsearch#8

Original commit: elastic/x-pack-elasticsearch@3d201ac2bf
2017-04-26 08:00:03 -04:00
Jay Modi d2b909fe5a Test: remove roles from user on remote cluster (elastic/x-pack-elasticsearch#1144)
This change removes the roles from the user in the remote cluster to show that the roles passed
from the source cluster are used in evaluating the user's permissions during multi-cluster search.

Original commit: elastic/x-pack-elasticsearch@35658168fb
2017-04-24 09:18:26 -04:00
David Roberts 5b19b8e7f0 [TEST] Wait for all ML index template before ML security tests (elastic/x-pack-elasticsearch#1148)
Previously we were only waiting for the results index template

Original commit: elastic/x-pack-elasticsearch@3a9e6ab82f
2017-04-21 07:50:10 +01:00
David Roberts caa080ec02 [TEST] Fix exception specification
Caused by a clash between two PRs

Original commit: elastic/x-pack-elasticsearch@47910636eb
2017-04-20 10:04:30 +01:00
David Roberts b03147bea9 [TEST] Don't duplicate the MlRestTestStateCleaner class (elastic/x-pack-elasticsearch#1127)
We didn't realise it was possible for a qa module to depend on the
test classes of the plugin module, so we duplicated a test class.
But it turns out it IS possible to declare this dependency and avoid
the duplication.

Original commit: elastic/x-pack-elasticsearch@b6a21cda28
2017-04-20 09:13:04 +01:00
David Kyle d8a70138cd [TEST] Give jobs meaningful names in ML integ tests (elastic/x-pack-elasticsearch#1103)
Original commit: elastic/x-pack-elasticsearch@aaf1807172
2017-04-19 12:00:53 +01:00
Ryan Ernst b86cdd6c8e Test: Update rest base class parameters signature (elastic/x-pack-elasticsearch#1101)
This is the xpack side of elastic/elasticsearch#21392

Original commit: elastic/x-pack-elasticsearch@b760815f54
2017-04-18 15:07:14 -07:00
Ryan Ernst ef3d3b51a4 Move integ test runner deps to cluster deps (elastic/x-pack-elasticsearch#1096)
This is the xpack side of elastic/elasticsearch#24142

Original commit: elastic/x-pack-elasticsearch@d502f06cea
2017-04-17 16:04:09 -07:00
Jay Modi b59b6bbdd4 Remove SecuredString and use SecureString from elasticsearch core (elastic/x-pack-elasticsearch#1092)
This commit removes the SecuredString class that was previously used throughout the security code
and replaces it with the SecureString class from core that was added as part of the new secure
settings infrastructure.

relates elastic/x-pack-elasticsearch#421

Original commit: elastic/x-pack-elasticsearch@e9cd117ca1
2017-04-17 13:28:46 -04:00
Martijn van Groningen d114a55b99 [ML] Make open job and start datafeed apis master node actions and
let close job and stop datafeed apis redirect to elected master node.

This is for cluster state observation purposes, so that a subsequent open and then close job or
start and then stop datafeed see the same local cluster state and sanity validation doesn't fail.

Original commit: elastic/x-pack-elasticsearch@21a63184b9
2017-04-14 10:43:21 +02:00
David Kyle 17a8c9b9e8 Revert "Muted test."
This reverts commit elastic/x-pack-elasticsearch@da69d049ad.

Original commit: elastic/x-pack-elasticsearch@e676c4267c
2017-04-13 14:15:24 +01:00
Martijn van Groningen fc06f09b91 [TEST] Only run qa cluster with ml enabled.
Relates to elastic/x-pack-elasticsearch#997

Original commit: elastic/x-pack-elasticsearch@863bf9abfe
2017-04-13 10:50:03 +02:00
Martijn van Groningen 4f34af20ba Muted test.
Original commit: elastic/x-pack-elasticsearch@da69d049ad
2017-04-11 20:51:52 +02:00
Simon Willnauer a7dfbcd2cb [TEST] Use SecurityClusterClientYamlTestCase to ensure mappings and templates are present
Original commit: elastic/x-pack-elasticsearch@742633c93c
2017-04-11 18:11:31 +02:00
Ali Beyad c468a4d3a3 [TEST] Security index template/mappings check on tests (elastic/x-pack-elasticsearch#1031)
Extends the security index check that ensure the template/mappings
are up-to-date to other security integration tests that depend on this
check.

relates elastic/x-pack-elasticsearch#794

Original commit: elastic/x-pack-elasticsearch@ec8e5b37bd
2017-04-11 11:22:41 -04:00
Simon Willnauer 9c542173b7 [TEST] Make test more robust against slow connection establishing
Original commit: elastic/x-pack-elasticsearch@6f918a44c9
2017-04-11 14:43:44 +02:00
Dimitris Athanasiou 4da1c5b9dc [ML] Remove batch_span and period from analysis config (elastic/x-pack-elasticsearch#1043)
relates elastic/x-pack-elasticsearch#1040

Original commit: elastic/x-pack-elasticsearch@aed5e9912c
2017-04-11 12:57:58 +01:00
Hendrik Muhs 7ef9a16f45 [ML] implement '_all' for stopping datafeeds (elastic/x-pack-elasticsearch#995)
Add a '_all' functionality for stopping ML datafeeds.

For cluster shutdown due to maintenance and major upgrades we recommend the user to stop all datafeeds and jobs. This change add the ability to stop all datafeeds at once where previously it was required to iterate over all feeds and do a explicit stop. This is part two of elastic/x-pack-elasticsearch#795, part one can be found in elastic/x-pack-elasticsearch#962 .

relates elastic/x-pack-elasticsearch#795

Original commit: elastic/x-pack-elasticsearch@ed1eff83d5
2017-04-11 13:39:22 +02:00
Simon Willnauer 617c3ead5c Add tests for _remote/info API (elastic/x-pack-elasticsearch#1009)
Relates to elastic/elasticsearch#23925
Depends on elastic/elasticsearch#23969

Original commit: elastic/x-pack-elasticsearch@d1e8754a57
2017-04-11 11:24:22 +02:00
Tim Vernum 8840042751 Only log LDAP errors if authentication failed overall (elastic/x-pack-elasticsearch#960)
Changes the logging of LDAP authentication failures from "always" to "only if the user failed to be authenticated"

Previously there were cases (such has having 2 AD realms) where successful user authentication would still cause an INFO message to be written to the log for every request. 
Now that message is suppressed, but a WARN message is added _if-and-only-if_ the user cannot be authenticated by any realm.

This is implemented via a new value stored in the ThreadContext that the AuthenticationService choses to log (or not log) depending on the result of the authenticate process.
  
Closes: elastic/x-pack-elasticsearch#887

Original commit: elastic/x-pack-elasticsearch@b81b363729
2017-04-11 09:37:21 +10:00
Tanguy Leroux 6396edc6a7 [Test] Add Monitoring Bulk API REST tests with privileges (elastic/x-pack-elasticsearch#968)
This commit removes the smoke-test-monitoring-with-security project
and replaces it with a REST test.

Original commit: elastic/x-pack-elasticsearch@f1665815c2
2017-04-10 15:08:19 +02:00
Ryan Ernst e8399cfccf Build: Build 5.x version of xpack for bwc tests to use (elastic/x-pack-elasticsearch#885)
This is analagous of the bwc-zip for elasticsearch. The one caveat is
due to the structure of how ES+xpack must be checked out, we end up with
a third clone of elasticsearch (the second being in :distribution:bwc-zip).
But the rolling upgrade integ test passes with this change.

relates elastic/x-pack-elasticsearch#870

Original commit: elastic/x-pack-elasticsearch@34bdce6e99
2017-04-07 22:19:30 -07:00
Hendrik Muhs c7fd1aacff [ML] implement _all for closing jobs (elastic/x-pack-elasticsearch#962)
Add a '_all' functionality for closing ML jobs.

For cluster shutdown due to maintenance and major upgrades we recommend the user to stop all datafeeds and jobs. This change add the ability to close all jobs at once where previously it was required to iterate over all jobs and do a explicit close. This is part one of elastic/x-pack-elasticsearch#795, part two can be found in elastic/x-pack-elasticsearch#995.

relates elastic/x-pack-elasticsearch#795

Original commit: elastic/x-pack-elasticsearch@9b251ed7e1
2017-04-07 14:51:13 +02:00
Martijn van Groningen 2b5c0faebd [TEST] wait for cs version to be the same on all nodes before stopping datafeed
Original commit: elastic/x-pack-elasticsearch@6fd6224047
2017-04-05 11:37:36 +02:00
Martijn van Groningen 4a57c92e6f [TEST] unmuted test
Original commit: elastic/x-pack-elasticsearch@5dd80c32c7
2017-04-05 09:46:02 +02:00
Jason Tedor dee1bd1825 Respond to rename random ASCII helper methods
This commit is response to the renaming of the random ASCII helper
methods in ESTestCase. The name of this method was changed because these
methods only produce random strings generated from [a-zA-Z], not from
all ASCII characters.

Relates elastic/x-pack-elasticsearch#942

Original commit: elastic/x-pack-elasticsearch@a6085964d3
2017-04-04 11:04:37 -04:00
Yannick Welsch e52fbdf63c Use random http port for watcher REST tests (elastic/x-pack-elasticsearch#936)
Adapts the Watcher REST tests so that they don't require a fixed http port anymore.

Original commit: elastic/x-pack-elasticsearch@14919b16e7
2017-04-04 09:02:23 +02:00
David Kyle 5b66c7a7ba [ML] Set job create time on server (elastic/x-pack-elasticsearch#910)
* [ML] Set job create time on server

* Job.Builder serialisation tests

* Make setCreateTime package private

Original commit: elastic/x-pack-elasticsearch@d2d75e0d7b
2017-04-03 18:30:47 +01:00
Simon Willnauer 5e6bfb9a82 Fix cross cluster search with security (elastic/x-pack-elasticsearch#904)
This commit adds an integration test that runs basic cross cluster search
actions across 2 clusters with security installed. This commit also fixes
several issues with respect to internal actions and proxy actions in the context
of cross cluster search.

Relates to elastic/elasticsearch#23830
relates elastic/x-pack-elasticsearch#892

Original commit: elastic/x-pack-elasticsearch@2e5486c259
2017-04-03 09:53:48 +02:00
Ryan Ernst e1949ee362 Test: Use separate wrapper tasks for tribe clusters setup (elastic/x-pack-elasticsearch#905)
This commit adds separate tasks for tribe clusteres which the
cluster formation tasks build their own tasks off. This ensures each
cluster will have its own wait task, so that the tribe node will be able
to wait on the other clusters being up before even trying to start.

relates elastic/x-pack-elasticsearch#877

Original commit: elastic/x-pack-elasticsearch@1e4c729372
2017-03-31 14:30:44 -07:00
David Roberts 95fd6777a0 [ML] Add 'xpack' into action names to match what the rest of X-Pack does (elastic/x-pack-elasticsearch#909)
Original commit: elastic/x-pack-elasticsearch@36be904ad9
2017-03-31 11:22:53 +01:00
Jason Tedor f52760d2ac Revert "Silence UpgradeClusterClientYamlTestSuiteIT"
This reverts commit elastic/x-pack-elasticsearch@e29f94b24c.

Original commit: elastic/x-pack-elasticsearch@d7d27b20ef
2017-03-30 08:39:10 -04:00
David Roberts 72f5e92972 [ML] Adjust error to make sense for FAILED as well as CLOSED jobs (elastic/x-pack-elasticsearch#888)
Previously trying to perform an operation on a FAILED job would tell
you that it hadn't been opened.  This was very confusing if you knew
very well you'd just opened the job.

Relates elastic/x-pack-elasticsearch#879

Original commit: elastic/x-pack-elasticsearch@2dbe41caaa
2017-03-30 10:33:56 +01:00
David Roberts c335e79508 [TEST] Enable two blacklisted tests that now work
Previously force closing a job required extra privileges.  Following
the full discussion about what privileges should be required.

Original commit: elastic/x-pack-elasticsearch@4d85314b35
2017-03-30 10:26:15 +01:00
Martijn van Groningen 61b4f6480e [TEST] muted test
Original commit: elastic/x-pack-elasticsearch@550b55739e
2017-03-30 11:24:27 +02:00
Martijn van Groningen f2654b5872 [ML] Changed job and datafeed lifecycle management
* Removed OPENING and CLOSING job states. Instead when persistent task has been created and
   status hasn't been set then this means we haven't yet started, when the executor changes it to STARTED we have.
   The coordinating node will monitor cs for a period of time until that happens and then returns or times out.
* Refactored job close api to go to node running job task and close job there.
* Changed unexpected job and datafeed exception messages to not mention the state and instead mention that job/datafeed haven't yet started/stopped.

Original commit: elastic/x-pack-elasticsearch@37e778b585
2017-03-29 20:35:10 +02:00
Daniel Mitterdorfer 2355338625 Silence UpgradeClusterClientYamlTestSuiteIT
Relates elastic/x-pack-elasticsearch#870
Relates elastic/x-pack-kibana#690

Original commit: elastic/x-pack-elasticsearch@e29f94b24c
2017-03-29 10:38:28 +02:00
Dimitrios Athanasiou 991e5d8833 [TEST] Set required euery on delete-by-query request
Original commit: elastic/x-pack-elasticsearch@b9b6234f3d
2017-03-28 17:15:23 +01:00
Colin Goodheart-Smithe 99db6013ea [ML] Adds a test for ml transport client (elastic/x-pack-elasticsearch#851)
This change also adds synchronous methods to `MachineLearningClient`.

relates elastic/x-pack-elasticsearch#567

Original commit: elastic/x-pack-elasticsearch@b3a4b38a51
2017-03-28 09:05:55 +01:00
Martijn van Groningen fc29fd159a [TEST] Don't lose the actual cause of the exception.
Original commit: elastic/x-pack-elasticsearch@8720288ef5
2017-03-27 15:33:04 +02:00
David Roberts ecbfaace38 [TEST] Improve ML security test comments and account for new close assertion
Original commit: elastic/x-pack-elasticsearch@004845bf2f
2017-03-27 10:24:05 +01:00
David Roberts c0445fac4d [TEST] Silence failing ML security tests
Original commit: elastic/x-pack-elasticsearch@af4ed2019d
2017-03-24 17:58:30 +00:00
Colin Goodheart-Smithe caa9afad2b [TEST] fixed content type in ML QA tests
Original commit: elastic/x-pack-elasticsearch@b7749e1319
2017-03-24 14:08:39 +00:00
Hendrik Muhs a8b1b3e863 Revert "disable integration test, see elastic/x-pack-elasticsearch#823 for details (elastic/x-pack-elasticsearch#824)"
This reverts commit elastic/x-pack-elasticsearch@2338b3c972.

Original commit: elastic/x-pack-elasticsearch@a9c7168762
2017-03-23 20:10:58 +01:00
Hendrik Muhs 27dce8669c disable integration test, see elastic/x-pack-elasticsearch#823 for details (elastic/x-pack-elasticsearch#824)
[ML] disable failing integration test, see elastic/x-pack-elasticsearch#823 for details

Original commit: elastic/x-pack-elasticsearch@2338b3c972
2017-03-23 19:54:02 +01:00
Hendrik Muhs 6f7f466fa3 [ML] move DataStreamDiagnostics into DataCountsReporter (elastic/x-pack-elasticsearch#775)
repair DataStreamDiagnostics

Moves DataStreamDiagnostics into DataCountsReporter to survive if job is opened/closed/fed in chunks.

relates elastic/x-pack-elasticsearch#764

Original commit: elastic/x-pack-elasticsearch@29c221a451
2017-03-23 16:43:51 +01:00
Jay Modi 8ba6e8b3eb Fix merging of field level security rules (elastic/x-pack-elasticsearch#796)
This commit fixes the merging of field level security rules from multiple roles. Prior to 5.2, the
merging was treated as the merging of languages whereas after 5.2, this incorrectly became a merge
of all rules meaning a single wildcard could cause denials to be ignored.

Original commit: elastic/x-pack-elasticsearch@42f9e6d8b0
2017-03-22 05:10:30 -07:00