Commit Graph

376 Commits

Author SHA1 Message Date
Alexander Reelsen 9b0a1a34e0 Upgrade: Remove watcher/security upgrade checks (elastic/x-pack-elasticsearch#2338)
The checks are used for the 5.6 to 6.x transition, thus they do
not make sense to keep in 7.x.

Original commit: elastic/x-pack-elasticsearch@c6c6fa819e
2017-08-25 17:24:49 +02:00
David Kyle c6b6a5c804 Fix failing ML test after bucket count change (elastic/x-pack-elasticsearch#2351)
Original commit: elastic/x-pack-elasticsearch@c215ba1c16
2017-08-25 10:25:18 +01:00
Albert Zaharovits 026729e911 TOKEN_SERVICE_ENABLED_SETTING enabled if HTTP_SSL_ENABLED (elastic/x-pack-elasticsearch#2321)
`authc.token.enabled` is true unless `http.ssl.enabled` is `false` and `http.enabled` is `true`.

* TokenService default enabled if HTTP_ENABLED == false

* Fixed tests that need TokenService explicitly enabled

* [DOC] Default value for `xpack.security.authc.token.enabled`

Original commit: elastic/x-pack-elasticsearch@bd154d16eb
2017-08-23 13:21:30 +03:00
Alexander Reelsen 3f541fa556 Tests: Ensure watcher is started via awaitBusy in bwc test
Original commit: elastic/x-pack-elasticsearch@a8c0cf04c9
2017-08-22 00:39:11 +02:00
Alexander Reelsen 17980ab360 Tests: Remove randomized unsupported code upgrading two indices at once
Original commit: elastic/x-pack-elasticsearch@c86b87927d
2017-08-22 00:09:01 +02:00
Alexander Reelsen 27f39c615b Watcher: Create two index ugprade checks for watcher upgrade (elastic/x-pack-elasticsearch#2298)
As there are two indices to upgrade for watcher, it makes a lot of sense
to also have two upgrade checks.

There is one upgrader for the watches index, which deletes
old templates, adds the new one before and then does the reindexing.
Same for the triggered watches index.

This also means, that there will be two entries popping up in the kibana
UI.

Note: Each upgrade check checks if the other index (for the .watches
upgrade check the triggered watches index and vice versa) is already
upgraded and only if that is true, watcher is restarted.

relates elastic/x-pack-elasticsearch#2238

Original commit: elastic/x-pack-elasticsearch@2c92040ed6
2017-08-21 17:36:16 +02:00
Simon Willnauer 71827b70a0 Bump token service BWC version to 6.0.0-beta2
Original commit: elastic/x-pack-elasticsearch@ef688f02cb
2017-08-18 17:06:45 +02:00
Simon Willnauer ac9ab974f4 Ensure token service can boostrap itself without a pre-shared key (elastic/x-pack-elasticsearch#2240)
Today we require a pre-shared key to use the token service. Beside the
additional setup step it doesn't allow for key-rotation which is a major downside.

This change adds a TokenService private ClusterState.Custom that is used to distribute
the keys used to encrypt tokens. It also has the infrastructur to add automatic key
rotation which is not in use yet but included here to illustrate how it can work down
the road.

This is considered a prototype and requires additioanl integration testing. Yet, it's fully
BWC with a rolling / full cluster restart from a previous version (also from 5.6 to 6.x)
since if the password is set it will just use it instead of generating a new one.
Once we implement the automatic key rotation via the clusterstate we need to ensure that we are
fully upgraded before we do that.
Also note that the ClusterState.Custom is fully transient and will never be serialized to disk.

Original commit: elastic/x-pack-elasticsearch@1ae22f5d41
2017-08-18 14:23:43 +02:00
Simon Willnauer 724325f161 Fallback to `keystore.seed` as a bootstrap password if actual password is not present (elastic/x-pack-elasticsearch#2295)
Today we require the `bootstrap.password` to be present in the keystore in order to
bootstrap xpack. With the addition of `keystore.seed` we have a randomly generated password
per node to do the bootstrapping. This will improve the initial user experience significantly
since the user doesn't need to create a keystore and add a password, they keystore is created
automatically unless already present and is always created with this random seed.

Relates to elastic/elasticsearch#26253

Original commit: elastic/x-pack-elasticsearch@5a984b4fd8
2017-08-17 16:42:32 +02:00
Alexander Reelsen 6d30806996 Watcher: Improvements on the rolling restart tests (elastic/x-pack-elasticsearch#2286)
This improves the rolling restart tests (tests different paths in
different ways) and aligns the upgrade code with the 5.6 branch from

Relates elastic/x-pack-elasticsearch#2238

Original commit: elastic/x-pack-elasticsearch@01b0954558
2017-08-17 11:41:11 +02:00
Alexander Reelsen 0a86f00d7e Tests: Ensure responses are closed in watch backwards compat tests
The HTTP client has to have it's response entities closed, otherwise
it might block further requests because the underlying connection cannot
be reused.

Relates elastic/x-pack-elasticsearch#2004

Original commit: elastic/x-pack-elasticsearch@a24ecb9764
2017-08-16 16:29:52 +02:00
Alexander Reelsen 69b3ffa40a Tests: Remove useless wait time in watcher REST test
The rest test waited for the watch to run in the background, but there
were no guarantees that this really happened. Also it waited for five
seconds, instead of just executing the watch manually.

relates elastic/x-pack-elasticsearch#2255

Original commit: elastic/x-pack-elasticsearch@56765a649e
2017-08-14 11:52:26 +02:00
Alexander Reelsen 3ad2f5e9f5 Tests: Increase logging for watcher smoke tests
These tests have repeating but not reproducible failures,
where the stash is filled with a second PUT operation and the
watcher stats response does not match. Setting the log to trace
should shed some light on this.

As the smoke tests are only four tests this will not lead to a
log explosion.

Relates elastic/x-pack-elasticsearch#1513, elastic/x-pack-elasticsearch#1874

Original commit: elastic/x-pack-elasticsearch@5832dc7990
2017-08-11 15:29:52 +02:00
Alexander Reelsen 26c5766a0d Tests: Do not delete index templates on bwc tests
In order to run the bwc tests there is no need to delete the
index template at the end of a test run which results
in recreation of those due to all the cluster state listener.

Relates elastic/x-pack-elasticsearch#2228

Original commit: elastic/x-pack-elasticsearch@702d1c61ed
2017-08-11 14:11:50 +02:00
Alexander Reelsen 2d08477093 Tests: Fix upgrade tests by not running stats against arbitrary hosts
Original commit: elastic/x-pack-elasticsearch@1b858aad52
2017-08-07 13:27:36 +02:00
Alexander Reelsen 8f6874abf9 Tests: Increase logging in watcher upgrade tests to debug test failures
Original commit: elastic/x-pack-elasticsearch@380a8541dc
2017-08-07 11:31:16 +02:00
David Roberts 05cbe8dc0c [ML] Disallow creating a job against a closed results or state index (elastic/x-pack-elasticsearch#2186)
Previously if this was attempted you'd get an NPE (5.x) or hang (6.x).
Following this change you get an error message telling you what the
problem is.

relates elastic/x-pack-elasticsearch#2170

Original commit: elastic/x-pack-elasticsearch@ea12a9ff46
2017-08-07 08:53:12 +01:00
Jay Modi 8b0fb5eae8 Re-enable OpenLDAP tests and run against vagrant instance (elastic/x-pack-elasticsearch#2121)
This commit re-enables the OpenLDAP tests that were previously running against a one-off instance
in AWS but now run against a vagrant fixture. There were some IntegTests that would run against the
OpenLDAP instance randomly but with this change they no longer run against OpenLDAP. This is ok as
the functionality that is tested by these has coverage elsewhere.

relates elastic/x-pack-elasticsearch#1823

Original commit: elastic/x-pack-elasticsearch@ac9bc82297
2017-08-04 09:44:08 -06:00
Jay Modi a7d6138f83 Fix the building of the default URL for the setup password tool (elastic/x-pack-elasticsearch#2176)
This commit fixes the building of the default URL for the setup password tool so that a default
elasticsearch.yml file will still result in a succesful run of the tool.

relates elastic/x-pack-elasticsearch#2174

Original commit: elastic/x-pack-elasticsearch@2291b14875
2017-08-03 15:14:24 -06:00
Ryan Ernst 45a55d16cf Bump master version to 7.0.0-alpha1 (elastic/x-pack-elasticsearch#2135)
This is the xpack side of
https://github.com/elastic/elasticsearch/pull/25876

Original commit: elastic/x-pack-elasticsearch@c86ea25009
2017-08-01 15:48:04 -04:00
Alexander Reelsen 4bf5d9536a Tests: Remove @ClusterScope tests, create REST tests (elastic/x-pack-elasticsearch#2131)
Replacing integration tests with rest tests and unit tests, thus removing integration tests that require start of a new cluster. Removing unused testing methods

Original commit: elastic/x-pack-elasticsearch@265966d80c
2017-08-01 14:15:36 +02:00
Jason Tedor 5b1bf9a31e Remove max script compilation settings in tests
Some standalone tests set the max script compilation limit. However,
this setting is now set in the main cluster setup in core so it is no
longer needed here. This commit removes these obviated settings in
standalone tests.

Original commit: elastic/x-pack-elasticsearch@089328c8d7
2017-08-01 14:57:57 +09:00
Jason Tedor 50b8a56d34 Add packaging assertions for new scripts
This commit adds packaging assertions for recent script additions:
setup-passwords and x-pack-env.

Original commit: elastic/x-pack-elasticsearch@d83624f419
2017-07-30 09:32:39 +09:00
Jay Modi db4c00b565 Update the full cluster restart tests to be more generic (elastic/x-pack-elasticsearch#2107)
The full cluster restart tests are currently geared towards the 5.6 -> 6.0 upgrade and have some
issues when the versions are changed to 6.x -> 7.0. One issue is a real code issue in that the
security code always expects the mappings to have the same version as the version of the node, but
we no longer update the mappings on the security index during a rolling upgrade. We know look at
the index format to determine if the index is up to date.

Original commit: elastic/x-pack-elasticsearch@14c1c72ff6
2017-07-28 10:31:44 -06:00
Jason Tedor 826df4eb72 Fix migrate tool integration test
The command line flag --path.conf was removed yet this test was still
using it. This commit replaces the usage of this flag in this test with
the new mechanism.

Original commit: elastic/x-pack-elasticsearch@c37050894c
2017-07-28 21:33:54 +09:00
David Kyle bb360c1ff3 Mute MigrateToolIT::testRunMigrateTool
Original commit: elastic/x-pack-elasticsearch@48cb30169a
2017-07-28 11:19:08 +01:00
Michael Basnight 6a7e51d9c0 Use shaded rest client dependencies
This commit modifies all org.apache.http to use the shaded rest clients
org.elasticsearch.client.http packages. It also removes a few unused
licenses due to the change.

Relates elastic/elasticsearch#25780

Original commit: elastic/x-pack-elasticsearch@8605560232
2017-07-24 12:56:17 -05:00
Ali Beyad 5190a05b75 Improves error message on non-upgraded security index (elastic/x-pack-elasticsearch#2061)
This commit improves the error message in 6x if the security index has
not been upgraded, and warns the user that the native realm will not be
functional until the upgrade API is run.

Original commit: elastic/x-pack-elasticsearch@710b7634b4
2017-07-21 15:56:15 -04:00
jaymode 9e14bff948 Test: increase number of script compilations per minute to 20
This commit increases the number of script compilations per minute to 20 to account for security
and watcher upgrade scripts in addition to the scripts contained within cluster alerts.

Relates elastic/x-pack-elasticsearch#2041

Original commit: elastic/x-pack-elasticsearch@6c15cb9b89
2017-07-21 13:49:22 -06:00
Alexander Reelsen 139513fdd3 Tests: Fix WatchBackwardsCompatibilityIT temporarily
the upgrade API is lacking some functionality in a special case,
where triggered_watches exists, but .watches does not. This
deletes the triggered watches index manually until we integrated
this properly in the upgrade API to fix the tests

Original commit: elastic/x-pack-elasticsearch@e9d1b0d35d
2017-07-18 18:00:55 +02:00
Ali Beyad 37cc602aef Adds upgrade API functionality for security (elastic/x-pack-elasticsearch#2012)
This commit adds the upgrade API functionality and script for security.
It also enables previously muted tests that would fail due to the lack
of security upgrade features in testing cluster restarts and old
security index backward compatibility.

Original commit: elastic/x-pack-elasticsearch@4abe9f1263
2017-07-18 11:44:28 -04:00
Alexander Reelsen 8200b18e9f Tests: Allow to run BWC in isolation
The current tests were only able to finish successfully, if the earlier
tests were run as well, you could not run the restart test in isolation.

This commit ensures an upgrade is executed if needed.

Original commit: elastic/x-pack-elasticsearch@616ebbd6eb
2017-07-18 15:29:07 +02:00
Alexander Reelsen 443cfb94be Tests: Ensure waiting time between stopping and starting watcher (elastic/x-pack-elasticsearch#2008)
Otherwise we might run into race conditions that prevent a useful
start up again.

Those tests can be massively improved (no need to run against the real
master node), once the watcher BWC compatible stats are in.

relates elastic/x-pack-elasticsearch#2004

Original commit: elastic/x-pack-elasticsearch@52ca77809c
2017-07-18 10:19:33 +02:00
Tim Brooks 6d04eacdec Require elastic password be bootstrapped (elastic/x-pack-elasticsearch#1962)
This is related to elastic/x-pack-elasticsearch#1217. This commit requires that the elastic password
be bootstrapped for the user to be authenticated. As a result it removes
the special "setup" mode that allowed the user to be authenticated from
localhost.

Additionally, this commit updates the tests to work with this
functionality.

Original commit: elastic/x-pack-elasticsearch@d0d5d697a7
2017-07-13 19:59:50 -05:00
Igor Motov 4de6d9ebe5 Upgrade API: upgrade assistance shouldn't throw 404 on an empty cluster (elastic/x-pack-elasticsearch#1997)
When a user asks for upgrade information for all indices and there are no indices in the cluster, upgrade assistance should just return an empty response indicating that no indices require upgrade or reindexing. This commit also reverts the temporary fix in WatchBackwardsCompatibilityIT tests that was added as a workaround for this issue.

Original commit: elastic/x-pack-elasticsearch@2ea9707867
2017-07-13 17:01:50 -04:00
Alexander Reelsen eb118b365c Tests: Ignore 404 errors in watcher bwc tests
Original commit: elastic/x-pack-elasticsearch@8e583cf293
2017-07-13 13:41:51 +02:00
Alexander Reelsen 32bc0cd5d5 Tests: Add logging and error tracing to watcher bwc tests
Original commit: elastic/x-pack-elasticsearch@4a4b65bcb7
2017-07-13 11:08:48 +02:00
Ryan Ernst df41b8342d Convert put_template uses to put_script in rest tests
Original commit: elastic/x-pack-elasticsearch@5f5f52b955
2017-07-13 00:01:56 -07:00
Tim Vernum e4c8851a24 Convert Realm.authenticate to provide a richer result (elastic/x-pack-elasticsearch#1932)
This allows for messages to be returned, and distinguishes between 4 different results:
- I have authenticated the user
- I don't know how to authenticate that user. Try another realm.
- I tried to authenticate the user, but failed. Try another realm.
- I tried to authenticate the user, but failed. Fail the authentication attempt.

Original commit: elastic/x-pack-elasticsearch@f796949cfb
2017-07-13 14:24:08 +10:00
Simon Willnauer 84ee21ed26 Followup for elastic/elasticsearch#25658 (elastic/x-pack-elasticsearch#1984)
This is the xpack side fo elastic/elasticsearch#25658 which is mainly refactorings
of a ctor and added tests.

Original commit: elastic/x-pack-elasticsearch@d8e2a2a057
2017-07-12 22:19:34 +02:00
Alexander Reelsen e64cf23b13 Watcher: Start watcher on master node only with mixed versions (elastic/x-pack-elasticsearch#1983)
When there are data or master nodes in the cluster, that are older
than ES 6.0 alpha3, then watcher will only start on the master node.

Changed all transport actions to be master node actions, as there is
already a method to decide to run locally, which we can piggyback on.

Original commit: elastic/x-pack-elasticsearch@65cecb6d69
2017-07-12 20:58:47 +02:00
Jack Conradson 32dbfba0c2 Disallow lang to specified in requests where a stored script is used. (elastic/x-pack-elasticsearch#1949)
Requests that execute a stored script will no longer be allowed to specify the lang of the script. This information is stored in the cluster state making only an id necessary to execute against. Putting a stored script will still require a lang.

Original commit: elastic/x-pack-elasticsearch@926a7b2d86
2017-07-12 07:56:08 -07:00
Alexander Reelsen 14c12cfcde Revert "Tests: Disable all watcher BWC tests until fixed correctly"
This reverts commit elastic/x-pack-elasticsearch@8043ec1858.

Original commit: elastic/x-pack-elasticsearch@39180b095c
2017-07-11 00:40:05 +02:00
Alexander Reelsen 4dc13c3698 Tests: Disable all watcher BWC tests until fixed correctly
Original commit: elastic/x-pack-elasticsearch@8043ec1858
2017-07-10 23:12:19 +02:00
Igor Motov 37075bd201 Upgrade API: Add end-to-end bwc test for watcher upgrade (elastic/x-pack-elasticsearch#1939)
This test creates watches in old versions of elasticsearch, upgrades them after upgrading cluster to the latest version and then tests that they were upgraded correctly.

Original commit: elastic/x-pack-elasticsearch@b9d45eb2a5
2017-07-09 11:21:57 -04:00
Tim Brooks d95c365e64 Loosen setup mode restrictions for upgrade tests (elastic/x-pack-elasticsearch#1927)
This commit is related to elastic/x-pack-elasticsearch#1896. Currently setup mode means that the
password must be set post 6.0 for using x-pack. This interferes with
upgrade tests as setting the password fails without a properly
upgraded security index.

This commit loosens two aspects of the security.

1. The old default password will be accept in setup mode (requests
from localhost).
2. All request types can be submitted in setup mode.

Original commit: elastic/x-pack-elasticsearch@8a2a577038
2017-07-06 10:37:48 -05:00
Dimitrios Athanasiou 2e0560528f [TEST] Fix MlBasicMultiNodeIT after changing flush response
Relates elastic/x-pack-elasticsearch#1914

Original commit: elastic/x-pack-elasticsearch@5175bf64d9
2017-07-05 13:30:25 +01:00
Jay Modi a9707a461d Use a secure setting for the watcher encryption key (elastic/x-pack-elasticsearch#1831)
This commit removes the system key from master and changes watcher to use a secure setting instead
for the encryption key.

Original commit: elastic/x-pack-elasticsearch@5ac95c60ef
2017-06-29 14:58:35 -06:00
Tim Brooks f2cbe20ea0 Remove default passwords from reserved users (elastic/x-pack-elasticsearch#1665)
This is related to elastic/x-pack-elasticsearch#1217. This PR removes the default password of
"changeme" from the reserved users.

This PR adds special behavior for authenticating the reserved users. No
ReservedRealm user can be authenticated until its password is set. The
one exception to this is the elastic user. The elastic user can be
authenticated with an empty password if the action is a rest request
originating from localhost. In this scenario where an elastic user is
authenticated with a default password, it will have metadata indicating
that it is in setup mode. An elastic user in setup mode is only
authorized to execute a change password request.

Original commit: elastic/x-pack-elasticsearch@e1e101a237
2017-06-29 15:27:57 -05:00
Jay Modi f60c0f893c Test: add a basic rest test for CCS with non-matching remote index patterns (elastic/x-pack-elasticsearch#1866)
This commit adds a basic rest test to verify that security works with cross cluster search when a
remote pattern is provided and no remote indices match.

Relates elastic/elasticsearch#25436
relates elastic/x-pack-elasticsearch#1854

Original commit: elastic/x-pack-elasticsearch@e804d0bb12
2017-06-29 07:14:11 -06:00