Commit Graph

133 Commits

Author SHA1 Message Date
Alexander Reelsen c62f6f8177 Watcher: Distributed watch execution (elastic/x-pack-elasticsearch#544)
The distribution of watches now happens on the node which holds the
watches index, instead of on the master node. This requires several
changes to the current implementation.

1. Running on shards and replicas
   In order to run watches on the nodes with the watches index on its
   primaries and replicas. To ensure that watches do not run twice, there is
   a logic which checks the local shards, runs a murmurhash on the id and
   runs modulo against the number of shards and replicas, this is the way to
   find out, if a watch should run local. Reloading happens
2. Several master node actions moved to a HandledTransportAction, as they
   are basically just aliases for indexing actions, among them the
   put/delete/get watch actions, the acknowledgement action, the de/activate
   actions
3. Stats action moved to a broadcast node action, because we potentially
   have to query every node to get watcher statistics
4. Starting/Stopping watcher now is a master node action, which updates
   the cluster state and then listeners acts on those. Because of this watches
   can be running on two systems, if you those have different cluster state
   versions, until the new watcher state is propagated
5. Watcher is started on all nodes now. With the exception of the ticker
   schedule engine most classes do not need a lot of resources while running.
   However they have to run, because of the execute watch API, which can hit
   any node - it does not make sense to find the right shard for this watch
   and only then execute (as this also has to work with a watch, that has not
   been stored before)
6. By using a indexing operation listener, each storing of a watch now
   parses the watch first and only stores on successful parsing
7. Execute watch API now uses the watcher threadpool for execution
8. Getting the number of watches for the stats now simply queries the
   different execution engines, how many watches are scheduled, so this is
   not doing a search anymore

There will be follow up commits on this one, mainly to ensure BWC compatibility.

Original commit: elastic/x-pack-elasticsearch@0adb46e658
2017-05-02 10:12:46 +02:00
Ryan Ernst b86cdd6c8e Test: Update rest base class parameters signature (elastic/x-pack-elasticsearch#1101)
This is the xpack side of elastic/elasticsearch#21392

Original commit: elastic/x-pack-elasticsearch@b760815f54
2017-04-18 15:07:14 -07:00
Ryan Ernst ef3d3b51a4 Move integ test runner deps to cluster deps (elastic/x-pack-elasticsearch#1096)
This is the xpack side of elastic/elasticsearch#24142

Original commit: elastic/x-pack-elasticsearch@d502f06cea
2017-04-17 16:04:09 -07:00
Ali Beyad c468a4d3a3 [TEST] Security index template/mappings check on tests (elastic/x-pack-elasticsearch#1031)
Extends the security index check that ensure the template/mappings
are up-to-date to other security integration tests that depend on this
check.

relates elastic/x-pack-elasticsearch#794

Original commit: elastic/x-pack-elasticsearch@ec8e5b37bd
2017-04-11 11:22:41 -04:00
Ryan Ernst e8399cfccf Build: Build 5.x version of xpack for bwc tests to use (elastic/x-pack-elasticsearch#885)
This is analagous of the bwc-zip for elasticsearch. The one caveat is
due to the structure of how ES+xpack must be checked out, we end up with
a third clone of elasticsearch (the second being in :distribution:bwc-zip).
But the rolling upgrade integ test passes with this change.

relates elastic/x-pack-elasticsearch#870

Original commit: elastic/x-pack-elasticsearch@34bdce6e99
2017-04-07 22:19:30 -07:00
Jason Tedor f52760d2ac Revert "Silence UpgradeClusterClientYamlTestSuiteIT"
This reverts commit elastic/x-pack-elasticsearch@e29f94b24c.

Original commit: elastic/x-pack-elasticsearch@d7d27b20ef
2017-03-30 08:39:10 -04:00
Daniel Mitterdorfer 2355338625 Silence UpgradeClusterClientYamlTestSuiteIT
Relates elastic/x-pack-elasticsearch#870
Relates elastic/x-pack-kibana#690

Original commit: elastic/x-pack-elasticsearch@e29f94b24c
2017-03-29 10:38:28 +02:00
Ryan Ernst 8c01d6ea69 Tests: Add cluster health check to xpack integ wait conditions (elastic/x-pack-elasticsearch#740)
The wait condition used for integ tests by default calls the cluster
health api with wait_for_nodes nd wait_for_status. However, xpack
overrides the wait condition to add auth, but most of these conditions
still looked at the root ES url, which means the tests are susceptible
to race conditions with the check and node startup. This change modifies
the url for the authenticated wait condtion to check the health api,
with the appropriate wait_for_nodes and wait_for_status.

Original commit: elastic/x-pack-elasticsearch@0b23ef528f
2017-03-15 10:23:26 -07:00
javanna 33ccc3bd6c adapt to ObjectPath changes to support binary formats
Original commit: elastic/x-pack-elasticsearch@11782418e9
2017-02-27 12:28:04 +01:00
Ali Beyad 4a001706e7 Simplifies security index state changes and template/mapping updates (elastic/x-pack-elasticsearch#551)
Currently, both the NativeUsersStore and NativeRolesStore can undergo
multiple state transitions. This is done primarily to check if the
security index is usable before it proceeds. However, such checks are
only needed for the tests, because if the security index is unavailable
when it is needed, the downstream actions invoked by the
NativeUsersStore and NativeRolesStore will throw the appropriate
exceptions notifying of that condition. In addition, both the
NativeUsersStore and NativeRolesStore had much duplicate code that
listened for cluster state changes and made the exact same state
transitions.

This commit removes the complicated state transitions in both classes
and enables both classes to use the SecurityTemplateService to monitor
all of the security index lifecycle changes they need to be aware of.

This commit also moves the logic for determining if the security index
needs template and/or mapping updates to the SecurityLifecycleService,
and makes the NativeRealmMigrator solely responsible for applying the
updates.

Original commit: elastic/x-pack-elasticsearch@b31d144597
2017-02-23 13:05:39 -05:00
Ryan Ernst 8527bc2415 Build: Convert integ test dsl to new split cluster/runner dsl
This is the xpack side of elastic/elasticsearch#23304

Original commit: elastic/x-pack-elasticsearch@8eddd7fb0d
2017-02-22 00:56:52 -08:00
Ali Beyad b3d72af644 [TEST] bumps rolling upgrade bwc version to 5.4.0-snapshot
Original commit: elastic/x-pack-elasticsearch@2f321682d2
2017-02-15 11:00:11 -05:00
Ryan Ernst 2571921605 Rename x-pack project names to new names with split repo
Original commit: elastic/x-pack-elasticsearch@5a908f5dcc
2017-02-10 11:02:42 -08:00
Tim Vernum 734a4ee66d Prevent default passwords in production mode (elastic/elasticsearch#4724)
Adds a new `xpack.security.authc.accept_default_password` setting that defaults to `true`. If it is set to false, then the default password is not accepted in the reserved realm.

Adds a bootstrap check that the above setting must be set to `false` if security is enabled.  

Adds docs for the new setting and bootstrap.

Changed `/_enable` and `/_disable`, to store a blank password if the user record did not previously exist, which is interpreted to mean "treat this user as having the default password". The previous functionality would explicitly set the user's password to `changeme`, which would then prevent the new configuration setting from doing its job.

For any existing reserved users that had their password set to `changeme`, migrates them to the blank password (per above paragraph)

Closes: elastic/elasticsearch#4333

Original commit: elastic/x-pack-elasticsearch@db64564093
2017-02-08 16:19:55 +11:00
Jay Modi bd04b30acd security: transport always uses TLS (elastic/elasticsearch#4738)
This commit brings back support an auto-generated certificate and private key for
transport traffic. The auto-generated certificate and key can only be used in development
mode; when moving to production a key and certificate must be provided.

For the edge case of a user not wanting to encrypt their traffic, the user can set
the cipher_suites setting to `TLS_RSA_WITH_NULL_SHA256` or a like cipher, but a key/cert
is still required.

Closes elastic/elasticsearch#4332

Original commit: elastic/x-pack-elasticsearch@b7a1e629f5
2017-02-07 11:39:31 -05:00
Jason Tedor 02bcd9510f Enable disabled rolling upgrade tests
These tests were disabled due to a bug introduced when
sequence-number-based recovery was introduced. This commit enables these
tests again.

Relates elastic/elasticsearch#4801

Original commit: elastic/x-pack-elasticsearch@e555bfec1a
2017-02-03 08:12:52 -05:00
Ali Beyad 8cab4fec4b [TEST] adds AwaitsFix to failing test
Original commit: elastic/x-pack-elasticsearch@9e8f6ce047
2017-01-31 22:50:17 -05:00
Lee Hinman a75fecb472 flush after deleting docs
Original commit: elastic/x-pack-elasticsearch@1cd245c398
2017-01-10 14:26:21 -07:00
Lee Hinman 760c960c98 Revert "Revert "Add a qa/rolling-upgrade test that does single-document index and deletes""
This reverts commit elastic/x-pack@94b65dcf52.

Original commit: elastic/x-pack-elasticsearch@abd90463f8
2017-01-10 12:12:09 -07:00
Lee Hinman 05f72c0b91 Revert "Add a qa/rolling-upgrade test that does single-document index and deletes"
This reverts commit elastic/x-pack@ff05f28b7e.

Original commit: elastic/x-pack-elasticsearch@a05fc1f9e6
2017-01-10 11:19:15 -07:00
Lee Hinman 8326b6d83b Merge branch 'master' into enhancement/use_shard_bulk_for_single_ops
Original commit: elastic/x-pack-elasticsearch@98f4e74d2e
2017-01-09 14:22:18 -07:00
Jay Modi e0f0b4b7b8 rename the kibana role to kibana_system
This commit renames the kibana role to kibana_system and provides a backwards compatibility
layer so that kibana access still works properly during a rolling upgrade.

Closes elastic/elasticsearch#4525

Original commit: elastic/x-pack-elasticsearch@5c5796e53a
2017-01-09 16:06:50 -05:00
Lee Hinman 93720505b8 Merge branch 'master' into enhancement/use_shard_bulk_for_single_ops
Original commit: elastic/x-pack-elasticsearch@089fa9977d
2017-01-09 11:39:37 -07:00
Lee Hinman 7387d04139 Add a qa/rolling-upgrade test that does single-document index and deletes
Original commit: elastic/x-pack-elasticsearch@5850439b22
2017-01-09 11:16:33 -07:00
Nik Everett 4ff6279865 Remove exception from client suite
Original commit: elastic/x-pack-elasticsearch@51e94561d9
2016-12-22 09:10:42 -05:00
Ryan Ernst 6d4d599f91 Build: Convert xplugins to use new extra projects setup (elastic/elasticsearch#4175)
* Build: Convert xplugins to use new extra projects setup

This change makes the gradle initialization for xplugins look in the
correct location for elasticsearch, which is now as a sibling of an
elasticsearch-extra directory, with x-plugins as a child of the extra
directory.

The elasticsearch side of this change is
elastic/elasticsearchelastic/elasticsearch#21773. This change will enable renaming x-plugins
to x-pack, see elastic/elasticsearch#3643.

Original commit: elastic/x-pack-elasticsearch@09398aea5a
2016-12-14 15:02:28 -08:00
Jay Modi e8836f99b0 test: add tests that exercise the security index during rolling upgrades
This commit adds basic tests that store a user and a role using the native API. The test checks
that the user and role can be used prior to starting the upgrade. The realm and roles caches are
also cleared to ensure the next authentication will require a read from the security index; this
ensures we are actually testing reads from the index.

Original commit: elastic/x-pack-elasticsearch@396862da94
2016-12-01 11:15:15 -05:00
Yannick Welsch c7d7a2bafc [TEST] Wait in rolling_upgrade rest test for monitoring indices to be fully allocated
Fixes previous commit elastic/x-pack@8bb4e6a so that it also accounts for monitoring indices that are automatically created by x-pack while the test is running.

Original commit: elastic/x-pack-elasticsearch@e50e1a2717
2016-11-25 12:54:41 +01:00
Yannick Welsch 16b624b1e4 [TEST] Wait in rolling_upgrade rest test for old cluster to have all indices fully allocated
When one of the 2 nodes in the old cluster is shut down, shards that were on that node will become unassigned and be marked to be
delay-allocated, i.e. either a node with shard data for that shard must be available or the allocation of the shards will be delayed for a minute.
In the mixed cluster the replica shard might not be allocated as the primary is already on the node with the newer version and replicas are not allowed
then to be allocated to a node of an older version of ES. Once both nodes are upgraded, the delay might still be in place, and can only be nullified if there
is shard data available on the node. If there never was a shard on that node though, it will take a minute and run into the timeout checking for green.

This commit ensures that all shards are fully-allocated before we do the rolling restart scenario

Original commit: elastic/x-pack-elasticsearch@a0d9b1b043
2016-11-24 16:43:08 +01:00
Yannick Welsch b503f05916 [Test] Fix rolling upgrade test to only wait for yellow in a mixed cluster
If the primary shard of an index with (number_of_replicas > 0) ends up on a new node in a mixed cluster, the replica cannot be allocated to the old node as
the new node might have written segments that use a new postings format or codec that is not available on the older node.
As x-pack automatically creates indices with number_of_replicas > 0, for example monitoring-data-*, the test can only wait for yellow in a mixed cluster.

Original commit: elastic/x-pack-elasticsearch@945d9e3811
2016-11-23 14:36:48 +01:00
jaymode 9126600fc3 test: add timeout to cluster health call that we are waiting on
This commit adds a timeout to the cluster health call that we wait on so that we can
see the status of the health request instead of getting timeouts failures with no
information to go on.

Original commit: elastic/x-pack-elasticsearch@2f34d01e00
2016-11-22 15:17:21 -05:00
jaymode 6170f3d22c test: use the right number of bwc nodes in old cluster
Original commit: elastic/x-pack-elasticsearch@0afc045bbc
2016-11-22 15:14:29 -05:00
Jay Modi 4239ba5415 allow reads of native users and roles when template version hasn't been updated
This change allows reads of our native users and roles when the template version has not been updated to
match the current version. This is useful for rolling upgrades where the nodes are also being actively
queried and/or indexed into. Without this, we can wreak havoc on a cluster by causing exceptions during
replication, which leads to shard failures. On nodes that match the version defined in the template,
write operations are allowed since we know that we are backwards compatible in terms of format but we
may have added new fields and shouldn't index them until the mappings and template have been updated.

As part of this, the rolling upgrade tests from core were used as the basis for a very basic set of tests
for doing a rolling upgrade with x-pack.

Closes elastic/elasticsearch#4126

Original commit: elastic/x-pack-elasticsearch@9be518ef00
2016-11-22 12:00:09 -05:00