the upgrade API is lacking some functionality in a special case,
where triggered_watches exists, but .watches does not. This
deletes the triggered watches index manually until we integrated
this properly in the upgrade API to fix the tests
Original commit: elastic/x-pack-elasticsearch@e9d1b0d35d
This commit adds the upgrade API functionality and script for security.
It also enables previously muted tests that would fail due to the lack
of security upgrade features in testing cluster restarts and old
security index backward compatibility.
Original commit: elastic/x-pack-elasticsearch@4abe9f1263
The current tests were only able to finish successfully, if the earlier
tests were run as well, you could not run the restart test in isolation.
This commit ensures an upgrade is executed if needed.
Original commit: elastic/x-pack-elasticsearch@616ebbd6eb
Otherwise we might run into race conditions that prevent a useful
start up again.
Those tests can be massively improved (no need to run against the real
master node), once the watcher BWC compatible stats are in.
relates elastic/x-pack-elasticsearch#2004
Original commit: elastic/x-pack-elasticsearch@52ca77809c
This is related to elastic/x-pack-elasticsearch#1217. This commit requires that the elastic password
be bootstrapped for the user to be authenticated. As a result it removes
the special "setup" mode that allowed the user to be authenticated from
localhost.
Additionally, this commit updates the tests to work with this
functionality.
Original commit: elastic/x-pack-elasticsearch@d0d5d697a7
When a user asks for upgrade information for all indices and there are no indices in the cluster, upgrade assistance should just return an empty response indicating that no indices require upgrade or reindexing. This commit also reverts the temporary fix in WatchBackwardsCompatibilityIT tests that was added as a workaround for this issue.
Original commit: elastic/x-pack-elasticsearch@2ea9707867
When there are data or master nodes in the cluster, that are older
than ES 6.0 alpha3, then watcher will only start on the master node.
Changed all transport actions to be master node actions, as there is
already a method to decide to run locally, which we can piggyback on.
Original commit: elastic/x-pack-elasticsearch@65cecb6d69
This commit is related to elastic/x-pack-elasticsearch#1896. Currently setup mode means that the
password must be set post 6.0 for using x-pack. This interferes with
upgrade tests as setting the password fails without a properly
upgraded security index.
This commit loosens two aspects of the security.
1. The old default password will be accept in setup mode (requests
from localhost).
2. All request types can be submitted in setup mode.
Original commit: elastic/x-pack-elasticsearch@8a2a577038
This commit removes the system key from master and changes watcher to use a secure setting instead
for the encryption key.
Original commit: elastic/x-pack-elasticsearch@5ac95c60ef
This is related to elastic/x-pack-elasticsearch#1217. This PR removes the default password of
"changeme" from the reserved users.
This PR adds special behavior for authenticating the reserved users. No
ReservedRealm user can be authenticated until its password is set. The
one exception to this is the elastic user. The elastic user can be
authenticated with an empty password if the action is a rest request
originating from localhost. In this scenario where an elastic user is
authenticated with a default password, it will have metadata indicating
that it is in setup mode. An elastic user in setup mode is only
authorized to execute a change password request.
Original commit: elastic/x-pack-elasticsearch@e1e101a237
The .security index used several different types to differentiate the
documents added to the index (users, reserved-users, roles, etc). Since
types are deprecated in 6.x, this commit changes the .security index
access layer to only use a single type and have all documents in the
index be of that single type. To differentiate documents that may have
the same id (e.g. the same user name and role name), the appropriate
type of the document is prepended to the id. For example, a user named
"jdoe" will now have the document id "user-jdoe".
This commit also ensures that any native realm security index operations
that lead to auto creation of the security index first go through the process
of creating the internal security index (.security-v6) and creating the alias
.security to point to the internal index.
Lastly, anytime the security index is accessed without having been
upgraded, an exception is thrown notifying the user to use the
upgrade API to upgrade the security index.
Original commit: elastic/x-pack-elasticsearch@cc0a474aed
This commit adds new settings for the ssl keystore (not the ES keystore)
passphrase settings. New setting names are used, instead of trying to
support the existing names in both yml and the ES keystore, so that
there does not need to be complicated logic between the two. Note that
the old settings remain the only way to set the ssl passphrases for the
transport client, but the Settings object for transport clients are
created in memory by users, so they are already as "secure" as having a
loaded ES keystore. Also note that in the long term future (6.x
timeframe?) these settings should be deprecated and the keys/certs
themselves should be moved into the ES keystore, so there will be no
need for separate keystores/passphrases.
relates elastic/elasticsearch#22475
Original commit: elastic/x-pack-elasticsearch@be5275fa3d
Adds tests similar to `:qa:full-cluster-restart` for x-pack. You
run them with `gradle :x-pack:qa:full-cluster-restart:check`.
The actual tests are as basic as it gets: create a doc and load it,
shut down, upgrade to master, startup, and load it. Create a user
and load it, shut down, upgrade to master, startup, and load it.
Relates to elastic/x-pack-elasticsearch#1629
Original commit: elastic/x-pack-elasticsearch@8994bec8e7
This commit fixes the default password migration tests that had been failing reproducibly. The
first fix skips tests using the set enabled api when running against a version prior to 5.1.2 as
this api would otherwise trip an assertion that the xcontent builder was not closed. The second
fix is to ensure we set the password field in the user object.
relates elastic/x-pack-elasticsearch#1529
relates elastic/x-pack-elasticsearch#1516
Original commit: elastic/x-pack-elasticsearch@2f9c804309
This commit increases the amount of time to wait for green cluster health during a rolling upgrade
to account for the time that may be needed in the case of delayed shards. Additionally some old
timeout values were removed as they were used due to the default timeout of 30s.
Relates elastic/x-pack-elasticsearch#1683
Original commit: elastic/x-pack-elasticsearch@9996673db0
Real upgrades will have preserved their templates, so rolling upgrade tests should preserve them as well and internal services should be expected to replace them as needed.
Original commit: elastic/x-pack-elasticsearch@93a155951e
We wait for index templates to be installed before running tests, but
these can get upgraded when the master node is upgraded. If we don't
wait again in this case then tests can fail due to the cleanup code
being overwhelmed by pending cluster state updates that weren't waited
for before the test.
Original commit: elastic/x-pack-elasticsearch@eff0b24c11
Reworks the rolling restart tests so that all configuration
options share code. Now there is a project per configuration:
* `qa:rolling-upgrade:with-ssl-with-system-key`
* `qa:rolling-upgrade:with-ssl-without-system-key`
Original commit: elastic/x-pack-elasticsearch@5672b4a4f2
This commit adds better security for scroll requests in that they are now tied to a single user as
we only authorize the request that creates the scroll. This is accomplished by adding a
SearchOperationListener that listens for new scroll contexts and stores the authentication on the
ScrollContext. Then upon
retrieval of the search context for a query or fetch, the current authentication is compared to the
authentication that was present when the scroll context was created. If the current authentication
belongs to a different user, then a SearchContextMissingException will be thrown to prevent leaking
a valid vs invalid scroll id.
Additionally, signing of a scroll id is only performed when there is a older node in the cluster
that would expect the scroll id to be signed. Once this is backported to 5.x, we can remove this
bwc layer for 6.0/master.
Original commit: elastic/x-pack-elasticsearch@0e5dcafd32
The kibana user role in mixed cluster test expects a old cluster test to have run first that would
disable the kibana user, but that test is not executed on 5.1.1 and prior versions. This change
makes a update request an upsert to account for this.
relates elastic/x-pack-elasticsearch#1527
Original commit: elastic/x-pack-elasticsearch@df2945610e
Reworks the rolling restart tests so they'd have caugh an
incompatibility in the wire protocol that we say between 5.4.0 and
5.4.1.
Original commit: elastic/x-pack-elasticsearch@f5e69cf58e
The rolling upgrade nodes need a keystore for SSL configuration but
there was no dependency on the task that copies the keystore into the
output directory for the nodes to pick up as an extra configuration
file. This commit addresses this by adding such a dependency. To do
this, we need to break the dependency of the keystore copy task on the
REST spec copy task; this is not an issue since the dependency was for
convenience of ordering the task and not actually needed.
Original commit: elastic/x-pack-elasticsearch@fddbc06e9f
This commit marks the default password migration test as awaits fix as
the test is reliabily failling.
Original commit: elastic/x-pack-elasticsearch@4377480d98
The authentication object was changed in 5.4.0 in that it was conditionally signed depending on
the version and other factors. A bug was introduced however that causes the authentication to
actually get written with the version of the node it is being sent to even if that version is
greater than the version of the current node, which causes rolling upgrades to fail.
Original commit: elastic/x-pack-elasticsearch@a718ff8a52
This commit fixes the kibana write rolling upgrade test. The test needs to enable the kibana user
as another test explicitly disables it. After the test runs it disables the kibana user again.
relates elastic/x-pack-elasticsearch#1460
Original commit: elastic/x-pack-elasticsearch@437495432d
As fields with underscores will be disallowed in master, and we have to
prepare the upgrade, this commit renames the _status field to status.
When the 5.x upgrade logic is in place in the 5.x we can remove all the
old style _status handling from the master branch.
Note: All the BWC compatibility tests, that load 5.x indices are now
faking a finished upgrade by adding the `status` field to the mapping
of the watches index.
Original commit: elastic/x-pack-elasticsearch@9d5cc9aaec
The distribution of watches now happens on the node which holds the
watches index, instead of on the master node. This requires several
changes to the current implementation.
1. Running on shards and replicas
In order to run watches on the nodes with the watches index on its
primaries and replicas. To ensure that watches do not run twice, there is
a logic which checks the local shards, runs a murmurhash on the id and
runs modulo against the number of shards and replicas, this is the way to
find out, if a watch should run local. Reloading happens
2. Several master node actions moved to a HandledTransportAction, as they
are basically just aliases for indexing actions, among them the
put/delete/get watch actions, the acknowledgement action, the de/activate
actions
3. Stats action moved to a broadcast node action, because we potentially
have to query every node to get watcher statistics
4. Starting/Stopping watcher now is a master node action, which updates
the cluster state and then listeners acts on those. Because of this watches
can be running on two systems, if you those have different cluster state
versions, until the new watcher state is propagated
5. Watcher is started on all nodes now. With the exception of the ticker
schedule engine most classes do not need a lot of resources while running.
However they have to run, because of the execute watch API, which can hit
any node - it does not make sense to find the right shard for this watch
and only then execute (as this also has to work with a watch, that has not
been stored before)
6. By using a indexing operation listener, each storing of a watch now
parses the watch first and only stores on successful parsing
7. Execute watch API now uses the watcher threadpool for execution
8. Getting the number of watches for the stats now simply queries the
different execution engines, how many watches are scheduled, so this is
not doing a search anymore
There will be follow up commits on this one, mainly to ensure BWC compatibility.
Original commit: elastic/x-pack-elasticsearch@0adb46e658
This is analagous of the bwc-zip for elasticsearch. The one caveat is
due to the structure of how ES+xpack must be checked out, we end up with
a third clone of elasticsearch (the second being in :distribution:bwc-zip).
But the rolling upgrade integ test passes with this change.
relates elastic/x-pack-elasticsearch#870
Original commit: elastic/x-pack-elasticsearch@34bdce6e99
The wait condition used for integ tests by default calls the cluster
health api with wait_for_nodes nd wait_for_status. However, xpack
overrides the wait condition to add auth, but most of these conditions
still looked at the root ES url, which means the tests are susceptible
to race conditions with the check and node startup. This change modifies
the url for the authenticated wait condtion to check the health api,
with the appropriate wait_for_nodes and wait_for_status.
Original commit: elastic/x-pack-elasticsearch@0b23ef528f
Currently, both the NativeUsersStore and NativeRolesStore can undergo
multiple state transitions. This is done primarily to check if the
security index is usable before it proceeds. However, such checks are
only needed for the tests, because if the security index is unavailable
when it is needed, the downstream actions invoked by the
NativeUsersStore and NativeRolesStore will throw the appropriate
exceptions notifying of that condition. In addition, both the
NativeUsersStore and NativeRolesStore had much duplicate code that
listened for cluster state changes and made the exact same state
transitions.
This commit removes the complicated state transitions in both classes
and enables both classes to use the SecurityTemplateService to monitor
all of the security index lifecycle changes they need to be aware of.
This commit also moves the logic for determining if the security index
needs template and/or mapping updates to the SecurityLifecycleService,
and makes the NativeRealmMigrator solely responsible for applying the
updates.
Original commit: elastic/x-pack-elasticsearch@b31d144597
Adds a new `xpack.security.authc.accept_default_password` setting that defaults to `true`. If it is set to false, then the default password is not accepted in the reserved realm.
Adds a bootstrap check that the above setting must be set to `false` if security is enabled.
Adds docs for the new setting and bootstrap.
Changed `/_enable` and `/_disable`, to store a blank password if the user record did not previously exist, which is interpreted to mean "treat this user as having the default password". The previous functionality would explicitly set the user's password to `changeme`, which would then prevent the new configuration setting from doing its job.
For any existing reserved users that had their password set to `changeme`, migrates them to the blank password (per above paragraph)
Closes: elastic/elasticsearch#4333
Original commit: elastic/x-pack-elasticsearch@db64564093
This commit brings back support an auto-generated certificate and private key for
transport traffic. The auto-generated certificate and key can only be used in development
mode; when moving to production a key and certificate must be provided.
For the edge case of a user not wanting to encrypt their traffic, the user can set
the cipher_suites setting to `TLS_RSA_WITH_NULL_SHA256` or a like cipher, but a key/cert
is still required.
Closeselastic/elasticsearch#4332
Original commit: elastic/x-pack-elasticsearch@b7a1e629f5
* Build: Convert xplugins to use new extra projects setup
This change makes the gradle initialization for xplugins look in the
correct location for elasticsearch, which is now as a sibling of an
elasticsearch-extra directory, with x-plugins as a child of the extra
directory.
The elasticsearch side of this change is
elastic/elasticsearchelastic/elasticsearch#21773. This change will enable renaming x-plugins
to x-pack, see elastic/elasticsearch#3643.
Original commit: elastic/x-pack-elasticsearch@09398aea5a
This commit adds basic tests that store a user and a role using the native API. The test checks
that the user and role can be used prior to starting the upgrade. The realm and roles caches are
also cleared to ensure the next authentication will require a read from the security index; this
ensures we are actually testing reads from the index.
Original commit: elastic/x-pack-elasticsearch@396862da94
When one of the 2 nodes in the old cluster is shut down, shards that were on that node will become unassigned and be marked to be
delay-allocated, i.e. either a node with shard data for that shard must be available or the allocation of the shards will be delayed for a minute.
In the mixed cluster the replica shard might not be allocated as the primary is already on the node with the newer version and replicas are not allowed
then to be allocated to a node of an older version of ES. Once both nodes are upgraded, the delay might still be in place, and can only be nullified if there
is shard data available on the node. If there never was a shard on that node though, it will take a minute and run into the timeout checking for green.
This commit ensures that all shards are fully-allocated before we do the rolling restart scenario
Original commit: elastic/x-pack-elasticsearch@a0d9b1b043
If the primary shard of an index with (number_of_replicas > 0) ends up on a new node in a mixed cluster, the replica cannot be allocated to the old node as
the new node might have written segments that use a new postings format or codec that is not available on the older node.
As x-pack automatically creates indices with number_of_replicas > 0, for example monitoring-data-*, the test can only wait for yellow in a mixed cluster.
Original commit: elastic/x-pack-elasticsearch@945d9e3811
This commit adds a timeout to the cluster health call that we wait on so that we can
see the status of the health request instead of getting timeouts failures with no
information to go on.
Original commit: elastic/x-pack-elasticsearch@2f34d01e00
This change allows reads of our native users and roles when the template version has not been updated to
match the current version. This is useful for rolling upgrades where the nodes are also being actively
queried and/or indexed into. Without this, we can wreak havoc on a cluster by causing exceptions during
replication, which leads to shard failures. On nodes that match the version defined in the template,
write operations are allowed since we know that we are backwards compatible in terms of format but we
may have added new fields and shouldn't index them until the mappings and template have been updated.
As part of this, the rolling upgrade tests from core were used as the basis for a very basic set of tests
for doing a rolling upgrade with x-pack.
Closeselastic/elasticsearch#4126
Original commit: elastic/x-pack-elasticsearch@9be518ef00
Alexander Reelsen
Ali Beyad
Tim Brooks
Igor Motov
Jay Modi
Ryan Ernst
Nik Everett
jaymode
Jason Tedor
Boaz Leskes
Chris Earle
David Roberts
Chris Earle
David Kyle
Simon Willnauer
Daniel Mitterdorfer
javanna
Tim Vernum
Lee Hinman
Yannick Welsch