Fixes to the build system, particularly around BWC testing, and to make future
version bumps less painful.
Original commit: elastic/x-pack-elasticsearch@a1d456f30a
This adds a rolling upgrade test for X-Pack monitoring. It works by using the `_xpack/monitoring/_bulk` endpoint to send arbitrary data, then verify that it exists.
This forces a few things to happen, thereby testing the behavior:
1. The templates must exist.
2. The elected master node must be "ready" to work (hence the first
point).
3. The same "system_api_version" is accepted by every version of ES.
Original commit: elastic/x-pack-elasticsearch@012e5738bb
This change fixes the check for the version of the security template after the template updater was
changed to only run on the master node in elastic/elasticsearch#27294. Additionally, the wait time
for the cluster to have a yellow status has been increased to account for delayed shards and slower
machines.
Original commit: elastic/x-pack-elasticsearch@a2e72bed12
This commit adds back the ability to disable TLS on the transport layer and also disables TLS by
default to restore the 5.x behavior. The auto generation of key/cert and bundled CA certificate
have also been removed.
Relates elastic/x-pack-elasticsearch#2463
Original commit: elastic/x-pack-elasticsearch@abc66ec67d
`authc.token.enabled` is true unless `http.ssl.enabled` is `false` and `http.enabled` is `true`.
* TokenService default enabled if HTTP_ENABLED == false
* Fixed tests that need TokenService explicitly enabled
* [DOC] Default value for `xpack.security.authc.token.enabled`
Original commit: elastic/x-pack-elasticsearch@bd154d16eb
Today we require a pre-shared key to use the token service. Beside the
additional setup step it doesn't allow for key-rotation which is a major downside.
This change adds a TokenService private ClusterState.Custom that is used to distribute
the keys used to encrypt tokens. It also has the infrastructur to add automatic key
rotation which is not in use yet but included here to illustrate how it can work down
the road.
This is considered a prototype and requires additioanl integration testing. Yet, it's fully
BWC with a rolling / full cluster restart from a previous version (also from 5.6 to 6.x)
since if the password is set it will just use it instead of generating a new one.
Once we implement the automatic key rotation via the clusterstate we need to ensure that we are
fully upgraded before we do that.
Also note that the ClusterState.Custom is fully transient and will never be serialized to disk.
Original commit: elastic/x-pack-elasticsearch@1ae22f5d41
This is related to elastic/x-pack-elasticsearch#1217. This commit requires that the elastic password
be bootstrapped for the user to be authenticated. As a result it removes
the special "setup" mode that allowed the user to be authenticated from
localhost.
Additionally, this commit updates the tests to work with this
functionality.
Original commit: elastic/x-pack-elasticsearch@d0d5d697a7
This commit is related to elastic/x-pack-elasticsearch#1896. Currently setup mode means that the
password must be set post 6.0 for using x-pack. This interferes with
upgrade tests as setting the password fails without a properly
upgraded security index.
This commit loosens two aspects of the security.
1. The old default password will be accept in setup mode (requests
from localhost).
2. All request types can be submitted in setup mode.
Original commit: elastic/x-pack-elasticsearch@8a2a577038
This commit removes the system key from master and changes watcher to use a secure setting instead
for the encryption key.
Original commit: elastic/x-pack-elasticsearch@5ac95c60ef
This is related to elastic/x-pack-elasticsearch#1217. This PR removes the default password of
"changeme" from the reserved users.
This PR adds special behavior for authenticating the reserved users. No
ReservedRealm user can be authenticated until its password is set. The
one exception to this is the elastic user. The elastic user can be
authenticated with an empty password if the action is a rest request
originating from localhost. In this scenario where an elastic user is
authenticated with a default password, it will have metadata indicating
that it is in setup mode. An elastic user in setup mode is only
authorized to execute a change password request.
Original commit: elastic/x-pack-elasticsearch@e1e101a237
This commit adds new settings for the ssl keystore (not the ES keystore)
passphrase settings. New setting names are used, instead of trying to
support the existing names in both yml and the ES keystore, so that
there does not need to be complicated logic between the two. Note that
the old settings remain the only way to set the ssl passphrases for the
transport client, but the Settings object for transport clients are
created in memory by users, so they are already as "secure" as having a
loaded ES keystore. Also note that in the long term future (6.x
timeframe?) these settings should be deprecated and the keys/certs
themselves should be moved into the ES keystore, so there will be no
need for separate keystores/passphrases.
relates elastic/elasticsearch#22475
Original commit: elastic/x-pack-elasticsearch@be5275fa3d
Adds tests similar to `:qa:full-cluster-restart` for x-pack. You
run them with `gradle :x-pack:qa:full-cluster-restart:check`.
The actual tests are as basic as it gets: create a doc and load it,
shut down, upgrade to master, startup, and load it. Create a user
and load it, shut down, upgrade to master, startup, and load it.
Relates to elastic/x-pack-elasticsearch#1629
Original commit: elastic/x-pack-elasticsearch@8994bec8e7
Reworks the rolling restart tests so that all configuration
options share code. Now there is a project per configuration:
* `qa:rolling-upgrade:with-ssl-with-system-key`
* `qa:rolling-upgrade:with-ssl-without-system-key`
Original commit: elastic/x-pack-elasticsearch@5672b4a4f2
This commit adds better security for scroll requests in that they are now tied to a single user as
we only authorize the request that creates the scroll. This is accomplished by adding a
SearchOperationListener that listens for new scroll contexts and stores the authentication on the
ScrollContext. Then upon
retrieval of the search context for a query or fetch, the current authentication is compared to the
authentication that was present when the scroll context was created. If the current authentication
belongs to a different user, then a SearchContextMissingException will be thrown to prevent leaking
a valid vs invalid scroll id.
Additionally, signing of a scroll id is only performed when there is a older node in the cluster
that would expect the scroll id to be signed. Once this is backported to 5.x, we can remove this
bwc layer for 6.0/master.
Original commit: elastic/x-pack-elasticsearch@0e5dcafd32
Reworks the rolling restart tests so they'd have caugh an
incompatibility in the wire protocol that we say between 5.4.0 and
5.4.1.
Original commit: elastic/x-pack-elasticsearch@f5e69cf58e
The rolling upgrade nodes need a keystore for SSL configuration but
there was no dependency on the task that copies the keystore into the
output directory for the nodes to pick up as an extra configuration
file. This commit addresses this by adding such a dependency. To do
this, we need to break the dependency of the keystore copy task on the
REST spec copy task; this is not an issue since the dependency was for
convenience of ordering the task and not actually needed.
Original commit: elastic/x-pack-elasticsearch@fddbc06e9f
The authentication object was changed in 5.4.0 in that it was conditionally signed depending on
the version and other factors. A bug was introduced however that causes the authentication to
actually get written with the version of the node it is being sent to even if that version is
greater than the version of the current node, which causes rolling upgrades to fail.
Original commit: elastic/x-pack-elasticsearch@a718ff8a52
This commit fixes the kibana write rolling upgrade test. The test needs to enable the kibana user
as another test explicitly disables it. After the test runs it disables the kibana user again.
relates elastic/x-pack-elasticsearch#1460
Original commit: elastic/x-pack-elasticsearch@437495432d
This is analagous of the bwc-zip for elasticsearch. The one caveat is
due to the structure of how ES+xpack must be checked out, we end up with
a third clone of elasticsearch (the second being in :distribution:bwc-zip).
But the rolling upgrade integ test passes with this change.
relates elastic/x-pack-elasticsearch#870
Original commit: elastic/x-pack-elasticsearch@34bdce6e99
The wait condition used for integ tests by default calls the cluster
health api with wait_for_nodes nd wait_for_status. However, xpack
overrides the wait condition to add auth, but most of these conditions
still looked at the root ES url, which means the tests are susceptible
to race conditions with the check and node startup. This change modifies
the url for the authenticated wait condtion to check the health api,
with the appropriate wait_for_nodes and wait_for_status.
Original commit: elastic/x-pack-elasticsearch@0b23ef528f
This commit brings back support an auto-generated certificate and private key for
transport traffic. The auto-generated certificate and key can only be used in development
mode; when moving to production a key and certificate must be provided.
For the edge case of a user not wanting to encrypt their traffic, the user can set
the cipher_suites setting to `TLS_RSA_WITH_NULL_SHA256` or a like cipher, but a key/cert
is still required.
Closeselastic/elasticsearch#4332
Original commit: elastic/x-pack-elasticsearch@b7a1e629f5
* Build: Convert xplugins to use new extra projects setup
This change makes the gradle initialization for xplugins look in the
correct location for elasticsearch, which is now as a sibling of an
elasticsearch-extra directory, with x-plugins as a child of the extra
directory.
The elasticsearch side of this change is
elastic/elasticsearchelastic/elasticsearch#21773. This change will enable renaming x-plugins
to x-pack, see elastic/elasticsearch#3643.
Original commit: elastic/x-pack-elasticsearch@09398aea5a
This commit adds basic tests that store a user and a role using the native API. The test checks
that the user and role can be used prior to starting the upgrade. The realm and roles caches are
also cleared to ensure the next authentication will require a read from the security index; this
ensures we are actually testing reads from the index.
Original commit: elastic/x-pack-elasticsearch@396862da94
This change allows reads of our native users and roles when the template version has not been updated to
match the current version. This is useful for rolling upgrades where the nodes are also being actively
queried and/or indexed into. Without this, we can wreak havoc on a cluster by causing exceptions during
replication, which leads to shard failures. On nodes that match the version defined in the template,
write operations are allowed since we know that we are backwards compatible in terms of format but we
may have added new fields and shouldn't index them until the mappings and template have been updated.
As part of this, the rolling upgrade tests from core were used as the basis for a very basic set of tests
for doing a rolling upgrade with x-pack.
Closeselastic/elasticsearch#4126
Original commit: elastic/x-pack-elasticsearch@9be518ef00
David Turner
Chris Earle
jaymode
Yannick Welsch
Jay Modi
Albert Zaharovits
Simon Willnauer
Ryan Ernst
Tim Brooks
Nik Everett
Jason Tedor
Ali Beyad
Daniel Mitterdorfer