This change migrates xpack (security, watcher, and monitoring) to use the common ssl
configuration for the elastic stack. As part of this work, several aspects of how we deal
with SSL has been modified.
From a functionality perspective, an xpack wide configuration for SSL was added and
all of the code that needs SSL uses the SSLService now. The following is a list of all
of the aspects of xpack that can have their own SSL configuration, which are separate
from the xpack wide configuration:
* Transport
* Transport profiles
* HTTP Transport
* Realms
* Monitoring Exporters
* HTTP Client
In terms of the code, some cleanups were made with these changes. SSLConfiguration is
now a concrete class and SSLConfiguration.Custom and SSLConfiguration.Global have been
removed. The validate method on key and trust configurations has been removed and these
classes will now throw exceptions when they are constructed with bad values. The
OptionalSettings helper class has been removed as it was just a file with one line functions
that made the code harder to understand. The SSL configuration and service classes have
been moved from the security source directories to the main xpack source set. The SSLService
now handles more of the configuration of the SSLEngine it returns to prevent callers from
having to handle those aspects. The settings that get registered for SSL have been moved to
XPackSettings.
Also included in this PR is a update to the docs around SSL. This includes a large simplification to
the documentation in that the certificate authority configuration section has been removed and the
process that is documented for generating certificates only includes the CLI tool that we bundle.
Closeselastic/elasticsearch#3104Closeselastic/elasticsearch#2971Closeselastic/elasticsearch#3164
Original commit: elastic/x-pack-elasticsearch@5bd9e5ef38
After this commit, we have "security3" and "security4" as possible
transport and HTTP settings, we default to "security4" if it is not set,
and we randomize the selection in some of the integration tests.
Original commit: elastic/x-pack-elasticsearch@e56718911a
This changes the xpack.monitoring.agent.* settings so that they fall under xpack.monitoring.collection.*.
- This also changes Kibana's equivalent data_collection to collection.
Original commit: elastic/x-pack-elasticsearch@a69bc18055
We still have a wrapper called RestTestClient that is very specific to Rest tests, as well as RestTestResponse etc. but all the low level bits around http connections etc. are now handled by RestClient.
Original commit: elastic/x-pack-elasticsearch@304487ecc5
This commit adds the necessary changes to make SSL work on the transport layer by default. A large
portion of the SSL configuration/settings was re-worked with this change. Some notable highlights
include support for PEM cert/keys, reloadable SSL configuration, separate HTTP ssl configuration, and
separate LDAP configuration.
The following is a list of specific items addressed:
* `SSLSettings` renamed to `SSLConfiguration`
* `KeyConfig` and `TrustConfig` abstractions created. These hide the details of how `KeyManager[]` and `TrustManager[]` are loaded. These are also responsible for settings validation (ie keystore password is not null)
* Configuration fallback is changed. Previously any setting would fallback to the "global" value (`xpack.security.ssl.*`). Now a keystore path, key path, ca paths, or truststore path must be specified otherwise the configuration for that key/trust will fallback to the global configuration. In other words if you want to change part of a keystore or truststore in a profile you need to supply all the information. This could be considered breaking if a user relied on the old fallback
* JDK trusted certificates (`cacerts`) are trusted by default (breaking change). This can be disabled via a setting.
* We now monitor the SSL files for changes and enable dynamic reloading of the configuration. This will make it easier for users when they are getting set up with certificates so they do not need to restart every time. This can be disabled via a setting
* LDAP realms can now have their own SSL configurations
* HTTP can now have its own SSL configuration
* SSL is enabled by default on the transport layer only. Hostname verification is enabled as well. On startup if no global SSL settings are present and SSL is configured to be used, we auto generate one based on the default CA that is shipped. This process includes a best effort attempt to generate the subject alternative names.
* `xpack.security.ssl.hostname_verification` is deprecated in favor of `xpack.security.ssl.hostname_verification.enabled`
* added Bouncy Castle info to NOTICE
* consolidated NOTICE and LICENSE files
Closeselastic/elasticsearch#14Closeselastic/elasticsearch#34Closeselastic/elasticsearch#1483Closeselastic/elasticsearch#1933
Addresses security portion of elastic/elasticsearch#673
Original commit: elastic/x-pack-elasticsearch@7c359db90b
This commit adds reserved or built-in user and role support to x-pack. The reserved roles
cannot be modified by users. The reserved users also cannot be modified with the exception
of changing the password for a user.
In order to change the password for a user, a new API has been added. This API only supports
changing passwords for native and reserved users.
To support allowing a user to change their own password, a default role has been added to grant
access. This default role only grants access to user operations that pertain to the user that is
being authorized. In other words, the default role grants `joe` the ability to change their own password
but does not allow them to change the password of a different user.
Additionally, the authenticate API was made a transport action and is granted by the default role.
Closeselastic/elasticsearch#1727Closeselastic/elasticsearch#1185Closeselastic/elasticsearch#1158
Original commit: elastic/x-pack-elasticsearch@1a6689d90f
This commit migrates all of the `shield.` settings to `xpack.security.` and makes changes to
use the new Settings infrastructure in core.
As a cleanup, this commit also renames Shield to Security since this class is only in master
and will not affect 2.x.
See elastic/elasticsearch#1441
Original commit: elastic/x-pack-elasticsearch@a5a9798b1b
This commit fixes an issue in synchronization in Exporters class. The export() method is synchronized and when used with LocalExport can provoke a deadlock. LocalExporter exports data locally using bulk requests that can trigger cluster state updates for mapping updates. If a exporters settings update sneaks in, the settings update waits for the export to terminate but the export waits for the settings to be updated... and boom.
This commit removes the synchronized and refactor Exporters/LocalExporter to use state and dedicated instance of LocalBulk for each export so that synchronizing methods is not necessary anymore.
It also lower down some random settings in MonitoringBulkTests because the previous settings almost always fill the bulk thread pool.
closeselastic/elasticsearch#1769
Original commit: elastic/x-pack-elasticsearch@f50c916f8b
This commit is the forward port of renaming the type for esusers to file. There is no
backwards compatibility maintained here. Additionally, a few other renames and
cleanups have been made:
* `esusers` commands is now `users`
* org.elasticsearch.shield.authc.esusers -> org.elasticsearch.shield.authc.file
* Validation.ESUsers -> Validation.Users
* ESUsersTool -> UsersTool
* ESUsersToolTests -> UsersToolTests
* ESNativeUsersStore -> NativeUsersStore
* ESNativeRolesStore -> NativeRolesStore.
* org.elasticsearch.shield.authz.esnative collapsed to org.elasticsearch.shield.authz.store
* ESNativeTests -> NativeRealmIntegTests
Closeselastic/elasticsearch#1793
Original commit: elastic/x-pack-elasticsearch@d2a0c136f3
This commit modifies using system properties to configure an integration
test cluster and instead use settings in the generated Elasticsearch
config file.
Original commit: elastic/x-pack-elasticsearch@65211b93d0
The san string used by certificate generation for ssl tests currently
runs at gradle configuration time. This takes several seconds, and
significantly slows down gradle configuration on every invocation.
This change wraps the code into a lazy evaluator that will be invoked at
runtime, and cache the string once it is generated.
Original commit: elastic/x-pack-elasticsearch@812036f416
This removes the use of a specific address in smoke test ssl plugins and instead generates
the certificate with all of the IP addresses and DNS names of the system as subject
alternative names. This required duplication and modification of some code from core's
NetworkUtils.
Original commit: elastic/x-pack-elasticsearch@576824376f
This removes the use of group setting for `shield.` and introduces some individual settings
and some group settings that should not overlap and cause issues when iteration order
changes.
See elastic/elasticsearch#1520
Original commit: elastic/x-pack-elasticsearch@193e937193
Until we can fix the shield settings, we have bugs where we depend on the iteration
order of a map and discovery ec2 settings provoke this (most likely through a map
resize).
See elastic/elasticsearch#1520
Original commit: elastic/x-pack-elasticsearch@fbc32cf069
- Moved all settings in Marvel from `marvel.*` to `xpack.monitoring.*`
- Cleaned up marvel settings in general - they're all now under `MarvelSettings` class
- fixed some integration tests along the way (they were configured wrong and never actually tested anything)
- Updated the docs accordingly
- Added `migration-5_0.asciidoc` under the Marvel docs to explain how to migrate from Marvel 2.x to XPack 5.0.
- Replaced all `marvel` mentions in the logs to `monitoring`
- Removed the `xpack.monitoring.template.version` setting from the templates
- renamed the templates to `monitoring-es-data.json` and `monitoring-es.json`
- monitoring indices are now `.monitoring-es-<version>-data` and `.monitoring-es-<version>-<timestamp>`
Original commit: elastic/x-pack-elasticsearch@17f2abe17d
- Started to move configuration under the `xpack` name
- Cleaned up `ShieldPlugin`
- renamed `ShieldClient` to `SecurityClient`
- Introduced `XPackClient` that wraps security and watcher clients
Original commit: elastic/x-pack-elasticsearch@f05be0c180
This change migrates all of the xpack code to use the new ThreadContext when
dealing with headers and context data. For the most part this is a simple
cutover, but there are some things that required special casing. The internal
actions that executed by a user's requests need to forcefully drop the context
and set the system user. The workaround for this will be improved in a followup.
Additionally, the RequestContext still lives on due to the OptOutQueryCache,
which requires some core changes to fix this issue.
Original commit: elastic/x-pack-elasticsearch@87d2966d93
My previous change only fixed it when it resolves to something different than localhost.
But in some cases, it does not resolve at all, we must specify the IP.
Original commit: elastic/x-pack-elasticsearch@547eb4c42f
This commit fixes the integration tests and qa test after the reorganization to be
packaged as a single plugin.
Original commit: elastic/x-pack-elasticsearch@d6f488627f
jaymode
Jason Tedor
Chris Earle
Nik Everett
Ryan Ernst
Simon Willnauer
Tanguy Leroux
javanna
Alexander Reelsen
uboness
Robert Muir