This is a companion commit to elastic/elasticsearchelastic/elasticsearch#18514, fixing issues introduced by adding dedicated master nodes to the test infra
Original commit: elastic/x-pack-elasticsearch@8c0571f2de
There is a race condition between the smoke tests that get run because of the teardown conditions of
REST tests. By splitting them, we can avoid the unrealistic scenario/race condition.
Original commit: elastic/x-pack-elasticsearch@f95ae0e595
In order to move things from watcher to x-pack this created a notification module in x-pack.
This also means that the HTTPClient was moved up and settings have changed from
`xpack.watcher.http` to just `xpack.http`.
Further things done:
* Move http under o.e.xpack.common
* Moved secret service to o.e.xpack.common, initializing in XpackPlugin
* Moved text template to o.e.xpack.common.text
* Moved http client initialization into xpack plugin
* Renamed xpack.watcher.encrypt_sensitive_data setting, moved into Watch class
* Moved script service proxy to common
Original commit: elastic/x-pack-elasticsearch@41eb6e6946
This refactors the org.elasticsearch.watcher over to
org.elasticsearch.xpack.watcher
This also adds all watcher actions to the KnownActionsTests,
as watcher actions had not been taken care of until here.
Original commit: elastic/x-pack-elasticsearch@a046dc7c6a
This commit adds the necessary changes to make SSL work on the transport layer by default. A large
portion of the SSL configuration/settings was re-worked with this change. Some notable highlights
include support for PEM cert/keys, reloadable SSL configuration, separate HTTP ssl configuration, and
separate LDAP configuration.
The following is a list of specific items addressed:
* `SSLSettings` renamed to `SSLConfiguration`
* `KeyConfig` and `TrustConfig` abstractions created. These hide the details of how `KeyManager[]` and `TrustManager[]` are loaded. These are also responsible for settings validation (ie keystore password is not null)
* Configuration fallback is changed. Previously any setting would fallback to the "global" value (`xpack.security.ssl.*`). Now a keystore path, key path, ca paths, or truststore path must be specified otherwise the configuration for that key/trust will fallback to the global configuration. In other words if you want to change part of a keystore or truststore in a profile you need to supply all the information. This could be considered breaking if a user relied on the old fallback
* JDK trusted certificates (`cacerts`) are trusted by default (breaking change). This can be disabled via a setting.
* We now monitor the SSL files for changes and enable dynamic reloading of the configuration. This will make it easier for users when they are getting set up with certificates so they do not need to restart every time. This can be disabled via a setting
* LDAP realms can now have their own SSL configurations
* HTTP can now have its own SSL configuration
* SSL is enabled by default on the transport layer only. Hostname verification is enabled as well. On startup if no global SSL settings are present and SSL is configured to be used, we auto generate one based on the default CA that is shipped. This process includes a best effort attempt to generate the subject alternative names.
* `xpack.security.ssl.hostname_verification` is deprecated in favor of `xpack.security.ssl.hostname_verification.enabled`
* added Bouncy Castle info to NOTICE
* consolidated NOTICE and LICENSE files
Closeselastic/elasticsearch#14Closeselastic/elasticsearch#34Closeselastic/elasticsearch#1483Closeselastic/elasticsearch#1933
Addresses security portion of elastic/elasticsearch#673
Original commit: elastic/x-pack-elasticsearch@7c359db90b
This commit removes duplicated code in the authentication service by combining
the authentication logic for rest and transport requests. As part of this we no longer
cache the authentication token since we put the user in the context and serialize the
user.
Additionally we now pass the thread context to the AuthenticationFailureHandler to
restore access to the headers and context.
Original commit: elastic/x-pack-elasticsearch@79e2375a13
This mainly moves packages over to the x-pack directory and renames the settings
from `xpack.watcher.actions.` to `xpack.notification.`
Moved services include pagerduty, hipchat, slack and email.
Closeselastic/elasticsearch#1998
Original commit: elastic/x-pack-elasticsearch@40c16fe123
Our documentation states that we have default attributes for
message.format and message.color, which in fact we do not have
as an NPE was triggered in that case.
This commit falls back to unset defaults and allows for hipchat messages
to be sent without having to configure color/format in the action
or the account.
Closeselastic/elasticsearch#1666
Original commit: elastic/x-pack-elasticsearch@bfb7e35112
This removes the old usage of NetworkAddress.formatAddress in favor of the updated version, which is just
the method renamed to NetworkAddress.format (replacing the old version of that method).
There is no impact to x-plugins beyond making the build work because all places were currently using that
method variant already.
Original commit: elastic/x-pack-elasticsearch@05f0dcfa90
This commit adds support for the change in elasticsearch where the `/` rest
endpoint now delegates to an action and can be authorized.
Original commit: elastic/x-pack-elasticsearch@8ef38ce50f
This commit adds reserved or built-in user and role support to x-pack. The reserved roles
cannot be modified by users. The reserved users also cannot be modified with the exception
of changing the password for a user.
In order to change the password for a user, a new API has been added. This API only supports
changing passwords for native and reserved users.
To support allowing a user to change their own password, a default role has been added to grant
access. This default role only grants access to user operations that pertain to the user that is
being authorized. In other words, the default role grants `joe` the ability to change their own password
but does not allow them to change the password of a different user.
Additionally, the authenticate API was made a transport action and is granted by the default role.
Closeselastic/elasticsearch#1727Closeselastic/elasticsearch#1185Closeselastic/elasticsearch#1158
Original commit: elastic/x-pack-elasticsearch@1a6689d90f
This commit migrates all of the `shield.` settings to `xpack.security.` and makes changes to
use the new Settings infrastructure in core.
As a cleanup, this commit also renames Shield to Security since this class is only in master
and will not affect 2.x.
See elastic/elasticsearch#1441
Original commit: elastic/x-pack-elasticsearch@a5a9798b1b
This commit fixes an issue in synchronization in Exporters class. The export() method is synchronized and when used with LocalExport can provoke a deadlock. LocalExporter exports data locally using bulk requests that can trigger cluster state updates for mapping updates. If a exporters settings update sneaks in, the settings update waits for the export to terminate but the export waits for the settings to be updated... and boom.
This commit removes the synchronized and refactor Exporters/LocalExporter to use state and dedicated instance of LocalBulk for each export so that synchronizing methods is not necessary anymore.
It also lower down some random settings in MonitoringBulkTests because the previous settings almost always fill the bulk thread pool.
closeselastic/elasticsearch#1769
Original commit: elastic/x-pack-elasticsearch@f50c916f8b
This commit is the forward port of renaming the type for esusers to file. There is no
backwards compatibility maintained here. Additionally, a few other renames and
cleanups have been made:
* `esusers` commands is now `users`
* org.elasticsearch.shield.authc.esusers -> org.elasticsearch.shield.authc.file
* Validation.ESUsers -> Validation.Users
* ESUsersTool -> UsersTool
* ESUsersToolTests -> UsersToolTests
* ESNativeUsersStore -> NativeUsersStore
* ESNativeRolesStore -> NativeRolesStore.
* org.elasticsearch.shield.authz.esnative collapsed to org.elasticsearch.shield.authz.store
* ESNativeTests -> NativeRealmIntegTests
Closeselastic/elasticsearch#1793
Original commit: elastic/x-pack-elasticsearch@d2a0c136f3
* Add XPackExtension: an api class (like Plugin in core) for what a x-pack extension can do.
* Add XPackExtensionCli: a cli tool for adding, removing and listing extensions for x-pack.
* Add XPackExtensionService: loading of jars from pluginsdir/x-pack/extensions, into child classloader.
* Add bin/x-pack/extension script, similar to plugin cli, which installs an extension into pluginsdir/x-pack/extensions.
* Add XPack extension integration test.
Fixedelastic/elasticsearch#1515
Original commit: elastic/x-pack-elasticsearch@130ba03270
This commit adds support for the privilege naming defined in elastic/elasticsearch#1342 and removes the
support for the privileges that were deprecated in 2.3. This change also includes
updates to the documentation to account for the new roles format.
Original commit: elastic/x-pack-elasticsearch@98e9afd409
This commit remove the pre-existing file parsing code and replaces it with the updated
code in the RoleDescriptor class. This unifies the parsing for the files and API for roles.
Closeselastic/elasticsearch#1596
Original commit: elastic/x-pack-elasticsearch@9e0b58fcf1
As part of the search refactoring effort, we need to pass a Suggester
to the methods that parse X-Content to a SuggestBuilder in every
instance where we are parsing search/suggest requests.
Original commit: elastic/x-pack-elasticsearch@7f815c617a
This commit modifies using system properties to configure an integration
test cluster and instead use settings in the generated Elasticsearch
config file.
Original commit: elastic/x-pack-elasticsearch@65211b93d0
The san string used by certificate generation for ssl tests currently
runs at gradle configuration time. This takes several seconds, and
significantly slows down gradle configuration on every invocation.
This change wraps the code into a lazy evaluator that will be invoked at
runtime, and cache the string once it is generated.
Original commit: elastic/x-pack-elasticsearch@812036f416
This test assumes no modules are installed but the shield rest tests run with the modules
installed.
Original commit: elastic/x-pack-elasticsearch@2ba47fcd0f
This removes the use of a specific address in smoke test ssl plugins and instead generates
the certificate with all of the IP addresses and DNS names of the system as subject
alternative names. This required duplication and modification of some code from core's
NetworkUtils.
Original commit: elastic/x-pack-elasticsearch@576824376f
This removes the use of group setting for `shield.` and introduces some individual settings
and some group settings that should not overlap and cause issues when iteration order
changes.
See elastic/elasticsearch#1520
Original commit: elastic/x-pack-elasticsearch@193e937193
Until we can fix the shield settings, we have bugs where we depend on the iteration
order of a map and discovery ec2 settings provoke this (most likely through a map
resize).
See elastic/elasticsearch#1520
Original commit: elastic/x-pack-elasticsearch@fbc32cf069
- Moved all settings in Marvel from `marvel.*` to `xpack.monitoring.*`
- Cleaned up marvel settings in general - they're all now under `MarvelSettings` class
- fixed some integration tests along the way (they were configured wrong and never actually tested anything)
- Updated the docs accordingly
- Added `migration-5_0.asciidoc` under the Marvel docs to explain how to migrate from Marvel 2.x to XPack 5.0.
- Replaced all `marvel` mentions in the logs to `monitoring`
- Removed the `xpack.monitoring.template.version` setting from the templates
- renamed the templates to `monitoring-es-data.json` and `monitoring-es.json`
- monitoring indices are now `.monitoring-es-<version>-data` and `.monitoring-es-<version>-<timestamp>`
Original commit: elastic/x-pack-elasticsearch@17f2abe17d
- renaming `ShieldPlugin` to `Shield` (it's no longer a plugin)
- renaming `WatcherPlugin` to `Watcher` (it's no longer a plugin)
- renaming `MarvelPlugin` to `Marvel` (it's no longer a plugin)
- renaming `LicensePlugin` to `Licensing` (it's no longer a plugin)
- renamed setting:`watcher.enabled` -> `xpack.watcher.enabled`
- renamed setting:`marvel.enabled` -> `xpack.marvel.enabled`
Original commit: elastic/x-pack-elasticsearch@35a6540b11
- Consolidated the `bin` and `config` directories of watcher, shield and marvel under a single `config/xpack` and `bin/xpack` directories.
- updated docs accordingly
Original commit: elastic/x-pack-elasticsearch@c2aa6132fa
- Started to move configuration under the `xpack` name
- Cleaned up `ShieldPlugin`
- renamed `ShieldClient` to `SecurityClient`
- Introduced `XPackClient` that wraps security and watcher clients
Original commit: elastic/x-pack-elasticsearch@f05be0c180
This loads an index template for the watch history to make sure,
that field changes are taken into account.
Also, the dynamic mapping for the watch history template has been
changed from strict to false.
This means that new fields can be included in a document, but they
will not indexed and are not searchable.
In addition the index names have been changed from .watch_history-$date to
.watcher-history-$template-$date - using dashes to be more consistent.
Closeselastic/elasticsearch#1299
Original commit: elastic/x-pack-elasticsearch@794f982234
This commit fixes the bad apple tests that failed when running them. The
IndexAuditTrailEnabledTest was removed and the test was folded into the
IndexAuditIT. Some watcher tests that relied on mustache were moved
into the QA tests with the mustache plugin.
Additionally, fixing these tests uncovered a issue with the privileges needed
for writing data into an index. If the mappings need to be updated because
of a write, then the update mapping action gets executed. In 2.x this was
handled by the system user, but now is executed under the user's context,
which is the correct thing to do. The update mapping action is now added to
the read, index, crud, and write privileges for an index.
Original commit: elastic/x-pack-elasticsearch@30711f9625
In elastic/elasticsearch#1442 checkstyle checks were added, but also some files were freed from this.
If we have support for checkstyle, we should check this for all files and not allow
exceptions. This commit removes the file list to ignore any files and fixes all the
java files.
Original commit: elastic/x-pack-elasticsearch@99e6cbc5be
The lang-mustache module has been extended to meet Watcher's needs:
* The ability to refer the specific slots in arrays.
* An `content_type` option controls whether json string escaping is used. Otherwise there is no escaping.
Closeselastic/elasticsearch#1116
Other changes:
* I changed tests that were just using mustache just because it was around to not use mustache
* I moved tests to `test-xpack-with-mustache` module that were testing mustache with Watcher
* added smoke test for watcher and mustache
* moved some tests around
* instead of using DefaultTextTemplateEngine in watcher tests use MockTextTemplateEngine
* added a mock mustache script engine
* Cleanup some messy tests to not rely on mustache and move them back into xpack module
* moved array access test to smoke test watcher with mustache module
* test: simplified the condition search test to take the time component out of it, while still simulation a condition
* removed the mustache dependency in the messy-test-watcher-with-groovy module
Original commit: elastic/x-pack-elasticsearch@6a2a4e885f
This change migrates all of the xpack code to use the new ThreadContext when
dealing with headers and context data. For the most part this is a simple
cutover, but there are some things that required special casing. The internal
actions that executed by a user's requests need to forcefully drop the context
and set the system user. The workaround for this will be improved in a followup.
Additionally, the RequestContext still lives on due to the OptOutQueryCache,
which requires some core changes to fix this issue.
Original commit: elastic/x-pack-elasticsearch@87d2966d93
This commit is the x-plugins side of the refactoring of script settings.
Relates elastic/elasticsearchelastic/elasticsearch#16197
Original commit: elastic/x-pack-elasticsearch@4c429933b9
Shield expands wildcards to concrete names (aliases or indices) before each request gets executed in es core. It never resolves aliases to concrete indices though, as permissions may be set against aliases rather than indices. During this resolution, it also looks at the state of the indices and the current indices options (expand_wildcards) to expand only to indices with the relevant state. When it comes to aliases though, they may point to multiple indices each one having a different state, so it always expands ignoring expand_wildcards. At that point the request will contain the explicit name of the alias, no wildcards, thus the expand_wildcards option will have no effect in core. ignore_unavailable could be used instead when interacting with shield, which would affect how aliases are resolved to multiple indices. In this case we can only blacklist the test in shield, as it will return both the closed index and the open one.
Original commit: elastic/x-pack-elasticsearch@54c11dfc31
This feature is mainly done for the integration with the commercial reporting, but can be used
for anything else as well.
This adds a `attachments` to the email configuration, which can be used like this
```
"attachments" : {
"some_id" : {
"http" : {
"request" : {
"url" : "http://example.org/foo.pdf"
}
}
},
"other_id" : {
"data" : {
"format" : "json"
}
}
}
```
The main reason to pick this format is extensibility. If we would like to support another
attachment type, like an file reader, we could do so easily from an API point of view.
Closeselastic/elasticsearch#870
Original commit: elastic/x-pack-elasticsearch@66d14be965
My previous change only fixed it when it resolves to something different than localhost.
But in some cases, it does not resolve at all, we must specify the IP.
Original commit: elastic/x-pack-elasticsearch@547eb4c42f
Squashed commit of the following:
commit 54de841112778b01e817e465d2f59840970bb4f3
Author: Robert Muir <rmuir@apache.org>
Date: Thu Dec 10 08:47:12 2015 -0500
see exactly how far MessyTestPlugin can go (unit tests depend on both groovy and mustache!)
commit 743b5a4e0cc9a05e307339dfcb4569feed31f337
Author: Robert Muir <rmuir@apache.org>
Date: Thu Dec 10 08:35:03 2015 -0500
fix x-pack compile/tests to work with https://github.com/elastic/elasticsearch/pull/15328
Original commit: elastic/x-pack-elasticsearch@4307bb163b
This commit disables marvel when running the watcher with groovy tests. Marvel creates
indices and expect the cluster to be green, which won't happen with the marvel indices
existing and expecting a replica.
See elastic/elasticsearch#1087
Original commit: elastic/x-pack-elasticsearch@8d163a53b7
* move static initialization hack for UnboundID Debug to XPackPlugin
* cleanup bundlePlugin calls in build file
* properly disable watcher and marvel for shield core tests
Original commit: elastic/x-pack-elasticsearch@2b89cf2225
This commit fixes the integration tests and qa test after the reorganization to be
packaged as a single plugin.
Original commit: elastic/x-pack-elasticsearch@d6f488627f
Boaz Leskes
Robert Muir
Tanguy Leroux
Adrien Grand
jaymode
Chris Earle
Lee Hinman
Alexander Reelsen
Martijn van Groningen
uboness
Nik Everett
javanna
Jim Ferenczi
Ali Beyad
Jason Tedor
Ryan Ernst
markharwood
Colin Goodheart-Smithe
Simon Willnauer
Zachary Tong