The view index metadata privilege did not grant access to the shard level field mapping
action or the shard level validate query action. This caused the apis to restrict access to
the data when it should have been allowed.
Closeselastic/elasticsearch#1827
Original commit: elastic/x-pack-elasticsearch@7832699cb6
This commit synchronizes the start of the threads that are executing monitoring bulk requests concurrently
to ensure all threads are ready before starting. Without this some threads will execute requests while
other threads are still being constructed.
Original commit: elastic/x-pack-elasticsearch@e777fb5c28
This commit changes the handling in the merge method of the IndexAccessControl class to
properly handle merging IndexAccessControl objects with differing values for the granted
flag. Prior to this commit, in a scenario where the flag differed, one IndexAccessControl granted
no access to an index, and the other granted access with DLS/FLS resulted in full access
being granted to the index.
Closeselastic/elasticsearch#1821
Original commit: elastic/x-pack-elasticsearch@e403e43689
This commit removes the parsing of the username in the request body of a put user
request. Additionally, we use the name passed into the put role request builder rather
than the name from the parsed role descriptor.
Original commit: elastic/x-pack-elasticsearch@0a085d5844
This commit is the forward port of renaming the type for esusers to file. There is no
backwards compatibility maintained here. Additionally, a few other renames and
cleanups have been made:
* `esusers` commands is now `users`
* org.elasticsearch.shield.authc.esusers -> org.elasticsearch.shield.authc.file
* Validation.ESUsers -> Validation.Users
* ESUsersTool -> UsersTool
* ESUsersToolTests -> UsersToolTests
* ESNativeUsersStore -> NativeUsersStore
* ESNativeRolesStore -> NativeRolesStore.
* org.elasticsearch.shield.authz.esnative collapsed to org.elasticsearch.shield.authz.store
* ESNativeTests -> NativeRealmIntegTests
Closeselastic/elasticsearch#1793
Original commit: elastic/x-pack-elasticsearch@d2a0c136f3
Some tests [reasonably] fail due to unexpected indices appearing in the cluster due to Watcher.
- Also had to reset shieldEnabled as a static field, which makes no sense, but tests were failing unpredictably without it
- Now they're passing unpredictably with it... will investigate
Original commit: elastic/x-pack-elasticsearch@9b6ce681d8
This is required to make sure that the integration for monitoring the Watcher Threadpool is actually working.
- Also added the full property name when the assertContains check fails
- Made shieldEnabled an instance level field rather than a static one
- Added watcherEnabled field in the same fashion (including enableWatcher method that by default randomly enables it)
- Added method to locally filter the expected field names based on watcher being enabled for the failing test
Original commit: elastic/x-pack-elasticsearch@2c56e2f26f
Node roles are now serialized as well, they are not part of the node attributes anymore. DiscoveryNodeService takes care of dividing settings into attributes and roles. DiscoveryNode always requires to pass in attributes and roles separately.
Original commit: elastic/x-pack-elasticsearch@32a4eb0fb4
This is needed in order to make `bin/elasticsearch-plugin install xpack`
work, as it expects the plugin in a certain path.
Original commit: elastic/x-pack-elasticsearch@252c55e5a8
This commit is the forward port of fixes made in 2.3 for the roles and users
pollers. The pollers now refresh since not all operations are guaranteed to
refresh.
The clear roles tests are also made more evil since the poller runs at different
intervals on each node and can sometimes run almost continuously. The
modification requests now randomize if they refresh or not as well.
Original commit: elastic/x-pack-elasticsearch@f61159c40a
Users running the `CleanerService` should not be able to disable it (via a `-1` as the time setting) because they'll just shoot themselves in the foot. This PR changes the behavior to allow extensive amounts via the setting (e.g., they could set it to 2 years). By doing this via the `Setting`, we can avoid a lot of boilerplate code for verification as well. If we decide to allow it to be disabled, then the setting should be explicit. I've found that users tend to not understand setting times to `-1`.
With the internal `IndicesCleaner` runnable, I have also moved the rescheduling code to `onAfter` so that it always happens, even if the license makes it temporarily invalid.
I also think that we should allow the user to dynamically set the setting regardless of it being allowed -- and warn on it. This way they can set it when it's expired or during the trial, but it will take effect when they apply the paid license. I think that this will provide a better user experience so that they do not have to remember to re-set it later.
This also removes the `LocalExporter`-specific setting that allowed it to override the global retention. If we ever add another listener, then we should add exporter-specific settings to support this kind of functionality.
Adds some tests for the settings as well as for the service, while also removing now unneeded ones.
Original commit: elastic/x-pack-elasticsearch@3abd41807e
Currently it first creates a scorer, then checks if the role bits are sparse,
and falls back to the bulk scorer if they are dense. The issue is that creating
scorers and bulk scorers is very expensive on some queries such as ranges,
prefix and terms queries. So it should rather check whether bits are sparse
first in order to decide whether to use the scorer or bulk scorer.
Original commit: elastic/x-pack-elasticsearch@067d630099
Similar to WatcherClientProxy, the elasticsearch client used in exporters must be proxied to avoid circular dependencies at Guice's injection time. This commit add a MonitoringClientProxy as well as a MonitoringClient to be used later in monitoring's transport actions.
(cherry picked from commit b70c095)
Original commit: elastic/x-pack-elasticsearch@17327cffe5
* Add XPackExtension: an api class (like Plugin in core) for what a x-pack extension can do.
* Add XPackExtensionCli: a cli tool for adding, removing and listing extensions for x-pack.
* Add XPackExtensionService: loading of jars from pluginsdir/x-pack/extensions, into child classloader.
* Add bin/x-pack/extension script, similar to plugin cli, which installs an extension into pluginsdir/x-pack/extensions.
* Add XPack extension integration test.
Fixedelastic/elasticsearch#1515
Original commit: elastic/x-pack-elasticsearch@130ba03270
This change always serializes the version with the user so that we have this information
for times when we need to make changes and deal with serialization changes. We do this
in the authentication service because the user object is also serialized as part of the get
users response and the StreamInput there will have the appropriate version set on it already
and we do not need to add it in that case.
Closeselastic/elasticsearch#1747
Original commit: elastic/x-pack-elasticsearch@a7ceece09c
Adds debug logging to try to get more information about random failures in these
tests. Also cleans up some potential issues with the code that handled the stopping
of random ldap servers to test failure cases.
See elastic/elasticsearch#1542
Original commit: elastic/x-pack-elasticsearch@573b4161dd
These privileges no longer need to be defined as a custom privilege since the
code is now consolidated into a single plugin. This also changes the manage
cluster privilege to be an alias to the all privilege.
Original commit: elastic/x-pack-elasticsearch@a7f444c898
This commit renames the security actions to not use shield in their action names. This
also includes updating the privileges as well.
Original commit: elastic/x-pack-elasticsearch@10460dffdb
This commit adds support for the privilege naming defined in elastic/elasticsearch#1342 and removes the
support for the privileges that were deprecated in 2.3. This change also includes
updates to the documentation to account for the new roles format.
Original commit: elastic/x-pack-elasticsearch@98e9afd409
This commit remove the pre-existing file parsing code and replaces it with the updated
code in the RoleDescriptor class. This unifies the parsing for the files and API for roles.
Closeselastic/elasticsearch#1596
Original commit: elastic/x-pack-elasticsearch@9e0b58fcf1
A lot of messages were being logged at the info level in the native user and roles
stores. This changes the logging to be more selective in the cases where the index
does not exist or the error is really an error and the user should be notified.
Closeselastic/elasticsearch#1339
Original commit: elastic/x-pack-elasticsearch@0bc0d9bf7a
As part of the search refactoring effort, we need to pass a Suggester
to the methods that parse X-Content to a SuggestBuilder in every
instance where we are parsing search/suggest requests.
Original commit: elastic/x-pack-elasticsearch@7f815c617a
The reported time waiting for watches can be slightly different from the actual timeout (e.g. 2.1 seconds instead of 2 seconds) so checking the time waited in the message makes the test sometimes fail
Original commit: elastic/x-pack-elasticsearch@c2cd9da486
Core reworked how it registered tasks status's with NamedWriteableRegistry
so it was more pluggable. It changed a few signatures and x-plugins needs
these small changes to keep compiling.
Original commit: elastic/x-pack-elasticsearch@3dcf1df152
This commit modifies using system properties to configure an integration
test cluster and instead use settings in the generated Elasticsearch
config file.
Original commit: elastic/x-pack-elasticsearch@65211b93d0
Adds a check to the settings at startup to ensure that the security and audit indices are
allowed to be auto created if a user has disabled auto create explicitly.
Additionally fixes a small issue with the error message for watcher passing the incorrect
value.
Closeselastic/elasticsearch#1453
Original commit: elastic/x-pack-elasticsearch@2b0698ff19
If a user configures only custom realms and they are not licensed to use the custom realms then
we need to return our default realms. The default realms should be the esusers and esnative realms.
We were only returning the esusers realm previously.
Closeselastic/elasticsearch#1491
Original commit: elastic/x-pack-elasticsearch@3dc2b5d3a8
Today Watcher tries to load stuff from the bin's parent directory which
is not readable since the shared data directory has been moved out of
the nodes parent in elasticsearchelastic/elasticsearch#17072 which causes security exception
now. The test copies trust stores into the config dir and that's where
we should read it from by default or even better explicitly configure the path?!
Original commit: elastic/x-pack-elasticsearch@1d32a595cf
The san string used by certificate generation for ssl tests currently
runs at gradle configuration time. This takes several seconds, and
significantly slows down gradle configuration on every invocation.
This change wraps the code into a lazy evaluator that will be invoked at
runtime, and cache the string once it is generated.
Original commit: elastic/x-pack-elasticsearch@812036f416
Don't override indices when concreteIndex is set on PutMappingRequest
PutMappingRequest has a special case since it can come with one and only
one concrete index. In such a case we can't replace the indices list
with all authorized indices but should rather only check if the index
is authorized and otherwise fail the request.
Original commit: elastic/x-pack-elasticsearch@8949b16f16
PutMappingRequest has a special case since it can come with one and only
one concrete index. In such a case we can't replace the indices list
with all authorized indices but should rather only check if the index
is authorized and otherwise fail the request.
Original commit: elastic/x-pack-elasticsearch@4ee20029e1
This commit changes the location where the ID and Version of the resolvers are defined. It was in each constructor, now it is in the ResolversRegistry class. It also rename MonitoringIds to MonitoredSystem.
Original commit: elastic/x-pack-elasticsearch@81d7711c40
This commit adds a synchronized "export()" method to the Exporters so that the Exporters class can be used as an export service for exporting monitoring docs.
Original commit: elastic/x-pack-elasticsearch@22bda986c5
Also make logging message String constant to allow static checks
Relates to elastic/elasticsearchelastic/elasticsearch#16707
Original commit: elastic/x-pack-elasticsearch@b5bd423de4
This commit moves the InitializingModule and InitializingService classes in the common XPack package so that it can be used by any plugin. It also renames the module and service from Initializing* to LazyInitializing* and add a ClientProxy class.
Original commit: elastic/x-pack-elasticsearch@fbdf9d1614
Deep down buried there is an option to not only allow global
script execution, but also allow a single scripting language for
watcher only. It is time to document it as well.
Renamed this option to `script.engine.groovy.inline.xpack_watch`
to align with xpack renaming.
Closeselastic/elasticsearch#1422
Original commit: elastic/x-pack-elasticsearch@845eb5a0c0
Fix to ensure that the email attachment has a correctly set filename, which is
also now explained in the documentation.
In addition there is a check now for email attachments, that a filename can only
be specified once, otherwise an exception is thrown.
Closeselastic/elasticsearch#1503
Original commit: elastic/x-pack-elasticsearch@2a399058b3
The shield settings need to be copied down to the tribe nodes so that they are
aware of the shield configuration. Otherwise there will be issues such as SSL
not carrying over or authentication realms not being available.
Closeselastic/elasticsearch#702
Original commit: elastic/x-pack-elasticsearch@7bd7674f3e
This test assumes no modules are installed but the shield rest tests run with the modules
installed.
Original commit: elastic/x-pack-elasticsearch@2ba47fcd0f
This commit adds the logic to protect the user and roles index that we store locally
by restricting access to the internal XPack user. We need to do this in two places;
the first is when resolving wildcards and the other is when authorizing requests
made against specific indices.
Original commit: elastic/x-pack-elasticsearch@8ee0ce02db
We would previosly check if a node was a client node, we can now check it by just verifying that it is not a transport client through client_type setting.
Original commit: elastic/x-pack-elasticsearch@bddd44866e
This commit removes various constructors in monitoring documents and add a single constructeur that accepts a monitoring id and version. It also renames *Renderer classes to *Resolver and centralizes the logic of resolving the index name, type name and id in 1 place. It changes Exporter so that they use these resolvers to know in which index a given document must be indexed.
Original commit: elastic/x-pack-elasticsearch@c2349a95a6
When thinking about applications and the need to update a user, we should not need to
update the password of the user when making changes to things like roles, email, full
name, or metadata. This commit changes how we handle operations where the password
field is missing.
When the password field is missing, we try to execute an update. If the user exists, all
values for the user are updated except for the password field. If the user does not exist
and the password field is missing then a ValidationException is returned.
When the password field is present, we always issue an index request.
Closeselastic/elasticsearch#1492
Original commit: elastic/x-pack-elasticsearch@3d8a5f2db6
This commit introduces the default refresh on user and role update and delete
operations. The behavior can be controlled via the `refresh` parameter on the
REST API and the refresh option in the Java API.
Closeselastic/elasticsearch#1494
Original commit: elastic/x-pack-elasticsearch@aff4d13886
Only current timestamped indices, like .marvel-es-1-* indices should be deleted. Other indices like the ones created by pre v2.3.0 plugin versions should be kept (like .marvel-es-YYYY.MM.dd)
Original commit: elastic/x-pack-elasticsearch@b2aff31875
This commit bumps the Elasticsearch version to 5.0.0-SNAPSHOT in line
with the alignment of versions across the stack.
Relates elastic/elasticsearchelastic/elasticsearch#16862
Original commit: elastic/x-pack-elasticsearch@155641c5e4
This commit changes the behavior of combining multiple document level security queries
from an AND operation to an OR operation.
Additionally, the behavior is also changed when evaluating the combination of roles that
have document level security and roles that do not have document level security. Previously
when the permissions for these roles were combined, the queries from the roles with document
level security were still being applied, even though the user had access to all the documents.
This change now grants the user access to all documents in this scenario and the same applies
for field level security.
Closeselastic/elasticsearch#1074
Original commit: elastic/x-pack-elasticsearch@291107ec27
- Renamed `AddRoleAction/Request/Response` to `PutRoleAction/Request/Response`
- also renamed the user/roles rest actions
- Changed the returned format for `RestGetRoleAction`. Previously this endpoint returned an array of role descriptor. Now it returns an object where the role names serve as the keys for the role objects. This is aligned with other APIs in ES (e.g. index templates).
- When `RestGetRoleAction` cannot find all the requested roles, it'll return an empty object and a 404 response status
- Also cleaned up `RoleDescriptor`
Original commit: elastic/x-pack-elasticsearch@742f6e0020
DiscoveryService was a bridge into the discovery universe. This is unneeded and we can just access discovery directly or do things in a different way.
This is a complement to elastic/elasticsearchelastic/elasticsearch#16821
Closeselastic/elasticsearch#1571
Original commit: elastic/x-pack-elasticsearch@496f0c4081
- Now it's more aligned with other APIs in ES (e.g. index template API)
- the "get user" API now returns an object as a response. The users are keyed by their username. If none of the requested users is found, an empty object will be returned with a 404 response status.
- the body of "put user" request doesn't require "username" anymore (as it's defined as part of the URL)
Original commit: elastic/x-pack-elasticsearch@f7c12648b1
In case that a single email attachment cannot be downloaded, this ensures
that the whole action fails with a correct Action.Failure.
This also fixes an NPE that would occur otherwise.
Original commit: elastic/x-pack-elasticsearch@7bb042a719
This is a small fix to use specified id when sending data attachments.
The current solution always used "data".
Also a minor refactoring was made to include get the different parser impls
from the EmailAttachmentsParser instead of specifying them twice in the
EmailAction.
Closeselastic/elasticsearch#1503
Original commit: elastic/x-pack-elasticsearch@9354e83c8b
The roles parsing does not currently handle null tokens since the YAML parser
was not emitting them. With the upgrade to Jackson 2.7.1, the parser is now
emitting the null token value.
Original commit: elastic/x-pack-elasticsearch@abcad633ad
The awaiting latch was not waiting as long as the sleep in the code
causing the latch to fail and the test to fail.
This code aligns the time to wait for the latch and the sleep code
in the mock http server.
Original commit: elastic/x-pack-elasticsearch@8a2cc61204
Going forward (from 5.0 on) we'll remove all occurrences of the "shield" name/word from the code base. For this reason we want to already start using `.security` index in 2.3 such that we won't need to migrate it to a `.security` index later on.
Original commit: elastic/x-pack-elasticsearch@74a1cbfcf2
The request timeout and the real time the webserver slept was 5000ms.
In case of loaded systems, there might be cases, where the request was
still received in time.
This commit increases the server side sleep time to 10 seconds, to ensure
that the client aborts the request early
Original commit: elastic/x-pack-elasticsearch@718c05519f
The current HTTP timeout tests had two problems.
* Binding to port 9200-9300
* The first request to hit was having a delay, the other ones had not,
so if any other component hit the test inbetween (likely in a CI env),
the HTTP request from the test itself will not be delayed.
Both cases are fixed in this commit.
Original commit: elastic/x-pack-elasticsearch@d696e020cc
This removes the use of a specific address in smoke test ssl plugins and instead generates
the certificate with all of the IP addresses and DNS names of the system as subject
alternative names. This required duplication and modification of some code from core's
NetworkUtils.
Original commit: elastic/x-pack-elasticsearch@576824376f
When an HTTP input returns an error body, right now we check if the
error code is below 400 and only then we include the body.
However using another method from URLConnection, the body can be
access always.
Closeselastic/elasticsearch#1550
Original commit: elastic/x-pack-elasticsearch@1743fd0a77
- roles are now reliably parsed
- in `Put Role` API, added a double check to verify that the role name in the URL matches the role name if the body. Also, if the body doesn't have a role name, the role name in the URL will be used.
Original commit: elastic/x-pack-elasticsearch@5054ce8567
- Renamed `AddRoleAction/Request/Response` to `PutRoleAction/Request/Response`
- also renamed the user/roles rest actions
Original commit: elastic/x-pack-elasticsearch@ae0ccd61e5
- `full_name` and `email` are optional user fields
- `metadata` is an optional arbitrary meta data that can be associated with the user
- cleaned up the user actions - consistent naming (e.g. `PutUserAction` vs. `AddUserAction`)
- moved source parsing from the `PutUserRequest` to the `PutUserRequestBuilder`
- renamed`WatcherXContentUtils` to `XContentUtils` and moved it to sit under `o.e.xpack.commons.xcontent`
Closeselastic/elasticsearch#412
Original commit: elastic/x-pack-elasticsearch@5460e3caf7
We shouldn't have marvel enabled for these tests because we get false test failures
due to marvel indices existing and failing to lock the shard.
Original commit: elastic/x-pack-elasticsearch@11123bb660
This removes the use of group setting for `shield.` and introduces some individual settings
and some group settings that should not overlap and cause issues when iteration order
changes.
See elastic/elasticsearch#1520
Original commit: elastic/x-pack-elasticsearch@193e937193
Until we can fix the shield settings, we have bugs where we depend on the iteration
order of a map and discovery ec2 settings provoke this (most likely through a map
resize).
See elastic/elasticsearch#1520
Original commit: elastic/x-pack-elasticsearch@fbc32cf069
- Moved all settings in Marvel from `marvel.*` to `xpack.monitoring.*`
- Cleaned up marvel settings in general - they're all now under `MarvelSettings` class
- fixed some integration tests along the way (they were configured wrong and never actually tested anything)
- Updated the docs accordingly
- Added `migration-5_0.asciidoc` under the Marvel docs to explain how to migrate from Marvel 2.x to XPack 5.0.
- Replaced all `marvel` mentions in the logs to `monitoring`
- Removed the `xpack.monitoring.template.version` setting from the templates
- renamed the templates to `monitoring-es-data.json` and `monitoring-es.json`
- monitoring indices are now `.monitoring-es-<version>-data` and `.monitoring-es-<version>-<timestamp>`
Original commit: elastic/x-pack-elasticsearch@17f2abe17d
Custom realms may enable the use of other authorization schemes than just basic authentication
and these schemes should work in addition to our built in realms. However, our built in realms use
the UsernamePasswordToken class to parse the Authorization header, which had a check to ensure
the token was for basic authentication and if not, an exception was thrown. The throwing of the
exception stops the authentication process and prevents custom realms from evaluating the header
if they come later in the ordering of realms.
This change removes the throwing of the exception unless the header starts with 'Basic ' and is invalid.
Original commit: elastic/x-pack-elasticsearch@fd438ded95
- renaming `ShieldPlugin` to `Shield` (it's no longer a plugin)
- renaming `WatcherPlugin` to `Watcher` (it's no longer a plugin)
- renaming `MarvelPlugin` to `Marvel` (it's no longer a plugin)
- renaming `LicensePlugin` to `Licensing` (it's no longer a plugin)
- renamed setting:`watcher.enabled` -> `xpack.watcher.enabled`
- renamed setting:`marvel.enabled` -> `xpack.marvel.enabled`
Original commit: elastic/x-pack-elasticsearch@35a6540b11
This commit removes the message digest providers in x-plugins by using
the MessageDigests abstraction in core. In particular, this permits the
removal of the use of MessageDigest#clone in x-plugins.
Closeselastic/elasticsearch#1489
Original commit: elastic/x-pack-elasticsearch@6868e6e8ed
- Consolidated the `bin` and `config` directories of watcher, shield and marvel under a single `config/xpack` and `bin/xpack` directories.
- updated docs accordingly
Original commit: elastic/x-pack-elasticsearch@c2aa6132fa
- Started to move configuration under the `xpack` name
- Cleaned up `ShieldPlugin`
- renamed `ShieldClient` to `SecurityClient`
- Introduced `XPackClient` that wraps security and watcher clients
Original commit: elastic/x-pack-elasticsearch@f05be0c180
This loads an index template for the watch history to make sure,
that field changes are taken into account.
Also, the dynamic mapping for the watch history template has been
changed from strict to false.
This means that new fields can be included in a document, but they
will not indexed and are not searchable.
In addition the index names have been changed from .watch_history-$date to
.watcher-history-$template-$date - using dashes to be more consistent.
Closeselastic/elasticsearch#1299
Original commit: elastic/x-pack-elasticsearch@794f982234
This commit fixes the bad apple tests that failed when running them. The
IndexAuditTrailEnabledTest was removed and the test was folded into the
IndexAuditIT. Some watcher tests that relied on mustache were moved
into the QA tests with the mustache plugin.
Additionally, fixing these tests uncovered a issue with the privileges needed
for writing data into an index. If the mappings need to be updated because
of a write, then the update mapping action gets executed. In 2.x this was
handled by the system user, but now is executed under the user's context,
which is the correct thing to do. The update mapping action is now added to
the read, index, crud, and write privileges for an index.
Original commit: elastic/x-pack-elasticsearch@30711f9625
In elastic/elasticsearch#1442 checkstyle checks were added, but also some files were freed from this.
If we have support for checkstyle, we should check this for all files and not allow
exceptions. This commit removes the file list to ignore any files and fixes all the
java files.
Original commit: elastic/x-pack-elasticsearch@99e6cbc5be
According to RFC 2616 HTTP headers are case insensitive.
But `HttpResponse#contentType()` only looks up for Content-Type.
This stores all header responses lower cased in the HTTP response.
Closeselastic/elasticsearch#1357
Original commit: elastic/x-pack-elasticsearch@c009be8365
This commit adds a new `source_node` field to all marvel documents that holds various information (node's name/ip/host/id/transport address) about the node that emitted the document.
(cherry picked from commit elastic/x-pack@29a411a931)
Original commit: elastic/x-pack-elasticsearch@66e057d334
If elasticsearch merges https://github.com/elastic/elasticsearch/pull/16413
then the build will fail catastrophically without this. The goal here is to
opt these files out of the line length checks while they don't pass and we
can get them passing as time permits and opt them back in. In the mean time
all files that pass the line length check will have the check enforced.
This also gives you a spot to add x-plugins opt outs for core's checkstyle
rules in case you have generated files or something like that.
Original commit: elastic/x-pack-elasticsearch@63a1ad2f79
This change registers all filtered settings up-front and removes all
the unnecessary wrappers around SettingsFilter. This is a pretty big
change and needs some review but after all things are generally simplified and
settings are always filtered even if shield is not enabled which is the right thing
todo.
Relates to elastic/elasticsearchelastic/elasticsearch#16425
Original commit: elastic/x-pack-elasticsearch@c7df85492b
The lang-mustache module has been extended to meet Watcher's needs:
* The ability to refer the specific slots in arrays.
* An `content_type` option controls whether json string escaping is used. Otherwise there is no escaping.
Closeselastic/elasticsearch#1116
Other changes:
* I changed tests that were just using mustache just because it was around to not use mustache
* I moved tests to `test-xpack-with-mustache` module that were testing mustache with Watcher
* added smoke test for watcher and mustache
* moved some tests around
* instead of using DefaultTextTemplateEngine in watcher tests use MockTextTemplateEngine
* added a mock mustache script engine
* Cleanup some messy tests to not rely on mustache and move them back into xpack module
* moved array access test to smoke test watcher with mustache module
* test: simplified the condition search test to take the time component out of it, while still simulation a condition
* removed the mustache dependency in the messy-test-watcher-with-groovy module
Original commit: elastic/x-pack-elasticsearch@6a2a4e885f
- Consolidated `InternalMarvelUser`, `InternalWatcherUser` and `InternalShieldUser` into a single `XPackUser` - this is the single internal user for xpack that has all the permissions internally required by xpack (for marvel, watcher and shield)
- Renamed `InternalSystemUser` to `SystemUser`
- Removed the notion of "reserved roles". Now that we have a single internal user we know its role. The authz service now checks to see if the current user is the internal xpack user, and if so, it just uses its role (and not trying to resolve it from the role store). With this model, it's no longer possible for outside users to use the internal role (it's fully internal)
- Consolidated the notion of an `InternalClient` (in Marvel it was knows as the `SecuredClient`). This is an ES client that xpack is using to manage itself. If shield is enabled, it will execute all request on behalf of the internal xpack user.
- Removed the verification of the license plugin on plugin installation - no need to do it anymore as the license plugin is part of the distribution.
Original commit: elastic/x-pack-elasticsearch@c851410f93
This commit cleans up the hack we had forcefully switching the request to execute under the system
user when a internal action gets triggered from a system request. The authorization service now tracks
the originating request in the context to allow us to validate if the request should be run as the system
user.
The system user should be used only when a user action causes an internal action, which needs to
be run by the system user.
Closeselastic/elasticsearch#1403
Original commit: elastic/x-pack-elasticsearch@4972df459f
- removed `/_shield/roles` and `/_shield/users` endpoints (only keeping the singular forms)
- fixed `ClearRealmsCacheTests` to use the correct endpoint for clearing the realms cache
- used action name constants where possible in `InternalShieldUser`
Original commit: elastic/x-pack-elasticsearch@d1481de389
- Moved all role action classes to live under `o.e.s.action.role`
- Moved all realm related action classes (for now just the clear cache) to live under `o.e.s.action.realm`
- Moved all user action classes to live under `o.e.s.action.user`
- Moved all the rest actions to live under `o.e.s.rest.action`
- Changed the `clear role cache` endpoint to `/_shield/role/{id}/_clear_cache` (aligned with all other role endpoints)
- Changed `InternalShieldUserHolder` to the `InternalShieldUser` singleton user... to be aligned with `InternalMarvelUser` and `InternalWatcherUser`.
- Removed the dedicated audit log user. The new `InternalShieldUser` is now the user that manages and writes to the audit log indices
- Extracted the `User.System` class to a top level `InternalSystemUser` class (to be aligned with the other internal user classes)
- Removed the `SystemRole` class (the `InternalSystemUser` class now holds all the needed info/logic)
Original commit: elastic/x-pack-elasticsearch@cf82b257d1
When using a path like `"/<logstash-{now%2Fd}>/_search"` in the
http webhook. The already escaped slash (%2F) got escaped twice
and thus did not work any more.
The escaping happened when the code created an URI and was done
as part of that constructor. This is now switched to an URL (which
is used at the end anyway) which does not do the escaping, even though
this was required for the query string, which is now done when constructing.
Closeselastic/elasticsearch#1364
Original commit: elastic/x-pack-elasticsearch@861b6d2378
Two regressions have been introduced in elastic/x-pack@156d9e4d5b: marvel index templates should not be deleted between tests and checking for marvel indices existence should not fail with IndexNotFoundException when the indices are not yet created and Shield enabled.
closeselastic/elasticsearch#1396elastic/elasticsearch#1394elastic/elasticsearch#1382
In MultiNodesStatsTests.java, multiple nodes are started in async: the first node may collect marvel data multiple times when the last one just started. So we should not check for exact 1 doc per node but at least 1 doc per node.
closeselastic/elasticsearch#1370
In HttpExporterTemplateTests.java, we must compare a long count with a long value.
Original commit: elastic/x-pack-elasticsearch@732fef995a
Similar to the lifecycle services, stopping the shield lifecycle should
also ensure that the poller threads are stopped, which is tricky, in case
they run through huge user/role lists.
Original commit: elastic/x-pack-elasticsearch@7a48f19853
Restoring empty contexts causes issues with searches, but failure to restore the
original context when executing index requests that auto-create results in a
the index operation being tried by the system user.
See elastic/elasticsearch#1380
Original commit: elastic/x-pack-elasticsearch@522f857de7
Load average is not available anymore on Windows, the tests should not check the presence of the field. Also, "node_stats.json" file is hard to maintain and quite useless so this commit removes it.
Original commit: elastic/x-pack-elasticsearch@74d2e0dce6
This removes a check in the ClearRolesCacheTests that is prone to failure due to the
possibility of the cache poller running while we modify documents and updating cached
values prior to the test issuing the get roles call.
See elastic/elasticsearch#1354
Original commit: elastic/x-pack-elasticsearch@ba0b803466
This change migrates all of the xpack code to use the new ThreadContext when
dealing with headers and context data. For the most part this is a simple
cutover, but there are some things that required special casing. The internal
actions that executed by a user's requests need to forcefully drop the context
and set the system user. The workaround for this will be improved in a followup.
Additionally, the RequestContext still lives on due to the OptOutQueryCache,
which requires some core changes to fix this issue.
Original commit: elastic/x-pack-elasticsearch@87d2966d93
This commit is the x-plugins side of the refactoring of script settings.
Relates elastic/elasticsearchelastic/elasticsearch#16197
Original commit: elastic/x-pack-elasticsearch@4c429933b9
As of elastic/elasticsearchelastic/elasticsearch#16054 all index level settings
must be registered and use the new settings infrastructure. This commit
prepares for the merge to provide a smooth transition.
Original commit: elastic/x-pack-elasticsearch@bc0a4fec07
This commit addresses the handling of load averages in Marvel due to
upstream changes in core Elasticsearch where the load average field was
changed from an array to an object.
Original commit: elastic/x-pack-elasticsearch@9ea57968bb
Shield expands wildcards to concrete names (aliases or indices) before each request gets executed in es core. It never resolves aliases to concrete indices though, as permissions may be set against aliases rather than indices. During this resolution, it also looks at the state of the indices and the current indices options (expand_wildcards) to expand only to indices with the relevant state. When it comes to aliases though, they may point to multiple indices each one having a different state, so it always expands ignoring expand_wildcards. At that point the request will contain the explicit name of the alias, no wildcards, thus the expand_wildcards option will have no effect in core. ignore_unavailable could be used instead when interacting with shield, which would affect how aliases are resolved to multiple indices. In this case we can only blacklist the test in shield, as it will return both the closed index and the open one.
Original commit: elastic/x-pack-elasticsearch@54c11dfc31
Due to lingering threads when shutting down when running this in CI
this disables watcher in this tests. It is really hard to reproduce
to get the correct order of start/stop. The reason here is, that watcher
is still starting and not finished, when stop is called on the plugin.
Original commit: elastic/x-pack-elasticsearch@2de85bcba6
In elastic/elasticsearchelastic/elasticsearch#15907, we changed the load average
structure to also include 5 and 15 minute load averages. This
commit adjusts the expected JSON structure for
NodeStatsRendererTests.
Original commit: elastic/x-pack-elasticsearch@59f6a1e9d9
When debugging role mapping it is useful to know the actual user DN, which is only logged
when something is actually mapped to the DN. Since this is logged at debug level, we should
always log it.
Original commit: elastic/x-pack-elasticsearch@b690c757d3
Previously we only exposed the use of a single URL for LDAP realms, while the code supported
multiple URLs. Internally we always used a failover server set, which would have provided failover
to another LDAP server if multiple existed. This change introduces a new setting `load_balance.type`
on the realm that indicates the type of load balancing. Valid options are:
* `failover` - the first server in the list will be used until it fails and then additional servers will be tried until
one succeeds. The first successful server will be used from now on. This is the default.
* `round_robin` - continuously iterates through the list of servers for each new connection. If a server is down,
the iteration will continue until a successful connection is made. The downfall here is that the list does not
get reordered on a down server, so there is overhead for always trying the servers in order.
* `dns_failover` - This server set takes a single URL that uses a DNS that will resolve to multiple IP addresses.
Connections will be consistently attempted to servers in the order they are retrieved from the name service; there
is no re-ordering and the first successful connection will be used.
* `dns_round_robin` - This server set takes a single URL that uses a DNS that will resolve to multiple IP addresses.
The addresses retrieved from the name service will connected to in the same order as `round_robin`.
Closeselastic/elasticsearch#31
Original commit: elastic/x-pack-elasticsearch@9ce9a1bf23
In order to correctly check for equality in an EmailAction, all the email
attachments have to implement equals/hashCode methods.
This has already been added to the 2.x branch in elastic/x-pack@ebde22507f
Original commit: elastic/x-pack-elasticsearch@bb980ea934
This feature is mainly done for the integration with the commercial reporting, but can be used
for anything else as well.
This adds a `attachments` to the email configuration, which can be used like this
```
"attachments" : {
"some_id" : {
"http" : {
"request" : {
"url" : "http://example.org/foo.pdf"
}
}
},
"other_id" : {
"data" : {
"format" : "json"
}
}
}
```
The main reason to pick this format is extensibility. If we would like to support another
attachment type, like an file reader, we could do so easily from an API point of view.
Closeselastic/elasticsearch#870
Original commit: elastic/x-pack-elasticsearch@66d14be965
This commit removes the current implementation in HttpExporter so that it does not automatically clean indices anymore.
Original commit: elastic/x-pack-elasticsearch@7d30338355
In order to have a shortcut for the execution of a watch and
specifying the record_execution and ignore_condition booleans,
so are now supported in the HTTP request parameters as well.
Closeselastic/elasticsearch#918
Original commit: elastic/x-pack-elasticsearch@bed5da40b7
* This action enables sending notifications to pager duty services.
* Utilizes pager duty's REST API
* Similar to the `email`, `hipchat` and `slack` actions, multiple `pagerduty` accounts can be configured, each with its own Service API key
* A `pagerduty` account is roughly mapped to a service in your pagerduty service.
* `pagerduty` actions are associated with an account, or if not, their events will be sent via the default account.
* An incident can be acknowledged, resolved or triggered
Closeselastic/elasticsearch#492
Original commit: elastic/x-pack-elasticsearch@72cc21d119
This commit merge the ClusterInfoCollector in the ClusterStatsCollector so that cluster stats are retrieved only once.
Original commit: elastic/x-pack-elasticsearch@fe70149210
This commit changes the templates so that they are now versionned using a number (starting from 1). This number is used in index templates names (ex: .marvel-es-data-1, .marvel-es-1) as well as in indices names (ex: .marvel-es-1-2015-12-30, .marvel-es-data-1).
If the template does not exist, it is created. Otherwise nothing (no update) is done.
Original commit: elastic/x-pack-elasticsearch@66c1a8bed0
This commit enable tribe nodes support for Marvel. It avoid ElasticsearchSecurityException when a tribe node is connected to a cluster that has been configured for both Shield and Marvel by loading the MarvelShieldIntegration support on tribe node even if marvel.enabled is set to false. It also allows tribe nodes to be monitored using Marvel with their own marvel settings.
closeselastic/elasticsearch#1088
Original commit: elastic/x-pack-elasticsearch@e0401c1288
Instead of having to specify `scheme`, `host`, `port`, `path` and `params` each
one can now use the `url` field and specify something like `http://example.org/foo?bar=baz`
This works for HttpRequest and HttpRequestTemplate
Original commit: elastic/x-pack-elasticsearch@8c052cf419
Because of specifying the concrete implementations in WatcherPlugin.nodeServices(),
all of those services got created twice for each node (guice requires the interface here).
This resulted in NPEs as the first instantiation did not run doStart() and thus had empty
settings, that resulted in a NPE.
Closeselastic/elasticsearch#1179
Original commit: elastic/x-pack-elasticsearch@c6a1f5093f
* allow bulk requests with updates if fls/dls isn't configured for targetted index
* throw a hard error if the current call can't resolve an associated transport request
marvel: node stats collector should use the client instead of the internal apis, because otherwise the index searcher wrapper can't locate the transport request that is associated with current call and would then throw an exception, which then prevents the marvel agent from collecting stats.
* if both field or document level security is enabled then all forbidden operations should fail
Original commit: elastic/x-pack-elasticsearch@b2c40d6559
The REST authenticate API can be used to test credentials and retrieve details about the
user that has been authenticated. This currently includes retrieving the user's username
and roles.
Closeselastic/elasticsearch#1019
Original commit: elastic/x-pack-elasticsearch@117a226187
My previous change only fixed it when it resolves to something different than localhost.
But in some cases, it does not resolve at all, we must specify the IP.
Original commit: elastic/x-pack-elasticsearch@547eb4c42f
This commit removes usage of Collections#shuffle(List) as this is a
forbidden method in upstream Elasticsearch.
Relates elastic/elasticsearchelastic/elasticsearch#15287, relates elastic/elasticsearchelastic/elasticsearch#15299
Original commit: elastic/x-pack-elasticsearch@2943569ee1
This commit simplifies xcontent usage in x-pack in a way that is compatible with
elastic/elasticsearchelastic/elasticsearch#15358.
Original commit: elastic/x-pack-elasticsearch@0d6ce5f22f
Squashed commit of the following:
commit 54de841112778b01e817e465d2f59840970bb4f3
Author: Robert Muir <rmuir@apache.org>
Date: Thu Dec 10 08:47:12 2015 -0500
see exactly how far MessyTestPlugin can go (unit tests depend on both groovy and mustache!)
commit 743b5a4e0cc9a05e307339dfcb4569feed31f337
Author: Robert Muir <rmuir@apache.org>
Date: Thu Dec 10 08:35:03 2015 -0500
fix x-pack compile/tests to work with https://github.com/elastic/elasticsearch/pull/15328
Original commit: elastic/x-pack-elasticsearch@4307bb163b
This commit update the Local & Http exporters so that they have the same behavior. Exporters now try to update the mappings of the current marvel indices if they exist (only new document types will be added)
It also adds dedicated tests to verify that the Marvel template is correctly updated depending of the current template version.
Original commit: elastic/x-pack-elasticsearch@ac650f5aba
When the logging audit trail is configured to add the node hostname or ip address
as a prefix, the logging audit trail can invoke guice dependency injection issues since
the transport that is injected is a proxy. This change makes the logging audit trail a
lifecycle component and waits for the transport to be started before initializing the
prefix.
Closeselastic/elasticsearch#1104
Original commit: elastic/x-pack-elasticsearch@3b19283969
In order to be able to send emails, the thread context classs loader
must be set to the watcher plugin class loader (and reset afterwards),
otherwise some javax.activation classes needed to create mime attachments
cannot be found.
In addition the activation jar dependency has been removed.
Closeselastic/elasticsearch#1066Closeselastic/elasticsearch#708
Original commit: elastic/x-pack-elasticsearch@10042375ec
This commit disables marvel when running the watcher with groovy tests. Marvel creates
indices and expect the cluster to be green, which won't happen with the marvel indices
existing and expecting a replica.
See elastic/elasticsearch#1087
Original commit: elastic/x-pack-elasticsearch@8d163a53b7
Since we combined all of the plugins marvel will also be enabled and will cause
the cluster to never get to a green state since we only have a single node.
Original commit: elastic/x-pack-elasticsearch@60fbfa97f1
* move static initialization hack for UnboundID Debug to XPackPlugin
* cleanup bundlePlugin calls in build file
* properly disable watcher and marvel for shield core tests
Original commit: elastic/x-pack-elasticsearch@2b89cf2225
This commit fixes the integration tests and qa test after the reorganization to be
packaged as a single plugin.
Original commit: elastic/x-pack-elasticsearch@d6f488627f
- added the license-plugin dir as a src dir
- Made XPackPlugin customizable for tests. Now tests that need to plug in different behaviours (e.g. a custom license plugin) can do that by simply resetting the relevant delegee plugins (i.e. shield, marvel, license, watcher) in the ctor of a new XPackPlugin.
Original commit: elastic/x-pack-elasticsearch@ab83104955
Tanguy Leroux
javanna
Adrien Grand
Nik Everett
Simon Willnauer
Jim Ferenczi
Boaz Leskes
jaymode
Alexander Reelsen
Chris Earle
Lee Hinman
Areek Zillur
Martijn van Groningen
markharwood
Lukas Olson
Ali Beyad
Colin Goodheart-Smithe
Jason Tedor
Yannick Welsch
Ryan Ernst
Simon Willnauer
David Pilato
Robert Muir
uboness
Chris Earle
Igor Motov
Jay Modi
Daniel Mitterdorfer
Yannick Welsch
Zachary Tong
Jack Conradson
Britta Weber