This commit adds a new `source_node` field to all marvel documents that holds various information (node's name/ip/host/id/transport address) about the node that emitted the document.
(cherry picked from commit elastic/x-pack@29a411a931)
Original commit: elastic/x-pack-elasticsearch@66e057d334
This change registers all filtered settings up-front and removes all
the unnecessary wrappers around SettingsFilter. This is a pretty big
change and needs some review but after all things are generally simplified and
settings are always filtered even if shield is not enabled which is the right thing
todo.
Relates to elastic/elasticsearchelastic/elasticsearch#16425
Original commit: elastic/x-pack-elasticsearch@c7df85492b
The lang-mustache module has been extended to meet Watcher's needs:
* The ability to refer the specific slots in arrays.
* An `content_type` option controls whether json string escaping is used. Otherwise there is no escaping.
Closeselastic/elasticsearch#1116
Other changes:
* I changed tests that were just using mustache just because it was around to not use mustache
* I moved tests to `test-xpack-with-mustache` module that were testing mustache with Watcher
* added smoke test for watcher and mustache
* moved some tests around
* instead of using DefaultTextTemplateEngine in watcher tests use MockTextTemplateEngine
* added a mock mustache script engine
* Cleanup some messy tests to not rely on mustache and move them back into xpack module
* moved array access test to smoke test watcher with mustache module
* test: simplified the condition search test to take the time component out of it, while still simulation a condition
* removed the mustache dependency in the messy-test-watcher-with-groovy module
Original commit: elastic/x-pack-elasticsearch@6a2a4e885f
- Consolidated `InternalMarvelUser`, `InternalWatcherUser` and `InternalShieldUser` into a single `XPackUser` - this is the single internal user for xpack that has all the permissions internally required by xpack (for marvel, watcher and shield)
- Renamed `InternalSystemUser` to `SystemUser`
- Removed the notion of "reserved roles". Now that we have a single internal user we know its role. The authz service now checks to see if the current user is the internal xpack user, and if so, it just uses its role (and not trying to resolve it from the role store). With this model, it's no longer possible for outside users to use the internal role (it's fully internal)
- Consolidated the notion of an `InternalClient` (in Marvel it was knows as the `SecuredClient`). This is an ES client that xpack is using to manage itself. If shield is enabled, it will execute all request on behalf of the internal xpack user.
- Removed the verification of the license plugin on plugin installation - no need to do it anymore as the license plugin is part of the distribution.
Original commit: elastic/x-pack-elasticsearch@c851410f93
This commit cleans up the hack we had forcefully switching the request to execute under the system
user when a internal action gets triggered from a system request. The authorization service now tracks
the originating request in the context to allow us to validate if the request should be run as the system
user.
The system user should be used only when a user action causes an internal action, which needs to
be run by the system user.
Closeselastic/elasticsearch#1403
Original commit: elastic/x-pack-elasticsearch@4972df459f
- removed `/_shield/roles` and `/_shield/users` endpoints (only keeping the singular forms)
- fixed `ClearRealmsCacheTests` to use the correct endpoint for clearing the realms cache
- used action name constants where possible in `InternalShieldUser`
Original commit: elastic/x-pack-elasticsearch@d1481de389
- Moved all role action classes to live under `o.e.s.action.role`
- Moved all realm related action classes (for now just the clear cache) to live under `o.e.s.action.realm`
- Moved all user action classes to live under `o.e.s.action.user`
- Moved all the rest actions to live under `o.e.s.rest.action`
- Changed the `clear role cache` endpoint to `/_shield/role/{id}/_clear_cache` (aligned with all other role endpoints)
- Changed `InternalShieldUserHolder` to the `InternalShieldUser` singleton user... to be aligned with `InternalMarvelUser` and `InternalWatcherUser`.
- Removed the dedicated audit log user. The new `InternalShieldUser` is now the user that manages and writes to the audit log indices
- Extracted the `User.System` class to a top level `InternalSystemUser` class (to be aligned with the other internal user classes)
- Removed the `SystemRole` class (the `InternalSystemUser` class now holds all the needed info/logic)
Original commit: elastic/x-pack-elasticsearch@cf82b257d1
When using a path like `"/<logstash-{now%2Fd}>/_search"` in the
http webhook. The already escaped slash (%2F) got escaped twice
and thus did not work any more.
The escaping happened when the code created an URI and was done
as part of that constructor. This is now switched to an URL (which
is used at the end anyway) which does not do the escaping, even though
this was required for the query string, which is now done when constructing.
Closeselastic/elasticsearch#1364
Original commit: elastic/x-pack-elasticsearch@861b6d2378
Two regressions have been introduced in elastic/x-pack@156d9e4d5b: marvel index templates should not be deleted between tests and checking for marvel indices existence should not fail with IndexNotFoundException when the indices are not yet created and Shield enabled.
closeselastic/elasticsearch#1396elastic/elasticsearch#1394elastic/elasticsearch#1382
In MultiNodesStatsTests.java, multiple nodes are started in async: the first node may collect marvel data multiple times when the last one just started. So we should not check for exact 1 doc per node but at least 1 doc per node.
closeselastic/elasticsearch#1370
In HttpExporterTemplateTests.java, we must compare a long count with a long value.
Original commit: elastic/x-pack-elasticsearch@732fef995a
Similar to the lifecycle services, stopping the shield lifecycle should
also ensure that the poller threads are stopped, which is tricky, in case
they run through huge user/role lists.
Original commit: elastic/x-pack-elasticsearch@7a48f19853
Restoring empty contexts causes issues with searches, but failure to restore the
original context when executing index requests that auto-create results in a
the index operation being tried by the system user.
See elastic/elasticsearch#1380
Original commit: elastic/x-pack-elasticsearch@522f857de7
Load average is not available anymore on Windows, the tests should not check the presence of the field. Also, "node_stats.json" file is hard to maintain and quite useless so this commit removes it.
Original commit: elastic/x-pack-elasticsearch@74d2e0dce6
This removes a check in the ClearRolesCacheTests that is prone to failure due to the
possibility of the cache poller running while we modify documents and updating cached
values prior to the test issuing the get roles call.
See elastic/elasticsearch#1354
Original commit: elastic/x-pack-elasticsearch@ba0b803466
This change migrates all of the xpack code to use the new ThreadContext when
dealing with headers and context data. For the most part this is a simple
cutover, but there are some things that required special casing. The internal
actions that executed by a user's requests need to forcefully drop the context
and set the system user. The workaround for this will be improved in a followup.
Additionally, the RequestContext still lives on due to the OptOutQueryCache,
which requires some core changes to fix this issue.
Original commit: elastic/x-pack-elasticsearch@87d2966d93
This commit is the x-plugins side of the refactoring of script settings.
Relates elastic/elasticsearchelastic/elasticsearch#16197
Original commit: elastic/x-pack-elasticsearch@4c429933b9
As of elastic/elasticsearchelastic/elasticsearch#16054 all index level settings
must be registered and use the new settings infrastructure. This commit
prepares for the merge to provide a smooth transition.
Original commit: elastic/x-pack-elasticsearch@bc0a4fec07
This commit addresses the handling of load averages in Marvel due to
upstream changes in core Elasticsearch where the load average field was
changed from an array to an object.
Original commit: elastic/x-pack-elasticsearch@9ea57968bb
Due to lingering threads when shutting down when running this in CI
this disables watcher in this tests. It is really hard to reproduce
to get the correct order of start/stop. The reason here is, that watcher
is still starting and not finished, when stop is called on the plugin.
Original commit: elastic/x-pack-elasticsearch@2de85bcba6
In elastic/elasticsearchelastic/elasticsearch#15907, we changed the load average
structure to also include 5 and 15 minute load averages. This
commit adjusts the expected JSON structure for
NodeStatsRendererTests.
Original commit: elastic/x-pack-elasticsearch@59f6a1e9d9
When debugging role mapping it is useful to know the actual user DN, which is only logged
when something is actually mapped to the DN. Since this is logged at debug level, we should
always log it.
Original commit: elastic/x-pack-elasticsearch@b690c757d3
Previously we only exposed the use of a single URL for LDAP realms, while the code supported
multiple URLs. Internally we always used a failover server set, which would have provided failover
to another LDAP server if multiple existed. This change introduces a new setting `load_balance.type`
on the realm that indicates the type of load balancing. Valid options are:
* `failover` - the first server in the list will be used until it fails and then additional servers will be tried until
one succeeds. The first successful server will be used from now on. This is the default.
* `round_robin` - continuously iterates through the list of servers for each new connection. If a server is down,
the iteration will continue until a successful connection is made. The downfall here is that the list does not
get reordered on a down server, so there is overhead for always trying the servers in order.
* `dns_failover` - This server set takes a single URL that uses a DNS that will resolve to multiple IP addresses.
Connections will be consistently attempted to servers in the order they are retrieved from the name service; there
is no re-ordering and the first successful connection will be used.
* `dns_round_robin` - This server set takes a single URL that uses a DNS that will resolve to multiple IP addresses.
The addresses retrieved from the name service will connected to in the same order as `round_robin`.
Closeselastic/elasticsearch#31
Original commit: elastic/x-pack-elasticsearch@9ce9a1bf23
In order to correctly check for equality in an EmailAction, all the email
attachments have to implement equals/hashCode methods.
This has already been added to the 2.x branch in elastic/x-pack@ebde22507f
Original commit: elastic/x-pack-elasticsearch@bb980ea934
This feature is mainly done for the integration with the commercial reporting, but can be used
for anything else as well.
This adds a `attachments` to the email configuration, which can be used like this
```
"attachments" : {
"some_id" : {
"http" : {
"request" : {
"url" : "http://example.org/foo.pdf"
}
}
},
"other_id" : {
"data" : {
"format" : "json"
}
}
}
```
The main reason to pick this format is extensibility. If we would like to support another
attachment type, like an file reader, we could do so easily from an API point of view.
Closeselastic/elasticsearch#870
Original commit: elastic/x-pack-elasticsearch@66d14be965
This commit removes the current implementation in HttpExporter so that it does not automatically clean indices anymore.
Original commit: elastic/x-pack-elasticsearch@7d30338355
In order to have a shortcut for the execution of a watch and
specifying the record_execution and ignore_condition booleans,
so are now supported in the HTTP request parameters as well.
Closeselastic/elasticsearch#918
Original commit: elastic/x-pack-elasticsearch@bed5da40b7
* This action enables sending notifications to pager duty services.
* Utilizes pager duty's REST API
* Similar to the `email`, `hipchat` and `slack` actions, multiple `pagerduty` accounts can be configured, each with its own Service API key
* A `pagerduty` account is roughly mapped to a service in your pagerduty service.
* `pagerduty` actions are associated with an account, or if not, their events will be sent via the default account.
* An incident can be acknowledged, resolved or triggered
Closeselastic/elasticsearch#492
Original commit: elastic/x-pack-elasticsearch@72cc21d119
This commit merge the ClusterInfoCollector in the ClusterStatsCollector so that cluster stats are retrieved only once.
Original commit: elastic/x-pack-elasticsearch@fe70149210
This commit changes the templates so that they are now versionned using a number (starting from 1). This number is used in index templates names (ex: .marvel-es-data-1, .marvel-es-1) as well as in indices names (ex: .marvel-es-1-2015-12-30, .marvel-es-data-1).
If the template does not exist, it is created. Otherwise nothing (no update) is done.
Original commit: elastic/x-pack-elasticsearch@66c1a8bed0
This commit enable tribe nodes support for Marvel. It avoid ElasticsearchSecurityException when a tribe node is connected to a cluster that has been configured for both Shield and Marvel by loading the MarvelShieldIntegration support on tribe node even if marvel.enabled is set to false. It also allows tribe nodes to be monitored using Marvel with their own marvel settings.
closeselastic/elasticsearch#1088
Original commit: elastic/x-pack-elasticsearch@e0401c1288
Instead of having to specify `scheme`, `host`, `port`, `path` and `params` each
one can now use the `url` field and specify something like `http://example.org/foo?bar=baz`
This works for HttpRequest and HttpRequestTemplate
Original commit: elastic/x-pack-elasticsearch@8c052cf419
Because of specifying the concrete implementations in WatcherPlugin.nodeServices(),
all of those services got created twice for each node (guice requires the interface here).
This resulted in NPEs as the first instantiation did not run doStart() and thus had empty
settings, that resulted in a NPE.
Closeselastic/elasticsearch#1179
Original commit: elastic/x-pack-elasticsearch@c6a1f5093f
* allow bulk requests with updates if fls/dls isn't configured for targetted index
* throw a hard error if the current call can't resolve an associated transport request
marvel: node stats collector should use the client instead of the internal apis, because otherwise the index searcher wrapper can't locate the transport request that is associated with current call and would then throw an exception, which then prevents the marvel agent from collecting stats.
* if both field or document level security is enabled then all forbidden operations should fail
Original commit: elastic/x-pack-elasticsearch@b2c40d6559
The REST authenticate API can be used to test credentials and retrieve details about the
user that has been authenticated. This currently includes retrieving the user's username
and roles.
Closeselastic/elasticsearch#1019
Original commit: elastic/x-pack-elasticsearch@117a226187
This commit removes usage of Collections#shuffle(List) as this is a
forbidden method in upstream Elasticsearch.
Relates elastic/elasticsearchelastic/elasticsearch#15287, relates elastic/elasticsearchelastic/elasticsearch#15299
Original commit: elastic/x-pack-elasticsearch@2943569ee1
This commit simplifies xcontent usage in x-pack in a way that is compatible with
elastic/elasticsearchelastic/elasticsearch#15358.
Original commit: elastic/x-pack-elasticsearch@0d6ce5f22f
Squashed commit of the following:
commit 54de841112778b01e817e465d2f59840970bb4f3
Author: Robert Muir <rmuir@apache.org>
Date: Thu Dec 10 08:47:12 2015 -0500
see exactly how far MessyTestPlugin can go (unit tests depend on both groovy and mustache!)
commit 743b5a4e0cc9a05e307339dfcb4569feed31f337
Author: Robert Muir <rmuir@apache.org>
Date: Thu Dec 10 08:35:03 2015 -0500
fix x-pack compile/tests to work with https://github.com/elastic/elasticsearch/pull/15328
Original commit: elastic/x-pack-elasticsearch@4307bb163b
This commit update the Local & Http exporters so that they have the same behavior. Exporters now try to update the mappings of the current marvel indices if they exist (only new document types will be added)
It also adds dedicated tests to verify that the Marvel template is correctly updated depending of the current template version.
Original commit: elastic/x-pack-elasticsearch@ac650f5aba
When the logging audit trail is configured to add the node hostname or ip address
as a prefix, the logging audit trail can invoke guice dependency injection issues since
the transport that is injected is a proxy. This change makes the logging audit trail a
lifecycle component and waits for the transport to be started before initializing the
prefix.
Closeselastic/elasticsearch#1104
Original commit: elastic/x-pack-elasticsearch@3b19283969
In order to be able to send emails, the thread context classs loader
must be set to the watcher plugin class loader (and reset afterwards),
otherwise some javax.activation classes needed to create mime attachments
cannot be found.
In addition the activation jar dependency has been removed.
Closeselastic/elasticsearch#1066Closeselastic/elasticsearch#708
Original commit: elastic/x-pack-elasticsearch@10042375ec
Since we combined all of the plugins marvel will also be enabled and will cause
the cluster to never get to a green state since we only have a single node.
Original commit: elastic/x-pack-elasticsearch@60fbfa97f1
* move static initialization hack for UnboundID Debug to XPackPlugin
* cleanup bundlePlugin calls in build file
* properly disable watcher and marvel for shield core tests
Original commit: elastic/x-pack-elasticsearch@2b89cf2225
This commit fixes the integration tests and qa test after the reorganization to be
packaged as a single plugin.
Original commit: elastic/x-pack-elasticsearch@d6f488627f
- added the license-plugin dir as a src dir
- Made XPackPlugin customizable for tests. Now tests that need to plug in different behaviours (e.g. a custom license plugin) can do that by simply resetting the relevant delegee plugins (i.e. shield, marvel, license, watcher) in the ctor of a new XPackPlugin.
Original commit: elastic/x-pack-elasticsearch@ab83104955
Jason Tedor
Jim Ferenczi
Tanguy Leroux
jaymode
Ryan Ernst
Simon Willnauer
Nik Everett
Martijn van Groningen
uboness
Alexander Reelsen
Boaz Leskes
Ali Beyad
Jay Modi
Adrien Grand
Areek Zillur
markharwood
javanna
Lee Hinman
Daniel Mitterdorfer
Igor Motov
Robert Muir
Yannick Welsch
Simon Willnauer
Zachary Tong
Jack Conradson
Britta Weber