This commit modifies the construction of the poll interval setting in
the native realm tests in response to upstream change
elastic/elasticsearchelastic/elasticsearch#2f638b5a23597967a98b1ced1deac91d64af5a44.
Original commit: elastic/x-pack-elasticsearch@c6f60f51f4
This adds the `bin/shield/migrate` tool that allows migrating users and
roles from the files to the native (API-based) store.
It looks like this:
```
λ bin/shield/migrate native -U http://localhost:9200 -u test_user -p changeme -n lee,foo -r role1,role2,role3,role4,foo
starting migration of users and roles...
importing users from [/home/hinmanm/scratch/elasticsearch-2.4.0-SNAPSHOT/config/shield/users]...
found existing users: [test_user, joe3, joe2]
migrating user [lee]
{"user":{"created":true}}
no user [foo] found, skipping
importing roles from [/home/hinmanm/scratch/elasticsearch-2.4.0-SNAPSHOT/config/shield/roles.yml]...
found existing roles: [marvel_user, role_query_fields, admin_role, role3, admin, remote_marvel_agent, power_user, role_new_format_name_array, role_run_as, logstash, role_fields, role_run_as1, role_new_format, kibana4_server, user, transport_client, role1.ab, role_query]
migrating role [role1]
{"role":{"created":true}}
migrating role [role2]
{"role":{"created":true}}
role [role3] already exists, skipping
migrating role [role4]
failed to migrate role [role4] with body: {"indices":[{"names":["idx2"]},{"names":["idx2"]},{"names":["idx1"]}]}
java.io.IOException: {"error":{"root_cause":[{"type":"parse_exception","reason":"failed to parse indices privileges for role [role4]. missing required [privileges] field"}],"type":"parse_exception","reason":"failed to parse indices privileges for role [role4]. missing required [privileges] field"},"status":400}
at org.elasticsearch.shield.authc.esusers.tool.ESNativeRealmMigrateTool$MigrateUserOrRoles.postURL(ESNativeRealmMigrateTool.java:206)
at org.elasticsearch.shield.authc.esusers.tool.ESNativeRealmMigrateTool$MigrateUserOrRoles.importRoles(ESNativeRealmMigrateTool.java:389)
at org.elasticsearch.shield.authc.esusers.tool.ESNativeRealmMigrateTool$MigrateUserOrRoles.execute(ESNativeRealmMigrateTool.java:171)
at org.elasticsearch.common.cli.CliTool.execute(CliTool.java:153)
at org.elasticsearch.shield.authc.esusers.tool.ESNativeRealmMigrateTool.main(ESNativeRealmMigrateTool.java:91)
Caused by: java.io.IOException: Server returned HTTP response code: 400 for URL: http://localhost:9200/_shield/role/role4
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1840)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441)
at org.elasticsearch.shield.authc.esusers.tool.ESNativeRealmMigrateTool$MigrateUserOrRoles.postURL(ESNativeRealmMigrateTool.java:192)
... 4 more
no role [foo] found, skipping
users and roles imported.
```
Original commit: elastic/x-pack-elasticsearch@3ce47c0ffd
If an attachment is configured of disposition type INLINE, and is referred to
in HTML body parts, then some email clients can display images inside of an HTML
email and refer to those attachments.
Watcher already had support for inlined attachments, however this could not be configured
from a watch, but just via the Java API. Also it was not tested.
This commit changes the attachment to decide on creation if it should be inline or a regular
attachment and adds a test.
Relates elastic/elasticsearch#2381
Relates elastic/elasticsearch#2464Closeselastic/elasticsearch#2557
Original commit: elastic/x-pack-elasticsearch@84935ffb18
When parsing the privileges, we now no longer throw an exception if
there haven't been any names parsed out. This is not an issue though,
because we validate that the `names` array is not empty when we parse
it, and that it's not `null` before returning from the function.
Adds a rest test that sends things out of order to test this still
works.
Resolveselastic/elasticsearch#2606
Original commit: elastic/x-pack-elasticsearch@62a38bea8f
Add new REST API endpoint to acknoweldging actions. The old endpoints
have not been removed as part of this PR, but can be in the next major
version.
Update the documentation to remove the parameter based example, and
mention that the old endpoints are going to be removed in the future.
Closeselastic/elasticsearch#2517
Original commit: elastic/x-pack-elasticsearch@e2558e9e1f
The search type is overridden from default to query_and_fetch by the search action if the number of shards to request is equal to 1
Original commit: elastic/x-pack-elasticsearch@a9552c697c
Template support is going to be removed from the Search API to its own Search Template API in the lang-mustache module (see elastic/elasticsearch#17906, elastic/elasticsearch#18765). This commit changes Watcher's SearchInput and SearchTransform classes so that it now uses a WatcherSearchTemplateRequest that contains both the search request and the template. Search request and template are rendered using WatcherSearchTemplateRequestService before being executed.
Original commit: elastic/x-pack-elasticsearch@bfa16ab80f
We were ignoring the response code which is always 401 because the license is not good to start watcher. Plus all tests run fine without these methods.
Original commit: elastic/x-pack-elasticsearch@f93e1c2777
The watch index action was using the _timestamp field by default.
This functionality now needs to be configured explicitely for a special
field that is part of that document which is going to be indexed.
Relates elastic/elasticsearchelastic/elasticsearch#18980
Original commit: elastic/x-pack-elasticsearch@dfa4cf2296
This class should have never existed, I fixed all places where we messed aroudn with this
and resolved dependencies or let guice deal with it.
Original commit: elastic/x-pack-elasticsearch@6a42c4153d
This commit handles the use of `*` as a field in a role as effectively disabling field level
security. We do this to take advantage of caches that we disable when field level security
is active.
See elastic/elasticsearch#2407
Original commit: elastic/x-pack-elasticsearch@d96e18d57c
This change makes the default role check the authenticating realm when authorizing
a request for the current user (or run as user) where the user is trying to change their
own password. We need to do this, otherwise we open up the potential of a user in one
realm changing the password of a user in another realm.
As part of this work, the authentication service has been refactored and simplified. A
new object, Authentication, is now returned when authenticating. Currently, this object
contains the user, authenticating realm information, and if it is a run as request the
information of the realm that looked up the user.
Closeselastic/elasticsearch#2089
Original commit: elastic/x-pack-elasticsearch@3fd9c37a16
The WWW-Authenticate header can optionally specify the charset that the server uses after
decoding credentials. If this is not specified, most clients will limit the available characters to
ISO-8859-1, which causes issues for certain characters.
See RFC 7617
Closeselastic/elasticsearch#2290
Original commit: elastic/x-pack-elasticsearch@44411eebe7
The active state was not serialized in the PutWatchRequest leading to
to always setting it to active, when a different node than the master
node was hit with a put watch request.
Closeselastic/elasticsearch#2490
Original commit: elastic/x-pack-elasticsearch@060c0fa35f
s/request.setRefresh/request.setRefreshPolicy/
setRefresh is still supported on the builder for backwards
compatibility but not on the request itself.
Original commit: elastic/x-pack-elasticsearch@8763e2e65f
A RestClient instance is now created whenever EsIntegTestCase#getRestClient is invoked for the first time. It is then kept until the cluster is cleared (depending on the cluster scope of the test).
Renamed other two restClient methods to createRestClient, as that instance needs to be closed and managed in the tests.
Original commit: elastic/x-pack-elasticsearch@3a9d6f6e90
We still have a wrapper called RestTestClient that is very specific to Rest tests, as well as RestTestResponse etc. but all the low level bits around http connections etc. are now handled by RestClient.
Original commit: elastic/x-pack-elasticsearch@304487ecc5
Removes handlers internal:cluster/node/index/deleted and internal:cluster/node/index_store/deleted that have been removed in core as part of elastic/elasticsearchelastic/elasticsearch#18602
Original commit: elastic/x-pack-elasticsearch@e040871e5a
This commit fixes the version compatability test by updating the version
to reflect the current version in core.
Original commit: elastic/x-pack-elasticsearch@0bb6dbc1c3
If a chained input was used, and inside of this a search input was
used, that hat dots in its field names somewhere (like when sorting
or using a compare condition), then storing this in the history failed.
The reason for this was the broken watch history template, that did not take
nested requests bodies into account and thus tried to create an index mapping
for requests that were inside of a chained input.
This commit fixes the watch history index template.
Closeselastic/elasticsearch#2338
Original commit: elastic/x-pack-elasticsearch@d9f48234d3
This is a companion commit to elastic/elasticsearchelastic/elasticsearch#18514, fixing issues introduced by adding dedicated master nodes to the test infra
Original commit: elastic/x-pack-elasticsearch@8c0571f2de
This changes the IndexAuditTrailTests to use the actual timestamp of the message being indexed to determine
the index name. Some build failures occurred due to running right at the change of an hour and the rollover was
set to hourly. So the message was indexed in one index and the test expected a different index.
Original commit: elastic/x-pack-elasticsearch@9dd5012a73
This allows the colspan/rowspan attr on td/tr as well as
border/cellpadding attrs on table elements.
Original commit: elastic/x-pack-elasticsearch@e0b989f0ac
This commit removes the code to auto generate a ssl certificate on startup and disables ssl
on the transport layer by default.
Original commit: elastic/x-pack-elasticsearch@1dc9b17842
This api now just redirects to search api. All the special percolator logic has been replaced by a query that uses the Lucene index. (no caching of queries upon loading shards)
So verifying these deprecated actions is no longer needed
Original commit: elastic/x-pack-elasticsearch@da6d66fcb4
This api now just redirects to search api. All the special percolator logic has been replaced by a query that uses the Lucene index. (no caching of queries upon loading shards)
So these special tests are no longer needed
Original commit: elastic/x-pack-elasticsearch@335d6554fb
This commit adds access to the reporting indices for the role that the Kibana server role has
access to. This needed so that the server can use the async queue. Additionally the kibana
server should have access to .kibana*
Closeselastic/elasticsearch#2323
Original commit: elastic/x-pack-elasticsearch@e930e9d872
There were two api with same name, depending on the platform one or the other was being loaded first, hence the xpack info tests may fail due to unsupported params being used.
Original commit: elastic/x-pack-elasticsearch@bd44eef3cc
Also,
- changed the anonymous username to `_anonymous` (used to be `__es_anonymous_user` which I found needlessly, overly, redundantly and not to mention unnecessarily complex 🤷)
- changed the system username and role name to `_system` (used to be `__es_system_user` and `__es_system_role`... it introduced gratuitous and totally un-called for naming complexity 🤦)
Closeselastic/elasticsearch#2079
Original commit: elastic/x-pack-elasticsearch@63b6de2bba
This is the xplugins side of elastic/elasticsearchelastic/elasticsearch#18496
Most of the changes here are related to javax.activation.
Original commit: elastic/x-pack-elasticsearch@2a47f94ab5
- if active, `file` realm size
- if active, `native` realm size
- if active, `ldap` realm size, whether SSL is used, load balance type used, user search used
- if active, `active_directory` realm size, whether SSL is used, load balance type used
`size` is scale estimation based on the local cache. Scales are: `small` (under 10 users), `medium` (under 50 users), `large` (under 250 users) and `x-large` (above 250 users).
Original commit: elastic/x-pack-elasticsearch@c6efb17aa4
- Each `XPackFeatureSet` can now return a `Usage` object that encapsulates the feature usage stats of the set
- A new `/_xpack/usage` REST API is introduced to access the usage stats of all features
- Intentionally not explicitly exposing the API in the `XPackClient` as this API is primarily meant for use by Kibana X-Pack (that said, it is still possible to call this API from the transport client using the `XPathUsageRequestBuilder`)
- For now the usage stats that are returned are minimal, once this infrastructure is in, we'll start adding more stats
Relates to elastic/elasticsearch#2210
Original commit: elastic/x-pack-elasticsearch@d651fe4b01
We check for an expected length but this is only valid if the address can be resolved and on some systems
127.0.0.1 may not map to a name.
Original commit: elastic/x-pack-elasticsearch@2f7c8da242
There is a race condition between the smoke tests that get run because of the teardown conditions of
REST tests. By splitting them, we can avoid the unrealistic scenario/race condition.
Original commit: elastic/x-pack-elasticsearch@f95ae0e595
This PR just drops the - in the existing REST API name by changing the base MonitoringRestHandler to extend
XPackRestHandler instead of BaseRestHandler directly, and using its URI_BASE.
Original commit: elastic/x-pack-elasticsearch@1bed2dba31
This commit fixes some test compilation issues due to upstream changes
in core that added I/O statistics on Linux.
Original commit: elastic/x-pack-elasticsearch@a0877aa0aa
The general use case is to provide only the {type} to the bulk and even that may not be specified. However,
by setting it up as
/_x-pack/monitoring/{type}/_bulk
/_x-pack/monitoring/{index}/{type}/_bulk
it fails to properly recognize the {type} parameter because the PathTrie that gets generated sees two wildcards
at the same location and the last one specified wins -- {index}. As a result, it's impossible to only set the
{type} without making the PathTrie logic convoluted for a niche use case (a list to try instead of a single path).
his fixes the issue by removing the completely unused option: you can no longer specify {index} outside of
individual bulk indexing operations. If we see a need to bring it back, then we can add it as an API param, but
that is an unusual place for the index field and I do not expect it to be needed that frequently.
Original commit: elastic/x-pack-elasticsearch@40d0d05404
Currently, license notification scheme treats no license (before trial license is auto-generated)
and a license tombstone in the cluster state in the same way. This caused a bug where licencees
were not notified of explicit license removal. Now, the notification scheme explicitly handles
license tombstone to notify the licensees and handles the case for no license in cluster state
as before.
Original commit: elastic/x-pack-elasticsearch@c90ec23398
- Introduced a `MISSING` operation mode
- now when the license is removed (and a tombstone license is placed), the licensees get notified with a `MISSING` license status
- the monitoring, security and watcher licensees were updated
Original commit: elastic/x-pack-elasticsearch@650d940666
Some changes in elastic/x-pack@d13557c517 change the testIgnoreTimestampedIndicesInOtherVersions method and it now sometime fails.
This commit revert the previous behavior of the test and ensures that at least 1 index is cleaned up/deleted in each test.
Original commit: elastic/x-pack-elasticsearch@3c6acb4ff8
This adds it so that a system can specify "_data" as the index to index into the
data index (without having to know its name). _Not_ supplying an index will use
the timestamped index. Any other index name (including wrong case) is invalid.
Original commit: elastic/x-pack-elasticsearch@6eeadfb3c8
With this commit we free all bytes reserved on the request circuit breaker.
Relates elastic/elasticsearchelastic/elasticsearch#18144
Original commit: elastic/x-pack-elasticsearch@3986436b8c
This changes the loose usage of onOrBefore to equals so that when we add beta1, this test fails
again.
Original commit: elastic/x-pack-elasticsearch@fe4f2cbdf0
This changes the security endpoints to _xpack/security, fixes the rest api spec to also use
the xpack.security prefix and adds documentation and tests.
Original commit: elastic/x-pack-elasticsearch@7977575f0e
This commit fixes an issue in HttpExporterTemplateTests caused by the
migration from Strings#splitStringToArray to String#split. Namely, the
previous would split a string like "/x/y/z/" into { "x", "y", "z" } but
the former will split this into { "", "x", "y", "z" }. This commit
modifies the test logic to respond to this change.
Original commit: elastic/x-pack-elasticsearch@c567b17180
This commit fixes an issue that was introduced in ObjectPath#evalContext
when refactoring from Strings#splitStringToArray to
String#split. Namely, the former would return an empty array when
receiving a null or empty string as input but the latter will NPE on a
null string and return an array containing the empty string on an empty
string input.
Original commit: elastic/x-pack-elasticsearch@2f509f9fa0
In order to move things from watcher to x-pack this created a notification module in x-pack.
This also means that the HTTPClient was moved up and settings have changed from
`xpack.watcher.http` to just `xpack.http`.
Further things done:
* Move http under o.e.xpack.common
* Moved secret service to o.e.xpack.common, initializing in XpackPlugin
* Moved text template to o.e.xpack.common.text
* Moved http client initialization into xpack plugin
* Renamed xpack.watcher.encrypt_sensitive_data setting, moved into Watch class
* Moved script service proxy to common
Original commit: elastic/x-pack-elasticsearch@41eb6e6946
With elastic/elasticsearchelastic/elasticsearch#7309 we enable HTTP compression by
default. However, this can pose a security risk for HTTPS
traffic (e.g. BREACH attack). Hence, we disable HTTP compression
by default again if HTTPS enabled (note that this still allows the
user to explicitly enable HTTP compression if they want to).
Relates elastic/elaticsearchelastic/elasticsearch#7309
Original commit: elastic/x-pack-elasticsearch@8da100c9a5
This refactors the org.elasticsearch.watcher over to
org.elasticsearch.xpack.watcher
This also adds all watcher actions to the KnownActionsTests,
as watcher actions had not been taken care of until here.
Original commit: elastic/x-pack-elasticsearch@a046dc7c6a
This commit adds the necessary changes to make SSL work on the transport layer by default. A large
portion of the SSL configuration/settings was re-worked with this change. Some notable highlights
include support for PEM cert/keys, reloadable SSL configuration, separate HTTP ssl configuration, and
separate LDAP configuration.
The following is a list of specific items addressed:
* `SSLSettings` renamed to `SSLConfiguration`
* `KeyConfig` and `TrustConfig` abstractions created. These hide the details of how `KeyManager[]` and `TrustManager[]` are loaded. These are also responsible for settings validation (ie keystore password is not null)
* Configuration fallback is changed. Previously any setting would fallback to the "global" value (`xpack.security.ssl.*`). Now a keystore path, key path, ca paths, or truststore path must be specified otherwise the configuration for that key/trust will fallback to the global configuration. In other words if you want to change part of a keystore or truststore in a profile you need to supply all the information. This could be considered breaking if a user relied on the old fallback
* JDK trusted certificates (`cacerts`) are trusted by default (breaking change). This can be disabled via a setting.
* We now monitor the SSL files for changes and enable dynamic reloading of the configuration. This will make it easier for users when they are getting set up with certificates so they do not need to restart every time. This can be disabled via a setting
* LDAP realms can now have their own SSL configurations
* HTTP can now have its own SSL configuration
* SSL is enabled by default on the transport layer only. Hostname verification is enabled as well. On startup if no global SSL settings are present and SSL is configured to be used, we auto generate one based on the default CA that is shipped. This process includes a best effort attempt to generate the subject alternative names.
* `xpack.security.ssl.hostname_verification` is deprecated in favor of `xpack.security.ssl.hostname_verification.enabled`
* added Bouncy Castle info to NOTICE
* consolidated NOTICE and LICENSE files
Closeselastic/elasticsearch#14Closeselastic/elasticsearch#34Closeselastic/elasticsearch#1483Closeselastic/elasticsearch#1933
Addresses security portion of elastic/elasticsearch#673
Original commit: elastic/x-pack-elasticsearch@7c359db90b
This commit makes a few modifications to the IndexAuditTrail class:
* Use `InternalAuditUser#is` to determine if the principal is the auditor when we have a user
and simply compare `InternalAuditUser#NAME` when only a string principal is available
* Remove the `Thread#setDaemon` call in the QueueConsumer as this thread should be terminated
as part of the shutdown of the node
In terms of tests, there are some issues and changes to how we test certain aspects. The muted tests
were not accurate since the tests immediately checked for the existence of an index and did not poll or
wait and this operation is asynchronous so the index could be created after the exists request was
executed. These tests were removed and a new class was added to test the muted behavior. In these
tests we override the audit trails implementation of a queue, which will set a flag to indicate a message
has been added to the queue. This is a synchronous operation so it can be checked immediately.
The other tests in the IndexAuditTrail tests remain but a few changes have been made to the execution.
* ensureYellow is called for the index we expect to be created before searching for documents
* the remote cluster is only setup at the beginning of the suite rather than before every test to ensure
quicker execution
* the maximum number of shards has been reduced to three since we do not really need up to 10 shards
for a single document
Original commit: elastic/x-pack-elasticsearch@501b6ce9da
This change adds the proper behavior for the standard license which is:
* authentication is enabled but only the reserved, native and file realms are available
* authorization is enabled
Features that are disabled:
* auditing
* ip filtering
* custom realms
* LDAP, Active Directory, PKI realms
See elastic/elasticsearch#1263
Original commit: elastic/x-pack-elasticsearch@920c045bf1
This commit removes duplicated code in the authentication service by combining
the authentication logic for rest and transport requests. As part of this we no longer
cache the authentication token since we put the user in the context and serialize the
user.
Additionally we now pass the thread context to the AuthenticationFailureHandler to
restore access to the headers and context.
Original commit: elastic/x-pack-elasticsearch@79e2375a13
In order to prevent shipping of RCs with the wrong license, this
smoke tester downloads the internal RC, installs x-pack and puts
a license in there.
if putting is successful, we can be sure, we got the right license.
Closeselastic/elasticsearch#2087
Original commit: elastic/x-pack-elasticsearch@021d228e29
build info
There are many other things that should be cleaned up around this (eg
XpackInfoResponse.BuildInfo should not exist, it is the exact same as
what XPackBuild has), but this change gets the build info output working
again.
closeselastic/elasticsearch#2116
Original commit: elastic/x-pack-elasticsearch@0730daf031
This mainly moves packages over to the x-pack directory and renames the settings
from `xpack.watcher.actions.` to `xpack.notification.`
Moved services include pagerduty, hipchat, slack and email.
Closeselastic/elasticsearch#1998
Original commit: elastic/x-pack-elasticsearch@40c16fe123
Updated okhttp and moved the jsr305 dependency into testing.
This required a minor change in tests using SSL, as otherwise
the security manager barfs, when the okhttp webserver tries
to load sun internal SSL based classes.
Original commit: elastic/x-pack-elasticsearch@77131589e0
The old implementation was to use properties at build-time. This however did not work,
as the tests could not be run in the IDE. This has been removed of monitoring for some
time already, but needs to be removed from watcher as well.
This commit uses static variables and refactors the code a bit. First, there is a generic
TemplateUtils class, to be used in monitoring and watcher. Also the watcher code has been changed
to copy the needed variables into the template registry class instead of keeping it in the
WatcherModule.
This commit also includes some refactoring to remove the version parameter in marvel, was static anyway
Closeselastic/elasticsearch#1372
Original commit: elastic/x-pack-elasticsearch@fbfc22ea09
- by default the response includes all info - build, license, features + human descriptions.
- you can still control the output using `categories` and `human` parameters
- Added docs to this API
Original commit: elastic/x-pack-elasticsearch@85115495ec
This avoids exceptional cases where node stats are not returned due to some concurrent modification.
Original commit: elastic/x-pack-elasticsearch@6f6b8ec393
Writeable#readFrom has become a method you just implement because
the interface requires it but the prefered way to actually do the
reading is a ctor that takes a StreamReader. readFrom just delegates
to the ctor. This removes readFrom entirely because it is not needed
anymore and is going away in core.
Relates to https://github.com/elastic/elasticsearch/issues/17085
Original commit: elastic/x-pack-elasticsearch@dd74db5ded
- introduced the "Feature Set" notion - graph, security, monitoring, watcher, these are all feature sets
- each feature set can be:
- `available` - indicates whether this feature set is available under the current license
- `enabled` - indicates whether this feature set is enabled (note that the feature set can be enabled, yet unavailable under the current license)
- while at it, cleaned up the main modules of watcher, security, monitoring and graph.
Original commit: elastic/x-pack-elasticsearch@5b3e19fe8c
From now on, if field level security and percolator is used then the percolator field needs to be included in the allowed fields.
Original commit: elastic/x-pack-elasticsearch@7d39b5caf6
The SessionFactory construction was calling the `ldapServers` method in the constructor,
which was fine for all of the session factories except for the ActiveDirectorySessionFactory.
The ActiveDirectorySessionFactory overrides the ldapServers method and use class variables
that are initialized in its constructor so the value was always null.
This change moves setup to an init method for objects that depend on variables set during
construction.
Closeselastic/elasticsearch#2011
Original commit: elastic/x-pack-elasticsearch@07c15ce171
Even though HEAD is a possible method, this implies that no data is returned
and thus the tests fail randomly.
If HEAD should be added to the api it needs it's own API definition IMO.
Original commit: elastic/x-pack-elasticsearch@a216393f6b
* The found-license project is removed since it is no longer needed
* The plugin-api classes have moved into the license-plugin since there is only one plugin
* The license/base project publishes the proper artifactId in the pom file
* The x-pack jar file is added as an artifact so that it can be installed
* The x-pack pom no longer declares the packaging as `zip`
* The x-pack pom uses the right artifactId for license-core
* Removed disabling of installing the x-plugins artifacts
* Cleaned up a use of guava in watcher (found when trying to remove guava as a compile
dependency but is needed by the HTML sanitizer)
* Removed the dependency on the mustache compiler since it is no longer necessary
Closeselastic/elasticsearch#1987
Original commit: elastic/x-pack-elasticsearch@9d3b50b054
Elasticsearch supports the concept of date match expressions for index names and
the authorization service was trying to authorize the names without resolving them
to their concrete index names. This change now resolves these names
Closeselastic/elasticsearch#1983
Original commit: elastic/x-pack-elasticsearch@3c6baa8e83
The license check in Watcher was issued in the wrong way,
so that new licenses were not affected by the check. This
commit explicitely lists the license types that are allowed
to execute watcher actions as well as fixing the tests.
Relates elastic/elasticsearch#1263
Original commit: elastic/x-pack-elasticsearch@afd55965b0
The eclipse compiler errors on this class because "the method containsInAnyOrder(T...) of type Matchers is not applicable as the formal varargs element type T is not accessible here". This is because the first common superclass of `XPackUser` and `KibanaUser` is `ReservedUser` which is package protected and not available to this test class. This change casts to `User` so the error does not occur in Eclipse.
Original commit: elastic/x-pack-elasticsearch@be8fa82720
Our documentation states that we have default attributes for
message.format and message.color, which in fact we do not have
as an NPE was triggered in that case.
This commit falls back to unset defaults and allows for hipchat messages
to be sent without having to configure color/format in the action
or the account.
Closeselastic/elasticsearch#1666
Original commit: elastic/x-pack-elasticsearch@bfb7e35112
The HTTP response toXContent() method contains the http response headers, which
are used as field names in Elasticsearch in the watch history.
These can contain dots, like `es.index` being returned when Elasticsearch
encounters an exception - which results in an index error.
This patch changes the dots to an underscore when calling toXContent()
Closeselastic/elasticsearch#1803
Original commit: elastic/x-pack-elasticsearch@e4070f8b70
With this commit we limit the size of all in-flight requests on
HTTP level. The size is guarded by the same circuit breaker that
is also used on transport level. Similarly, the size that is used
is HTTP content length.
Relates elastic/elasticsearchelastic/elasticsearch#16011
Original commit: elastic/x-pack-elasticsearch@318b7a4a8a
With this commit we limit the size of all in-flight requests on
transport level. The size is guarded by a circuit breaker and is
based on the content size of each request.
By default we use 100% of available heap meaning that the parent
circuit breaker will limit the maximum available size. This value
can be changed by adjusting the setting
network.breaker.inflight_requests.limit
Relates elastic/elasticsearchelastic/elasticsearch#16011
Original commit: elastic/x-pack-elasticsearch@d1c43fe8d9
This removes the old usage of NetworkAddress.formatAddress in favor of the updated version, which is just
the method renamed to NetworkAddress.format (replacing the old version of that method).
There is no impact to x-plugins beyond making the build work because all places were currently using that
method variant already.
Original commit: elastic/x-pack-elasticsearch@05f0dcfa90
This commit adds support for the change in elasticsearch where the `/` rest
endpoint now delegates to an action and can be authorized.
Original commit: elastic/x-pack-elasticsearch@8ef38ce50f
This commit adds reserved or built-in user and role support to x-pack. The reserved roles
cannot be modified by users. The reserved users also cannot be modified with the exception
of changing the password for a user.
In order to change the password for a user, a new API has been added. This API only supports
changing passwords for native and reserved users.
To support allowing a user to change their own password, a default role has been added to grant
access. This default role only grants access to user operations that pertain to the user that is
being authorized. In other words, the default role grants `joe` the ability to change their own password
but does not allow them to change the password of a different user.
Additionally, the authenticate API was made a transport action and is granted by the default role.
Closeselastic/elasticsearch#1727Closeselastic/elasticsearch#1185Closeselastic/elasticsearch#1158
Original commit: elastic/x-pack-elasticsearch@1a6689d90f
This commit migrates all of the `shield.` settings to `xpack.security.` and makes changes to
use the new Settings infrastructure in core.
As a cleanup, this commit also renames Shield to Security since this class is only in master
and will not affect 2.x.
See elastic/elasticsearch#1441
Original commit: elastic/x-pack-elasticsearch@a5a9798b1b
This commit allows authorized users to monitor the security index. This fixes an issue
with the _cat/indices api, which resolves the concrete indices using the cluster state
and then makes a indices stats request. Without this change, the api fails with an
authorization exception because it is specifically requesting the security index and the
user is not the internal user.
Closeselastic/elasticsearch#1895
Original commit: elastic/x-pack-elasticsearch@070a389833
Since elastic/elasticsearch#1832 exporters are created once, but the inner exporting bulks must be instanciated for each export. The CountingExporter and CountingBulk have not been updated to reflect this change.
Original commit: elastic/x-pack-elasticsearch@bbbde22363
(there are other indices too, montoring indices and waiting for green fails there because these indices have replica shards)
Original commit: elastic/x-pack-elasticsearch@63dd3e6ebb
A ParseField object is now required to register queries against the SearchModule rather than the QueryParser#names method. ParseField handles camel case automatically. Also this allows us to log deprecation warnings (or fail in strict mode) when deprecated names are used for queries (e.g. "in", "mlt", "fuzzy_match" etc.)
Original commit: elastic/x-pack-elasticsearch@b0146e6e3d
This adds default timeouts to the SMTP configuration to prevent infinite timeouts, that can lead to stuck watches.
This also requires to use time values instead of just milliseconds.
Closeselastic/elasticsearch#1830
Original commit: elastic/x-pack-elasticsearch@c886da7bff
One test was missing the check if posix permissions are supported by the file system.
As it does not make sense to not run 50% of the tests in 50% of the cases, the
logic to configure which capabilities a in-memory FS should have has been moved
into each test.
Original commit: elastic/x-pack-elasticsearch@59a32ea26d
When called without arguments, systemkey tool returned with an AIOOE.
This fixes the issue, but also ports over the tests to jimfs, so they
can actually run, as the security manager is always enabled and thus the
tests never ran before.
Closeselastic/elasticsearch#1926
Original commit: elastic/x-pack-elasticsearch@887b681607
This commit adds all the following segments stats to the current node_stats document:
- memory_in_bytes
- terms_memory_in_bytes
- stored_fields_memory_in_bytes
- term_vectors_memory_in_bytes
- norms_memory_in_bytes
- doc_values_memory_in_bytes
- index_writer_memory_in_bytes
- version_map_memory_in_bytes
- fixed_bit_set_memory_in_bytes
Original commit: elastic/x-pack-elasticsearch@ea4b8034ba
This commit makes the user and roles poller use a self rescheduling runnable to schedule the
next run of the poller rather than using scheduleWithFixedDelay. This is done because the
pollers perform blocking I/O operations and everything using that thread pool method runs on
the schedule thread and because of this, in certain situations this can lead to a deadlock which
will prevent the cluster from forming.
Original commit: elastic/x-pack-elasticsearch@9fd0748c8c
Too many places are checking for enumerations when they're really more interested in a "higher" level of
information. This will help with the forthcoming addition of the STANDARD operation mode as well.
Original commit: elastic/x-pack-elasticsearch@2799c27e19
This commit fixes an issue in synchronization in Exporters class. The export() method is synchronized and when used with LocalExport can provoke a deadlock. LocalExporter exports data locally using bulk requests that can trigger cluster state updates for mapping updates. If a exporters settings update sneaks in, the settings update waits for the export to terminate but the export waits for the settings to be updated... and boom.
This commit removes the synchronized and refactor Exporters/LocalExporter to use state and dedicated instance of LocalBulk for each export so that synchronizing methods is not necessary anymore.
It also lower down some random settings in MonitoringBulkTests because the previous settings almost always fill the bulk thread pool.
closeselastic/elasticsearch#1769
Original commit: elastic/x-pack-elasticsearch@f50c916f8b
The view index metadata privilege did not grant access to the shard level field mapping
action or the shard level validate query action. This caused the apis to restrict access to
the data when it should have been allowed.
Closeselastic/elasticsearch#1827
Original commit: elastic/x-pack-elasticsearch@7832699cb6
This commit synchronizes the start of the threads that are executing monitoring bulk requests concurrently
to ensure all threads are ready before starting. Without this some threads will execute requests while
other threads are still being constructed.
Original commit: elastic/x-pack-elasticsearch@e777fb5c28
This commit changes the handling in the merge method of the IndexAccessControl class to
properly handle merging IndexAccessControl objects with differing values for the granted
flag. Prior to this commit, in a scenario where the flag differed, one IndexAccessControl granted
no access to an index, and the other granted access with DLS/FLS resulted in full access
being granted to the index.
Closeselastic/elasticsearch#1821
Original commit: elastic/x-pack-elasticsearch@e403e43689
This commit removes the parsing of the username in the request body of a put user
request. Additionally, we use the name passed into the put role request builder rather
than the name from the parsed role descriptor.
Original commit: elastic/x-pack-elasticsearch@0a085d5844
This commit is the forward port of renaming the type for esusers to file. There is no
backwards compatibility maintained here. Additionally, a few other renames and
cleanups have been made:
* `esusers` commands is now `users`
* org.elasticsearch.shield.authc.esusers -> org.elasticsearch.shield.authc.file
* Validation.ESUsers -> Validation.Users
* ESUsersTool -> UsersTool
* ESUsersToolTests -> UsersToolTests
* ESNativeUsersStore -> NativeUsersStore
* ESNativeRolesStore -> NativeRolesStore.
* org.elasticsearch.shield.authz.esnative collapsed to org.elasticsearch.shield.authz.store
* ESNativeTests -> NativeRealmIntegTests
Closeselastic/elasticsearch#1793
Original commit: elastic/x-pack-elasticsearch@d2a0c136f3
Some tests [reasonably] fail due to unexpected indices appearing in the cluster due to Watcher.
- Also had to reset shieldEnabled as a static field, which makes no sense, but tests were failing unpredictably without it
- Now they're passing unpredictably with it... will investigate
Original commit: elastic/x-pack-elasticsearch@9b6ce681d8
This is required to make sure that the integration for monitoring the Watcher Threadpool is actually working.
- Also added the full property name when the assertContains check fails
- Made shieldEnabled an instance level field rather than a static one
- Added watcherEnabled field in the same fashion (including enableWatcher method that by default randomly enables it)
- Added method to locally filter the expected field names based on watcher being enabled for the failing test
Original commit: elastic/x-pack-elasticsearch@2c56e2f26f
Node roles are now serialized as well, they are not part of the node attributes anymore. DiscoveryNodeService takes care of dividing settings into attributes and roles. DiscoveryNode always requires to pass in attributes and roles separately.
Original commit: elastic/x-pack-elasticsearch@32a4eb0fb4
This is needed in order to make `bin/elasticsearch-plugin install xpack`
work, as it expects the plugin in a certain path.
Original commit: elastic/x-pack-elasticsearch@252c55e5a8
This commit is the forward port of fixes made in 2.3 for the roles and users
pollers. The pollers now refresh since not all operations are guaranteed to
refresh.
The clear roles tests are also made more evil since the poller runs at different
intervals on each node and can sometimes run almost continuously. The
modification requests now randomize if they refresh or not as well.
Original commit: elastic/x-pack-elasticsearch@f61159c40a
Users running the `CleanerService` should not be able to disable it (via a `-1` as the time setting) because they'll just shoot themselves in the foot. This PR changes the behavior to allow extensive amounts via the setting (e.g., they could set it to 2 years). By doing this via the `Setting`, we can avoid a lot of boilerplate code for verification as well. If we decide to allow it to be disabled, then the setting should be explicit. I've found that users tend to not understand setting times to `-1`.
With the internal `IndicesCleaner` runnable, I have also moved the rescheduling code to `onAfter` so that it always happens, even if the license makes it temporarily invalid.
I also think that we should allow the user to dynamically set the setting regardless of it being allowed -- and warn on it. This way they can set it when it's expired or during the trial, but it will take effect when they apply the paid license. I think that this will provide a better user experience so that they do not have to remember to re-set it later.
This also removes the `LocalExporter`-specific setting that allowed it to override the global retention. If we ever add another listener, then we should add exporter-specific settings to support this kind of functionality.
Adds some tests for the settings as well as for the service, while also removing now unneeded ones.
Original commit: elastic/x-pack-elasticsearch@3abd41807e
Currently it first creates a scorer, then checks if the role bits are sparse,
and falls back to the bulk scorer if they are dense. The issue is that creating
scorers and bulk scorers is very expensive on some queries such as ranges,
prefix and terms queries. So it should rather check whether bits are sparse
first in order to decide whether to use the scorer or bulk scorer.
Original commit: elastic/x-pack-elasticsearch@067d630099
Similar to WatcherClientProxy, the elasticsearch client used in exporters must be proxied to avoid circular dependencies at Guice's injection time. This commit add a MonitoringClientProxy as well as a MonitoringClient to be used later in monitoring's transport actions.
(cherry picked from commit b70c095)
Original commit: elastic/x-pack-elasticsearch@17327cffe5
Nik Everett
Lee Hinman
jaymode
Tanguy Leroux
Ryan Ernst
Areek Zillur
Yannick Welsch
Jason Tedor
Boaz Leskes
Luca Cavanna
Alexander Reelsen
javanna
Martijn van Groningen
Daniel Mitterdorfer
Simon Willnauer
Tim Sullivan
Timothy Sullivan
Jonathan Budzenski
Jim Ferenczi
Adrien Grand
markharwood
Christoph Büscher
Robert Muir
uboness
Ali Beyad
Chris Earle
Jay Greenberg
PhaedrusTheGreek
Robert Muir
Colin Goodheart-Smithe
Simon Willnauer