* Changed ML action names to allow distinguishing of admin and read-only actions
using wildcards
* Added manage_ml and monitor_ml built-in privileges as subsets of the existing
manage and monitor privileges
* Added out-of-the-box machine_learning_admin and machine_learning_user roles
* Changed machine learning results endpoints to use a NodeClient rather than an
InternalClient when searching for results so that index/document level permissions
applied to ML results are respected
Original commit: elastic/x-pack-elasticsearch@eee800aaa8
This commit adds an end-point to force deletion of expired data:
DELETE /_xpack/ml/_delete_expired_data
A few other things are changed too:
- Delete expired results from now rather than start of day
- Rename MlDaily{Management -> Maintenance}Service
- Refresh job indices when job is closing to ensure latest result
visibility
- Commit results when quantiles are persisted to ensure they are visible
for renormalization
Original commit: elastic/x-pack-elasticsearch@8ca5272a94
The yaml test runner now throws error when skip or do sections are malformed, such as they don't start with the proper token (START_OBJECT). That signals bad indentation, which was previously ignored. Thanks (or due to) our pull parsing code, we were still able to properly parse the sections, yet other runners weren't able to.
Original commit: elastic/x-pack-elasticsearch@920201207c
This commit adds the ability for x-pack extensions to optionally
provide custom roles providers, which are used to resolve any roles
into role descriptors that are not found in the reserved or native
realms. This feature enables the ability to define and provide roles
from other sources, without having to pre-define such roles in the security
config files.
relates elastic/x-pack-elasticsearch#77
Original commit: elastic/x-pack-elasticsearch@bbbe7a49bf
* Tribe node security tests with external clusters
This PR adds a qa module for security tests with tribe node
using external clusters. Existing SecurityTribeIT tests
have been ported to use external clusters with tribe setup
as a first step.
Currently the ports to the external clusters are passed to the
integration tests through system properties and external clusters
are built on test setup (the code for building external clusters is
copied from ESIntegTestCase). This is a WIP as we need a
more generic way to facilitate testing tribe setup with external
clusters. thoughts welcome.
* incorporate feedback
* update to master
Original commit: elastic/x-pack-elasticsearch@686887ca91
Currently, both the NativeUsersStore and NativeRolesStore can undergo
multiple state transitions. This is done primarily to check if the
security index is usable before it proceeds. However, such checks are
only needed for the tests, because if the security index is unavailable
when it is needed, the downstream actions invoked by the
NativeUsersStore and NativeRolesStore will throw the appropriate
exceptions notifying of that condition. In addition, both the
NativeUsersStore and NativeRolesStore had much duplicate code that
listened for cluster state changes and made the exact same state
transitions.
This commit removes the complicated state transitions in both classes
and enables both classes to use the SecurityTemplateService to monitor
all of the security index lifecycle changes they need to be aware of.
This commit also moves the logic for determining if the security index
needs template and/or mapping updates to the SecurityLifecycleService,
and makes the NativeRealmMigrator solely responsible for applying the
updates.
Original commit: elastic/x-pack-elasticsearch@b31d144597
This commit removes the usages and definition of deprecated methods that do not accept the XContentType for the source.
Additionally, usages of *Entity classes from the http client library have been changed to always specify the content
type.
Original commit: elastic/x-pack-elasticsearch@29d336a008
Also added a test to multi node qa module that tests the datafeeder, which should have caught this.
Original commit: elastic/x-pack-elasticsearch@89e4875f6c
Was accidentally injecting the script object, not the string version of the code. Also
added a check so we only inject for Painless scripts (and not groovy, etc).
Minor style tweaks too.
Original commit: elastic/x-pack-elasticsearch@58c7275bd8
This change adds licensing to the maching learning feature, and only allows access to machine learning if a trial or platinum license is installed.
Further, this change also renames `MlPlugin` to `MachineLearning` in line with the other feature plugin names and move the enabled setting to `XPackSettings`
Original commit: elastic/x-pack-elasticsearch@48ea9d781b
Adds a new `xpack.security.authc.accept_default_password` setting that defaults to `true`. If it is set to false, then the default password is not accepted in the reserved realm.
Adds a bootstrap check that the above setting must be set to `false` if security is enabled.
Adds docs for the new setting and bootstrap.
Changed `/_enable` and `/_disable`, to store a blank password if the user record did not previously exist, which is interpreted to mean "treat this user as having the default password". The previous functionality would explicitly set the user's password to `changeme`, which would then prevent the new configuration setting from doing its job.
For any existing reserved users that had their password set to `changeme`, migrates them to the blank password (per above paragraph)
Closes: elastic/elasticsearch#4333
Original commit: elastic/x-pack-elasticsearch@db64564093
This commit brings back support an auto-generated certificate and private key for
transport traffic. The auto-generated certificate and key can only be used in development
mode; when moving to production a key and certificate must be provided.
For the edge case of a user not wanting to encrypt their traffic, the user can set
the cipher_suites setting to `TLS_RSA_WITH_NULL_SHA256` or a like cipher, but a key/cert
is still required.
Closeselastic/elasticsearch#4332
Original commit: elastic/x-pack-elasticsearch@b7a1e629f5
This change adapts x-pack to pass on the parsed XContentType from rest requests to transport
requests and use this value in place of attempting to auto-detect the content type.
Original commit: elastic/x-pack-elasticsearch@57475fd403
If `domainSplit(` is detected in an inline script, the function and params are injected into
the script.
The majority of this PR is actually test-related. Adds a unit test to check for the injected
script/params. Also adds another QA test which -- through a very round-about mechanism --
confirms that the injected script compiles and functions correctly. The QA test can
be simplified greatly once the Preview API is added.
Original commit: elastic/x-pack-elasticsearch@c7c35a982c
This contains the Painless-based DomainSplit function, generated static maps and basic tests. Due to cross-module complications, the tests are run by executing searches with script_fields and checking the response
Original commit: elastic/x-pack-elasticsearch@c6c2942e01
This matches the way tests that need to run without an Elasticsearch
bootstrap are run in core Elasticsearch. This should make merging to
x-pack easier.
Note that the no bootstrap tests now run after the integration tests, but
this doesn't really matter.
Original commit: elastic/x-pack-elasticsearch@5547f457b6
* Gets build to use elasticsearch-extras
Also adds ci script for building repo on CI servers
To use this change you need to:
1. Clone elasticsearch: `git@github.com:elastic/elasticsearch.git`
2. create a directory at the same level as elasticsearch called `elasticsearch-extra`
3. Clone this repository into the `elasticsearch-extra` directory
4. Run `gradle build` from the `elasticsearch-extra/prelert-legacy` directory or run `gradle :prelert-legacy:build` from the `elasticsearch directory
* Adds USE_SSH option to ci script
* iter
Original commit: elastic/x-pack-elasticsearch@ea127dfef0
The job open api starts a task and ties that AutodetectCommunicator.
The job close api is a sugar api, that uses the list and cancel task api to close a AutodetectCommunicator instance.
The flush job and post data api redirect to the node holding the job task and then delegate the flush or data to the AutodetectCommunicator instance.
Also:
* Added basic multi node cluster test.
* Fixed cluster state diffs bugs, forgot to mark ml metadata diffs as named writeable.
* Moved waiting for open job logic into OpenJobAction.TransportAction and moved the logic that was original there to a new action named InternalOpenJobAction.
Original commit: elastic/x-pack-elasticsearch@194a058dd2
The watcher tests were recently marked with `@Network`, which prevents them from
normally being run. Unfortunately, this means no tests run by default and the
entire suite fails.
Original commit: elastic/x-pack-elasticsearch@40cfc75b26
This commit delete the JIRA issues after the integration test execution. All issues from the testing project XWT are deleted, even if they have not been created during this specific test execution.
closeselastic/elasticsearch#4535
Original commit: elastic/x-pack-elasticsearch@0362463633
In Jira integration tests, some watches are triggered every second whereas they are executed using the watch execute API. This commit increases the triggering interval to 1d so that the watches are not executed on slow machines.
Original commit: elastic/x-pack-elasticsearch@4d0462bc00
This commit moves the Jira rest integration tests from the smoke-test-watcher-with-mustache project to the smoke-test-watcher project.
Original commit: elastic/x-pack-elasticsearch@c6b03d557f
Watcher: Use Apache HttpClient for internal Watcher HttpClient
The current implementation based on URLConnection has several drawbacks.
* If server returned HTTP header but then got stuck, no timeout would help, the connection remained stuck
* GET requests with a body were not supported, the method was silently changed to POST
* More complex handling of input/error stream handling, the body could not be read from a single input stream
NOTE: This is a BWC breaker. From now on every part of the URL needs to be encoded properly before it is configured in the requeust builder. This requires an upgrade of all watches.
Closeselastic/elasticsearch#1141
Original commit: elastic/x-pack-elasticsearch@bbc8f85dd8
This commit enables the Jira integration tests with the Jira project and account provided by Edward Sy.
closes elastic/infraelastic/elasticsearch#1498
Original commit: elastic/x-pack-elasticsearch@78d1005064
standalone-rest-test doesn't configure unit tests and for these
integTest only projects that is what we want.
Original commit: elastic/x-pack-elasticsearch@f576dfdfbb
It used to be that RestTestPlugin "came with" StandaloneTestBasePlugin
but we'd like to use it with BuildPlugin for the high level rest client.
Also fix some license headers.
Original commit: elastic/x-pack-elasticsearch@3d5549d170
This PR changes how we use roles and how we look at the roles of a user. Previously we looked up each role individually, parsed each into their own `Role` object, and had a wrapper that essentially served as an iterator over the roles. The same pattern was also followed for the permissions that composed a role (ClusterPermission, IndicesPermission, and RunAsPermission). This resulted in a lot of code that was hard to follow and could be inefficient.
Now, we look up the roles for a user in bulk and only get the RoleDescriptor for each role. Once all role descriptors have been retrieved, we build a single Role that represents the user's permissions and we also cache this combination for better performance as authorization can happen many times for a single top level request as we authorize the top level request and any sub requests, which could be a large number in the case of shard requests.
This change also enabled a large cleanup of our permission and privilege classes, which should reduce the footprint of what needs to be followed. Some of the notable changes are:
* Consolidation of GeneralPrivilege and AbstractAutomatonPrivilege into the Privilege class
* The DefaultRole class has been removed and the permissions it provided were moved into the AuthorizationService
* The GlobalPermission class was removed as there is a single role that represents a user's permissions
* The Global inner classes for the various permissions were removed
* The Core inner class was removed and ClusterPermission, IndexPermission, RunAsPermission became final classes instead of interfaces
* The Permission interface has been removed. The isEmpty() method defined by this interface is not needed as we can simply evaluate the permission to get the same effect
* The ClusterPermission#check method only takes the action name again
* The AutomatonPredicate class was removed and replaced by Automatons#predicate
* IndicesAccessControl objects no longer need to be merged when evaluating permissions
* MergedFieldPermissions has been removed
* The Name class that was used to hold an array of strings has been removed and replaced with the use of a Set
* Privilege resolution is more efficient by only combining automata once
Other items:
* NativeRolesStore no longer does caching, so the RoleAndVersion class could be removed
* FileRolesStore doesn't need to be an AbstractLifecycleComponent
Relates elastic/elasticsearch#4327
Original commit: elastic/x-pack-elasticsearch@c1901bc82e
In https://github.com/elastic/elasticsearch/pull/21964, index
and delete operations are executed as single item bulk requests
internally. This means index and delete operations use the
bulk transport endpoints (indices:data/write/bulk[s][p] and
indices:data/write/bulk[s][r]).
This PR adds bulk transport endpoint to 'write' and 'delete'
index privilages and adds index and delete action as composite
actions to delay the authentication to the shard level.
Original commit: elastic/x-pack-elasticsearch@2305fc9ca0
* Build: Convert xplugins to use new extra projects setup
This change makes the gradle initialization for xplugins look in the
correct location for elasticsearch, which is now as a sibling of an
elasticsearch-extra directory, with x-plugins as a child of the extra
directory.
The elasticsearch side of this change is
elastic/elasticsearchelastic/elasticsearch#21773. This change will enable renaming x-plugins
to x-pack, see elastic/elasticsearch#3643.
Original commit: elastic/x-pack-elasticsearch@09398aea5a
In order to prepare to the distributed watch execution, this commit
removes the in memory watch store.
Whenever a watch is needed now, a get request is executed and the parsing
is done. This happens when
* Put
* Get
* Ack
* Activate/Deactivate
* Execute
Note: This also means there are no usage stats currently regarding
the watch count, because we would need to execute a query. This would
require the usage stats to be async, see elastic/elasticsearch#3569
Another advantage is, that there is no dirty flag in the watch itself
needed anymore, because the watch is always the latest. Also write
operations store immediately and dont leave anything in memory.
Also ActionListener.wrap() was used a lot instead of more verbose anonmyous
inner classes.
Original commit: elastic/x-pack-elasticsearch@c47465b47c
This change removes the deprecated methods from the realm class. These methods include
blocking authentication and lookup and the lookup supported method.
Original commit: elastic/x-pack-elasticsearch@cff21e21ee
The license test for the tribe node disables xpack features but these settings
are not passed to the tribe client nodes, so they need to manually be added.
Original commit: elastic/x-pack-elasticsearch@7273817c47
Transform and condition scripts were only compiled on its initial creation, so
when a new watch is created or when the master node loads all the watches. However
changing a script (like a stored one) did not lead to any changes in the in memory
watch store and thus the old script was executed again.
We do however have a mechanism in Elasticsearch's ScriptService that already does some
caching, and should reuse that one.
Closeselastic/elasticsearch#4237
Original commit: elastic/x-pack-elasticsearch@477548e237
This commit adds basic tests that store a user and a role using the native API. The test checks
that the user and role can be used prior to starting the upgrade. The realm and roles caches are
also cleared to ensure the next authentication will require a read from the security index; this
ensures we are actually testing reads from the index.
Original commit: elastic/x-pack-elasticsearch@396862da94
This commit improves the output of the migrate tool in cases when there are errors parsing entries
in the roles or users files. This is done through the use of a logger that delegates its output to
the terminal.
Additionally, the `-c` option has been removed. This option was used to set the configuration directory
but this should be handled one way only and that is through the use of the `-Epath.conf` setting.
Closeselastic/elasticsearch#3757Closeselastic/elasticsearch#3758
Original commit: elastic/x-pack-elasticsearch@811e367766
When we encounter a composite request, we authorize at first without looking at the indices, to see whether the action can be executed at all. We then rely on the action to delegate to an inner action per sub-request, which will be authorized based on the indices it refers to. The first step works great for the simulate mode of search template, as it doesn't involve any index. The second step will make sure that when search template involves a search, it will be authorized as a normal search request would, based on the indices it reads from.
Note that the wildcard expansion happens now on the search side, it doesn't have to happen when executing the first authorization step, hence SearchTemplateRequest doesn't have to implement IndicesRequest, only SearchRequest has to (which it does already).
Closeselastic/elasticsearch#4171
Original commit: elastic/x-pack-elasticsearch@d586bd90cb
When one of the 2 nodes in the old cluster is shut down, shards that were on that node will become unassigned and be marked to be
delay-allocated, i.e. either a node with shard data for that shard must be available or the allocation of the shards will be delayed for a minute.
In the mixed cluster the replica shard might not be allocated as the primary is already on the node with the newer version and replicas are not allowed
then to be allocated to a node of an older version of ES. Once both nodes are upgraded, the delay might still be in place, and can only be nullified if there
is shard data available on the node. If there never was a shard on that node though, it will take a minute and run into the timeout checking for green.
This commit ensures that all shards are fully-allocated before we do the rolling restart scenario
Original commit: elastic/x-pack-elasticsearch@a0d9b1b043
If the primary shard of an index with (number_of_replicas > 0) ends up on a new node in a mixed cluster, the replica cannot be allocated to the old node as
the new node might have written segments that use a new postings format or codec that is not available on the older node.
As x-pack automatically creates indices with number_of_replicas > 0, for example monitoring-data-*, the test can only wait for yellow in a mixed cluster.
Original commit: elastic/x-pack-elasticsearch@945d9e3811
This commit updates the watch_history.json file so that it includes mappings for the new Jira action. It also update the JiraIssue format so that it now includes the name of the account used to create the Jira issue. It also update the REST tests to check that Jira action result are searchable and hide the user's password.
Original commit: elastic/x-pack-elasticsearch@75888f7748
This commit adds a timeout to the cluster health call that we wait on so that we can
see the status of the health request instead of getting timeouts failures with no
information to go on.
Original commit: elastic/x-pack-elasticsearch@2f34d01e00
This change allows reads of our native users and roles when the template version has not been updated to
match the current version. This is useful for rolling upgrades where the nodes are also being actively
queried and/or indexed into. Without this, we can wreak havoc on a cluster by causing exceptions during
replication, which leads to shard failures. On nodes that match the version defined in the template,
write operations are allowed since we know that we are backwards compatible in terms of format but we
may have added new fields and shouldn't index them until the mappings and template have been updated.
As part of this, the rolling upgrade tests from core were used as the basis for a very basic set of tests
for doing a rolling upgrade with x-pack.
Closeselastic/elasticsearch#4126
Original commit: elastic/x-pack-elasticsearch@9be518ef00
Currently, a tribe node ignored underlying cluster licenses
due to inablity to select an appropriate license from multiple
licenses. Now that tribe node supports merging custom metadata
(elasticsearch#elastic/elasticsearch#21552), we can enable license support in tribe
node.
Now, tribe node chooses license with the highest operation
mode from underlying cluster licenses. This commit also
adds integration tests for licensing to verify that:
- autogenerated trial license propagates to tribe node
- tribe node chooses the highest operation mode license
- removing a license from underlying cluster license is
removed from tribe
closeselastic/elasticsearch#3212
Original commit: elastic/x-pack-elasticsearch@b5c003decd
This is a followup from elastic/elasticsearchelastic/elasticsearch#21590 and needs to be
committed first or at the same time since netty_3 is removed
Original commit: elastic/x-pack-elasticsearch@131d74dd6b
This commit adds a new Gradle sub project that makes use of the VagrantTestPlugin in order to test the installation of X-Pack.
Original commit: elastic/x-pack-elasticsearch@e09db6602c
Refactored ScriptType to clean up some of the variable and method names. Added more documentation. Deprecated the 'in' ParseField in favor of 'stored' to match the indexed scripts being replaced by stored scripts.
Original commit: elastic/x-pack-elasticsearch@d7c7bd7362
Reindex has grown support for simple whitelist patterns like
`localhost:*` and lost support for `myself`.
Original commit: elastic/x-pack-elasticsearch@31d9c4dc5c
Eagerly authorizing CompositeIndicesRequests allowed the security plugin to fail fast up until now, but it makes it very hard to reason about each specific item in a multi items request. Either all items fail, or none do. We would rather want to adopt a similar behaviour to es core, where individual items fail without affecting other items that are part of the same request. We can rely on the fact that es core always authorizes both main action and every subaction too, and skip authorization for the main action. By subaction we mean either all sub search requests in msearch, as well as each shard level get in mget or shard level bulk request for bulk.
BulkRequestInterceptor was converted to intercept BulkShardRequests rather than BulkRequest as that is where bulk is authorized after this change.
Split IndicesAndAliasesResolverIntegrationTests into ReadActionsTests and WriteActionsTests as they require different set of permissions, lots of tests added.
Explicitly listing the composite actions makes sure that the actions that can bypass security are known, somebody adding a similar action must to add it to the list, so we know it doesn't happen by mistake. At this point the CompositeIndicesRequest can be used as a marker interface only (it is not really needed but can be used to verify that composite actions use a request that implements such interface).
Given that we don't authorize composite actions based on their indices anymore, but only their sub-requests which implement IndicesRequest, printing out the indices names in the audit log for requests like bulk and msearch is confusing. Removed support for that.
Authorize composite indices actions based on their name only, their indices will be authorized at the sub-request/shard level
Rather than simply granting bulk, mget, msearch etc. and relying on authorization at the sub-request/shard level, we check that the current user can at least execute the action. This justifies the grant line that gets written in the audit log, the action is potentially possible without looking at the indices. Each specific item will fail or succeed later and will yield its own specific audit log entry.
Original commit: elastic/x-pack-elasticsearch@4570caf019
Supporting allowNoIndices means that the security plugin has a behaviour much more similar to vanilla es when dealing with wildcard expressions that match no indices, or empty clusters. The default for most request is to allow no indices, but security plugin could only disallow no indices all the time up until now.
The technical problem was that when anything gets resolved to an empty set of indices, we couldn't let that go through to es core, as that would become resolved to all indices by es core, which would be a security hole. We have now found a way though to replace an empty set of indices with something that es core will for sure resolve to no indices, so we can let the request through. We simply replace empty indices with '-*'.
Multi apis requests (e.g. _msearch) have yet to be fixed, as all their indices end up in the same bucket while they should each be authorized separately, so that every specific item can fail or be let through.
Original commit: elastic/x-pack-elasticsearch@0f67a0bfea
For all the requests that support multiple indices and wildcards, hence implementing IndicesRequest.Replaceable, we replace the wildcard expressions with the explicit names of the authorized indices they match. _all or empty indices is treated as a wildcard expression. We can also honour the ignore_unavailable option by going over all the explicit names and filter out the non authorized ones when ignore_unavailable is set to true. If ignore_unavailable is set to false, we leave everything as-is, which will cause an authorization exception to be thrown if only one of those explicit indices is not authorized for the current user.
This is the first step towards resolving elastic/elasticsearch#1250. The remaining issue is that in case we are left with no indices after stripping out the ones that the user is not authorized for, we throw an authorization exception rather than returning an empty response. That will require honouring the allow_no_indices option, which will also change the behaviour when a cluster is empty.
Relates to elastic/elasticsearch#1250
Original commit: elastic/x-pack-elasticsearch@e4ca940d05
When adding a watch which has a painless component, the scriptexception
was wrapped into a deprecated exception which means, that the awesome
painless descriptions were lost. This wrapping has been removed.
Closeselastic/elasticsearch#3161
Original commit: elastic/x-pack-elasticsearch@1703fe4eb6
This test has been blacklisted and deactivated months ago. This commit reenables this test and moves it at the right place. It also change the test to use the Execute Watch API instead of being sleep based.
Original commit: elastic/x-pack-elasticsearch@e7a9689375
I think the intent here is to just test that our SSL layers work,
not invoke a long chain of keytool + openssl commands.
This simplifies the build and will work on windows.
Original commit: elastic/x-pack-elasticsearch@af07d0d4f7
This is all the tests disabled from https://github.com/elastic/x-plugins/issues/724
At least, they will be running in the build in some way. If we can fix gradle to
add plugin metadata from lang-groovy to the test classpath, security manager
can be re-enabled for these as well.
But its also only 8 tests, maybe its easier to fix them?
Original commit: elastic/x-pack-elasticsearch@a5c407b80f
This checks that all ES plugins and xplugins are installed. I also
changed the rest check to be a simple plugin count, so it does not fail
when new plugins are added.
Original commit: elastic/x-pack-elasticsearch@eaab182e43
This is the xplugins side of elastic/elasticsearchelastic/elasticsearch#14986, making use of
the simplification in configuration.
Original commit: elastic/x-pack-elasticsearch@a24ad7b08a
This adds back the shield qa rest tests module with gradle. There is
also a small fix in ShieldPlugin for a bug that was discovered around
checking for a custom query cache (which was using the node settings
instead of index settings).
Original commit: elastic/x-pack-elasticsearch@28c6d58f37
The random index template can set a number of replicas that will prevent the index
from ever being green in a single node cluster...
Original commit: elastic/x-pack-elasticsearch@81ecfe0818
- this results in a `marvel-agent-2.0.0.zip` artifact, to better differentiate it from the kibana marvel plugin
- post 2.0 when we move the marvel kibana codebase to x-plugins we'll need to differentiate between these two modules anyway.
Closeselastic/elasticsearch#846
Original commit: elastic/x-pack-elasticsearch@6b6a76f7dd
This commit removes Guava as a dependency. Note that Guava will remain
as a test-only dependency (transitively through Elasticsearch through
Jimfs).
Relates elastic/elasticsearchelastic/elasticsearch#13224
Original commit: elastic/x-pack-elasticsearch@fe23d5f25f
David Roberts
Dimitris Athanasiou
Martijn van Groningen
Luca Cavanna
Nik Everett
Ali Beyad
Areek Zillur
javanna
Ryan Ernst
Jay Modi
David Kyle
Zachary Tong
Colin Goodheart-Smithe
Simon Willnauer
Tim Vernum
Jason Tedor
Zachary Tong
Jack Conradson
Lee Hinman
Tanguy Leroux
Alexander Reelsen
jaymode
Simon Willnauer
Yannick Welsch
jaymode
Robert Muir
Daniel Mitterdorfer
Robert Muir
uboness
Konrad Beiske