This commit adds a token based access mechanism that is a subset of the OAuth 2.0 protocol. The
token mechanism takes the same values as a OAuth 2 standard (defined in RFC 6749 and RFC 6750),
but differs in that we use XContent for the body instead of form encoded values. Additionally, this
PR provides a mechanism for expiration of a token; this can be used to implement logout
functionality that prevents the token from being used again.
The actual tokens are encrypted using AES-GCM, which also provides authentication. The key for
encryption is derived from a salt value and a passphrase that is stored on each node in the
secure settings store. By default, the tokens have an expiration time of 20 minutes and is
configurable up to a maximum of one hour.
Relates elastic/x-pack-elasticsearch#8
Original commit: elastic/x-pack-elasticsearch@3d201ac2bf
This change removes the roles from the user in the remote cluster to show that the roles passed
from the source cluster are used in evaluating the user's permissions during multi-cluster search.
Original commit: elastic/x-pack-elasticsearch@35658168fb
We didn't realise it was possible for a qa module to depend on the
test classes of the plugin module, so we duplicated a test class.
But it turns out it IS possible to declare this dependency and avoid
the duplication.
Original commit: elastic/x-pack-elasticsearch@b6a21cda28
This commit removes the SecuredString class that was previously used throughout the security code
and replaces it with the SecureString class from core that was added as part of the new secure
settings infrastructure.
relates elastic/x-pack-elasticsearch#421
Original commit: elastic/x-pack-elasticsearch@e9cd117ca1
let close job and stop datafeed apis redirect to elected master node.
This is for cluster state observation purposes, so that a subsequent open and then close job or
start and then stop datafeed see the same local cluster state and sanity validation doesn't fail.
Original commit: elastic/x-pack-elasticsearch@21a63184b9
Changes the logging of LDAP authentication failures from "always" to "only if the user failed to be authenticated"
Previously there were cases (such has having 2 AD realms) where successful user authentication would still cause an INFO message to be written to the log for every request.
Now that message is suppressed, but a WARN message is added _if-and-only-if_ the user cannot be authenticated by any realm.
This is implemented via a new value stored in the ThreadContext that the AuthenticationService choses to log (or not log) depending on the result of the authenticate process.
Closes: elastic/x-pack-elasticsearch#887
Original commit: elastic/x-pack-elasticsearch@b81b363729
This commit removes the smoke-test-monitoring-with-security project
and replaces it with a REST test.
Original commit: elastic/x-pack-elasticsearch@f1665815c2
This is analagous of the bwc-zip for elasticsearch. The one caveat is
due to the structure of how ES+xpack must be checked out, we end up with
a third clone of elasticsearch (the second being in :distribution:bwc-zip).
But the rolling upgrade integ test passes with this change.
relates elastic/x-pack-elasticsearch#870
Original commit: elastic/x-pack-elasticsearch@34bdce6e99
This commit is response to the renaming of the random ASCII helper
methods in ESTestCase. The name of this method was changed because these
methods only produce random strings generated from [a-zA-Z], not from
all ASCII characters.
Relates elastic/x-pack-elasticsearch#942
Original commit: elastic/x-pack-elasticsearch@a6085964d3
* [ML] Set job create time on server
* Job.Builder serialisation tests
* Make setCreateTime package private
Original commit: elastic/x-pack-elasticsearch@d2d75e0d7b
This commit adds separate tasks for tribe clusteres which the
cluster formation tasks build their own tasks off. This ensures each
cluster will have its own wait task, so that the tribe node will be able
to wait on the other clusters being up before even trying to start.
relates elastic/x-pack-elasticsearch#877
Original commit: elastic/x-pack-elasticsearch@1e4c729372
Previously force closing a job required extra privileges. Following
the full discussion about what privileges should be required.
Original commit: elastic/x-pack-elasticsearch@4d85314b35
* Removed OPENING and CLOSING job states. Instead when persistent task has been created and
status hasn't been set then this means we haven't yet started, when the executor changes it to STARTED we have.
The coordinating node will monitor cs for a period of time until that happens and then returns or times out.
* Refactored job close api to go to node running job task and close job there.
* Changed unexpected job and datafeed exception messages to not mention the state and instead mention that job/datafeed haven't yet started/stopped.
Original commit: elastic/x-pack-elasticsearch@37e778b585
This commit fixes the merging of field level security rules from multiple roles. Prior to 5.2, the
merging was treated as the merging of languages whereas after 5.2, this incorrectly became a merge
of all rules meaning a single wildcard could cause denials to be ignored.
Original commit: elastic/x-pack-elasticsearch@42f9e6d8b0
The wait condition used for integ tests by default calls the cluster
health api with wait_for_nodes nd wait_for_status. However, xpack
overrides the wait condition to add auth, but most of these conditions
still looked at the root ES url, which means the tests are susceptible
to race conditions with the check and node startup. This change modifies
the url for the authenticated wait condtion to check the health api,
with the appropriate wait_for_nodes and wait_for_status.
Original commit: elastic/x-pack-elasticsearch@0b23ef528f
* Changed ML action names to allow distinguishing of admin and read-only actions
using wildcards
* Added manage_ml and monitor_ml built-in privileges as subsets of the existing
manage and monitor privileges
* Added out-of-the-box machine_learning_admin and machine_learning_user roles
* Changed machine learning results endpoints to use a NodeClient rather than an
InternalClient when searching for results so that index/document level permissions
applied to ML results are respected
Original commit: elastic/x-pack-elasticsearch@eee800aaa8
This commit adds an end-point to force deletion of expired data:
DELETE /_xpack/ml/_delete_expired_data
A few other things are changed too:
- Delete expired results from now rather than start of day
- Rename MlDaily{Management -> Maintenance}Service
- Refresh job indices when job is closing to ensure latest result
visibility
- Commit results when quantiles are persisted to ensure they are visible
for renormalization
Original commit: elastic/x-pack-elasticsearch@8ca5272a94
The yaml test runner now throws error when skip or do sections are malformed, such as they don't start with the proper token (START_OBJECT). That signals bad indentation, which was previously ignored. Thanks (or due to) our pull parsing code, we were still able to properly parse the sections, yet other runners weren't able to.
Original commit: elastic/x-pack-elasticsearch@920201207c
This commit adds the ability for x-pack extensions to optionally
provide custom roles providers, which are used to resolve any roles
into role descriptors that are not found in the reserved or native
realms. This feature enables the ability to define and provide roles
from other sources, without having to pre-define such roles in the security
config files.
relates elastic/x-pack-elasticsearch#77
Original commit: elastic/x-pack-elasticsearch@bbbe7a49bf
* Tribe node security tests with external clusters
This PR adds a qa module for security tests with tribe node
using external clusters. Existing SecurityTribeIT tests
have been ported to use external clusters with tribe setup
as a first step.
Currently the ports to the external clusters are passed to the
integration tests through system properties and external clusters
are built on test setup (the code for building external clusters is
copied from ESIntegTestCase). This is a WIP as we need a
more generic way to facilitate testing tribe setup with external
clusters. thoughts welcome.
* incorporate feedback
* update to master
Original commit: elastic/x-pack-elasticsearch@686887ca91
Currently, both the NativeUsersStore and NativeRolesStore can undergo
multiple state transitions. This is done primarily to check if the
security index is usable before it proceeds. However, such checks are
only needed for the tests, because if the security index is unavailable
when it is needed, the downstream actions invoked by the
NativeUsersStore and NativeRolesStore will throw the appropriate
exceptions notifying of that condition. In addition, both the
NativeUsersStore and NativeRolesStore had much duplicate code that
listened for cluster state changes and made the exact same state
transitions.
This commit removes the complicated state transitions in both classes
and enables both classes to use the SecurityTemplateService to monitor
all of the security index lifecycle changes they need to be aware of.
This commit also moves the logic for determining if the security index
needs template and/or mapping updates to the SecurityLifecycleService,
and makes the NativeRealmMigrator solely responsible for applying the
updates.
Original commit: elastic/x-pack-elasticsearch@b31d144597
This commit removes the usages and definition of deprecated methods that do not accept the XContentType for the source.
Additionally, usages of *Entity classes from the http client library have been changed to always specify the content
type.
Original commit: elastic/x-pack-elasticsearch@29d336a008
Also added a test to multi node qa module that tests the datafeeder, which should have caught this.
Original commit: elastic/x-pack-elasticsearch@89e4875f6c
Was accidentally injecting the script object, not the string version of the code. Also
added a check so we only inject for Painless scripts (and not groovy, etc).
Minor style tweaks too.
Original commit: elastic/x-pack-elasticsearch@58c7275bd8
This change adds licensing to the maching learning feature, and only allows access to machine learning if a trial or platinum license is installed.
Further, this change also renames `MlPlugin` to `MachineLearning` in line with the other feature plugin names and move the enabled setting to `XPackSettings`
Original commit: elastic/x-pack-elasticsearch@48ea9d781b
Adds a new `xpack.security.authc.accept_default_password` setting that defaults to `true`. If it is set to false, then the default password is not accepted in the reserved realm.
Adds a bootstrap check that the above setting must be set to `false` if security is enabled.
Adds docs for the new setting and bootstrap.
Changed `/_enable` and `/_disable`, to store a blank password if the user record did not previously exist, which is interpreted to mean "treat this user as having the default password". The previous functionality would explicitly set the user's password to `changeme`, which would then prevent the new configuration setting from doing its job.
For any existing reserved users that had their password set to `changeme`, migrates them to the blank password (per above paragraph)
Closes: elastic/elasticsearch#4333
Original commit: elastic/x-pack-elasticsearch@db64564093
This commit brings back support an auto-generated certificate and private key for
transport traffic. The auto-generated certificate and key can only be used in development
mode; when moving to production a key and certificate must be provided.
For the edge case of a user not wanting to encrypt their traffic, the user can set
the cipher_suites setting to `TLS_RSA_WITH_NULL_SHA256` or a like cipher, but a key/cert
is still required.
Closeselastic/elasticsearch#4332
Original commit: elastic/x-pack-elasticsearch@b7a1e629f5
This change adapts x-pack to pass on the parsed XContentType from rest requests to transport
requests and use this value in place of attempting to auto-detect the content type.
Original commit: elastic/x-pack-elasticsearch@57475fd403
If `domainSplit(` is detected in an inline script, the function and params are injected into
the script.
The majority of this PR is actually test-related. Adds a unit test to check for the injected
script/params. Also adds another QA test which -- through a very round-about mechanism --
confirms that the injected script compiles and functions correctly. The QA test can
be simplified greatly once the Preview API is added.
Original commit: elastic/x-pack-elasticsearch@c7c35a982c
This contains the Painless-based DomainSplit function, generated static maps and basic tests. Due to cross-module complications, the tests are run by executing searches with script_fields and checking the response
Original commit: elastic/x-pack-elasticsearch@c6c2942e01
This matches the way tests that need to run without an Elasticsearch
bootstrap are run in core Elasticsearch. This should make merging to
x-pack easier.
Note that the no bootstrap tests now run after the integration tests, but
this doesn't really matter.
Original commit: elastic/x-pack-elasticsearch@5547f457b6
* Gets build to use elasticsearch-extras
Also adds ci script for building repo on CI servers
To use this change you need to:
1. Clone elasticsearch: `git@github.com:elastic/elasticsearch.git`
2. create a directory at the same level as elasticsearch called `elasticsearch-extra`
3. Clone this repository into the `elasticsearch-extra` directory
4. Run `gradle build` from the `elasticsearch-extra/prelert-legacy` directory or run `gradle :prelert-legacy:build` from the `elasticsearch directory
* Adds USE_SSH option to ci script
* iter
Original commit: elastic/x-pack-elasticsearch@ea127dfef0
The job open api starts a task and ties that AutodetectCommunicator.
The job close api is a sugar api, that uses the list and cancel task api to close a AutodetectCommunicator instance.
The flush job and post data api redirect to the node holding the job task and then delegate the flush or data to the AutodetectCommunicator instance.
Also:
* Added basic multi node cluster test.
* Fixed cluster state diffs bugs, forgot to mark ml metadata diffs as named writeable.
* Moved waiting for open job logic into OpenJobAction.TransportAction and moved the logic that was original there to a new action named InternalOpenJobAction.
Original commit: elastic/x-pack-elasticsearch@194a058dd2
Luca Cavanna
Jay Modi
David Roberts
David Kyle
Ryan Ernst
Martijn van Groningen
Simon Willnauer
Ali Beyad
Dimitris Athanasiou
Hendrik Muhs
Tim Vernum
Tanguy Leroux
Jason Tedor
Yannick Welsch
Daniel Mitterdorfer
Colin Goodheart-Smithe
Nik Everett
Areek Zillur
javanna
Zachary Tong
Zachary Tong
Jack Conradson