Commit Graph

323 Commits

Author SHA1 Message Date
Hendrik Muhs 7ef9a16f45 [ML] implement '_all' for stopping datafeeds (elastic/x-pack-elasticsearch#995)
Add a '_all' functionality for stopping ML datafeeds.

For cluster shutdown due to maintenance and major upgrades we recommend the user to stop all datafeeds and jobs. This change add the ability to stop all datafeeds at once where previously it was required to iterate over all feeds and do a explicit stop. This is part two of elastic/x-pack-elasticsearch#795, part one can be found in elastic/x-pack-elasticsearch#962 .

relates elastic/x-pack-elasticsearch#795

Original commit: elastic/x-pack-elasticsearch@ed1eff83d5
2017-04-11 13:39:22 +02:00
Simon Willnauer 617c3ead5c Add tests for _remote/info API (elastic/x-pack-elasticsearch#1009)
Relates to elastic/elasticsearch#23925
Depends on elastic/elasticsearch#23969

Original commit: elastic/x-pack-elasticsearch@d1e8754a57
2017-04-11 11:24:22 +02:00
Tim Vernum 8840042751 Only log LDAP errors if authentication failed overall (elastic/x-pack-elasticsearch#960)
Changes the logging of LDAP authentication failures from "always" to "only if the user failed to be authenticated"

Previously there were cases (such has having 2 AD realms) where successful user authentication would still cause an INFO message to be written to the log for every request. 
Now that message is suppressed, but a WARN message is added _if-and-only-if_ the user cannot be authenticated by any realm.

This is implemented via a new value stored in the ThreadContext that the AuthenticationService choses to log (or not log) depending on the result of the authenticate process.
  
Closes: elastic/x-pack-elasticsearch#887

Original commit: elastic/x-pack-elasticsearch@b81b363729
2017-04-11 09:37:21 +10:00
Tanguy Leroux 6396edc6a7 [Test] Add Monitoring Bulk API REST tests with privileges (elastic/x-pack-elasticsearch#968)
This commit removes the smoke-test-monitoring-with-security project
and replaces it with a REST test.

Original commit: elastic/x-pack-elasticsearch@f1665815c2
2017-04-10 15:08:19 +02:00
Ryan Ernst e8399cfccf Build: Build 5.x version of xpack for bwc tests to use (elastic/x-pack-elasticsearch#885)
This is analagous of the bwc-zip for elasticsearch. The one caveat is
due to the structure of how ES+xpack must be checked out, we end up with
a third clone of elasticsearch (the second being in :distribution:bwc-zip).
But the rolling upgrade integ test passes with this change.

relates elastic/x-pack-elasticsearch#870

Original commit: elastic/x-pack-elasticsearch@34bdce6e99
2017-04-07 22:19:30 -07:00
Hendrik Muhs c7fd1aacff [ML] implement _all for closing jobs (elastic/x-pack-elasticsearch#962)
Add a '_all' functionality for closing ML jobs.

For cluster shutdown due to maintenance and major upgrades we recommend the user to stop all datafeeds and jobs. This change add the ability to close all jobs at once where previously it was required to iterate over all jobs and do a explicit close. This is part one of elastic/x-pack-elasticsearch#795, part two can be found in elastic/x-pack-elasticsearch#995.

relates elastic/x-pack-elasticsearch#795

Original commit: elastic/x-pack-elasticsearch@9b251ed7e1
2017-04-07 14:51:13 +02:00
Martijn van Groningen 2b5c0faebd [TEST] wait for cs version to be the same on all nodes before stopping datafeed
Original commit: elastic/x-pack-elasticsearch@6fd6224047
2017-04-05 11:37:36 +02:00
Martijn van Groningen 4a57c92e6f [TEST] unmuted test
Original commit: elastic/x-pack-elasticsearch@5dd80c32c7
2017-04-05 09:46:02 +02:00
Jason Tedor dee1bd1825 Respond to rename random ASCII helper methods
This commit is response to the renaming of the random ASCII helper
methods in ESTestCase. The name of this method was changed because these
methods only produce random strings generated from [a-zA-Z], not from
all ASCII characters.

Relates elastic/x-pack-elasticsearch#942

Original commit: elastic/x-pack-elasticsearch@a6085964d3
2017-04-04 11:04:37 -04:00
Yannick Welsch e52fbdf63c Use random http port for watcher REST tests (elastic/x-pack-elasticsearch#936)
Adapts the Watcher REST tests so that they don't require a fixed http port anymore.

Original commit: elastic/x-pack-elasticsearch@14919b16e7
2017-04-04 09:02:23 +02:00
David Kyle 5b66c7a7ba [ML] Set job create time on server (elastic/x-pack-elasticsearch#910)
* [ML] Set job create time on server

* Job.Builder serialisation tests

* Make setCreateTime package private

Original commit: elastic/x-pack-elasticsearch@d2d75e0d7b
2017-04-03 18:30:47 +01:00
Simon Willnauer 5e6bfb9a82 Fix cross cluster search with security (elastic/x-pack-elasticsearch#904)
This commit adds an integration test that runs basic cross cluster search
actions across 2 clusters with security installed. This commit also fixes
several issues with respect to internal actions and proxy actions in the context
of cross cluster search.

Relates to elastic/elasticsearch#23830
relates elastic/x-pack-elasticsearch#892

Original commit: elastic/x-pack-elasticsearch@2e5486c259
2017-04-03 09:53:48 +02:00
Ryan Ernst e1949ee362 Test: Use separate wrapper tasks for tribe clusters setup (elastic/x-pack-elasticsearch#905)
This commit adds separate tasks for tribe clusteres which the
cluster formation tasks build their own tasks off. This ensures each
cluster will have its own wait task, so that the tribe node will be able
to wait on the other clusters being up before even trying to start.

relates elastic/x-pack-elasticsearch#877

Original commit: elastic/x-pack-elasticsearch@1e4c729372
2017-03-31 14:30:44 -07:00
David Roberts 95fd6777a0 [ML] Add 'xpack' into action names to match what the rest of X-Pack does (elastic/x-pack-elasticsearch#909)
Original commit: elastic/x-pack-elasticsearch@36be904ad9
2017-03-31 11:22:53 +01:00
Jason Tedor f52760d2ac Revert "Silence UpgradeClusterClientYamlTestSuiteIT"
This reverts commit elastic/x-pack-elasticsearch@e29f94b24c.

Original commit: elastic/x-pack-elasticsearch@d7d27b20ef
2017-03-30 08:39:10 -04:00
David Roberts 72f5e92972 [ML] Adjust error to make sense for FAILED as well as CLOSED jobs (elastic/x-pack-elasticsearch#888)
Previously trying to perform an operation on a FAILED job would tell
you that it hadn't been opened.  This was very confusing if you knew
very well you'd just opened the job.

Relates elastic/x-pack-elasticsearch#879

Original commit: elastic/x-pack-elasticsearch@2dbe41caaa
2017-03-30 10:33:56 +01:00
David Roberts c335e79508 [TEST] Enable two blacklisted tests that now work
Previously force closing a job required extra privileges.  Following
the full discussion about what privileges should be required.

Original commit: elastic/x-pack-elasticsearch@4d85314b35
2017-03-30 10:26:15 +01:00
Martijn van Groningen 61b4f6480e [TEST] muted test
Original commit: elastic/x-pack-elasticsearch@550b55739e
2017-03-30 11:24:27 +02:00
Martijn van Groningen f2654b5872 [ML] Changed job and datafeed lifecycle management
* Removed OPENING and CLOSING job states. Instead when persistent task has been created and
   status hasn't been set then this means we haven't yet started, when the executor changes it to STARTED we have.
   The coordinating node will monitor cs for a period of time until that happens and then returns or times out.
* Refactored job close api to go to node running job task and close job there.
* Changed unexpected job and datafeed exception messages to not mention the state and instead mention that job/datafeed haven't yet started/stopped.

Original commit: elastic/x-pack-elasticsearch@37e778b585
2017-03-29 20:35:10 +02:00
Daniel Mitterdorfer 2355338625 Silence UpgradeClusterClientYamlTestSuiteIT
Relates elastic/x-pack-elasticsearch#870
Relates elastic/x-pack-kibana#690

Original commit: elastic/x-pack-elasticsearch@e29f94b24c
2017-03-29 10:38:28 +02:00
Dimitrios Athanasiou 991e5d8833 [TEST] Set required euery on delete-by-query request
Original commit: elastic/x-pack-elasticsearch@b9b6234f3d
2017-03-28 17:15:23 +01:00
Colin Goodheart-Smithe 99db6013ea [ML] Adds a test for ml transport client (elastic/x-pack-elasticsearch#851)
This change also adds synchronous methods to `MachineLearningClient`.

relates elastic/x-pack-elasticsearch#567

Original commit: elastic/x-pack-elasticsearch@b3a4b38a51
2017-03-28 09:05:55 +01:00
Martijn van Groningen fc29fd159a [TEST] Don't lose the actual cause of the exception.
Original commit: elastic/x-pack-elasticsearch@8720288ef5
2017-03-27 15:33:04 +02:00
David Roberts ecbfaace38 [TEST] Improve ML security test comments and account for new close assertion
Original commit: elastic/x-pack-elasticsearch@004845bf2f
2017-03-27 10:24:05 +01:00
David Roberts c0445fac4d [TEST] Silence failing ML security tests
Original commit: elastic/x-pack-elasticsearch@af4ed2019d
2017-03-24 17:58:30 +00:00
Colin Goodheart-Smithe caa9afad2b [TEST] fixed content type in ML QA tests
Original commit: elastic/x-pack-elasticsearch@b7749e1319
2017-03-24 14:08:39 +00:00
Hendrik Muhs a8b1b3e863 Revert "disable integration test, see elastic/x-pack-elasticsearch#823 for details (elastic/x-pack-elasticsearch#824)"
This reverts commit elastic/x-pack-elasticsearch@2338b3c972.

Original commit: elastic/x-pack-elasticsearch@a9c7168762
2017-03-23 20:10:58 +01:00
Hendrik Muhs 27dce8669c disable integration test, see elastic/x-pack-elasticsearch#823 for details (elastic/x-pack-elasticsearch#824)
[ML] disable failing integration test, see elastic/x-pack-elasticsearch#823 for details

Original commit: elastic/x-pack-elasticsearch@2338b3c972
2017-03-23 19:54:02 +01:00
Hendrik Muhs 6f7f466fa3 [ML] move DataStreamDiagnostics into DataCountsReporter (elastic/x-pack-elasticsearch#775)
repair DataStreamDiagnostics

Moves DataStreamDiagnostics into DataCountsReporter to survive if job is opened/closed/fed in chunks.

relates elastic/x-pack-elasticsearch#764

Original commit: elastic/x-pack-elasticsearch@29c221a451
2017-03-23 16:43:51 +01:00
Jay Modi 8ba6e8b3eb Fix merging of field level security rules (elastic/x-pack-elasticsearch#796)
This commit fixes the merging of field level security rules from multiple roles. Prior to 5.2, the
merging was treated as the merging of languages whereas after 5.2, this incorrectly became a merge
of all rules meaning a single wildcard could cause denials to be ignored.

Original commit: elastic/x-pack-elasticsearch@42f9e6d8b0
2017-03-22 05:10:30 -07:00
Martijn van Groningen 60c79ccdbb [TEST] Set job close timeout to 20s.
Original commit: elastic/x-pack-elasticsearch@9b92c940cb
2017-03-16 21:57:36 +01:00
Martijn van Groningen 73c5c1576f [TEST] Added logging and removed unneeded jackson-databind test dependency.
Original commit: elastic/x-pack-elasticsearch@58c247d696
2017-03-16 21:17:14 +01:00
Dimitris Athanasiou 16efd4e474 [ML] Use TimeValue in job and datafeed configs (elastic/x-pack-elasticsearch#732)
relates elastic/x-pack-elasticsearch#679

Original commit: elastic/x-pack-elasticsearch@891edd5bfe
2017-03-16 14:07:47 +00:00
Ryan Ernst baaad36c5e Fix tribe test setup to look for a single node
numNodes isn't available because it is not in the normal context of
cluster configuration.

relates elastic/x-pack-elasticsearch#740

Original commit: elastic/x-pack-elasticsearch@336147f20c
2017-03-15 15:55:10 -07:00
Ryan Ernst 534584d525 Tests: Use cluster health api for wait condition in ml integ test
This was forgotten in elastic/x-pack-elasticsearch#740

Original commit: elastic/x-pack-elasticsearch@47c10dc543
2017-03-15 10:27:38 -07:00
Ryan Ernst 8c01d6ea69 Tests: Add cluster health check to xpack integ wait conditions (elastic/x-pack-elasticsearch#740)
The wait condition used for integ tests by default calls the cluster
health api with wait_for_nodes nd wait_for_status. However, xpack
overrides the wait condition to add auth, but most of these conditions
still looked at the root ES url, which means the tests are susceptible
to race conditions with the check and node startup. This change modifies
the url for the authenticated wait condtion to check the health api,
with the appropriate wait_for_nodes and wait_for_status.

Original commit: elastic/x-pack-elasticsearch@0b23ef528f
2017-03-15 10:23:26 -07:00
David Roberts 0b7c735aec [ML] Add machine learning privileges/roles (elastic/x-pack-elasticsearch#673)
* Changed ML action names to allow distinguishing of admin and read-only actions
  using wildcards
* Added manage_ml and monitor_ml built-in privileges as subsets of the existing
  manage and monitor privileges
* Added out-of-the-box machine_learning_admin and machine_learning_user roles
* Changed machine learning results endpoints to use a NodeClient rather than an
  InternalClient when searching for results so that index/document level permissions
  applied to ML results are respected

Original commit: elastic/x-pack-elasticsearch@eee800aaa8
2017-03-14 16:13:41 +00:00
Dimitris Athanasiou a7a36245c9 [ML] Add end-point for deleting expired results (elastic/x-pack-elasticsearch#670)
This commit adds an end-point to force deletion of expired data:

DELETE /_xpack/ml/_delete_expired_data

A few other things are changed too:

- Delete expired results from now rather than start of day
- Rename MlDaily{Management -> Maintenance}Service
- Refresh job indices when job is closing to ensure latest result
  visibility
- Commit results when quantiles are persisted to ensure they are visible
  for renormalization


Original commit: elastic/x-pack-elasticsearch@8ca5272a94
2017-03-03 11:33:22 +00:00
Dimitrios Athanasiou 0542a9eb92 [TEST] Disable ml in qa modules where necessary
Original commit: elastic/x-pack-elasticsearch@bb311b44d7
2017-03-02 17:01:35 +00:00
Martijn van Groningen 948a8594bb [TEST] Disable ml in audit qa test
Original commit: elastic/x-pack-elasticsearch@34b7243f69
2017-03-02 16:02:15 +01:00
Luca Cavanna 70a4a69e39 Fix indentation in some yaml tests (elastic/x-pack-elasticsearch#686)
The yaml test runner now throws error when skip or do sections are malformed, such as they don't start with the proper token (START_OBJECT). That signals bad indentation, which was previously ignored. Thanks (or due to) our pull parsing code, we were still able to properly parse the sections, yet other runners weren't able to.

Original commit: elastic/x-pack-elasticsearch@920201207c
2017-03-02 12:43:43 +01:00
Nik Everett dcec4bbf4b Handle Mustache not using Settings any more
It didn't need them so now it doesn't take them.

Original commit: elastic/x-pack-elasticsearch@f047c3926b
2017-03-01 14:59:14 -05:00
Ali Beyad b20578b9f6 Enables X-Pack extensions to implement custom roles providers (elastic/x-pack-elasticsearch#603)
This commit adds the ability for x-pack extensions to optionally
provide custom roles providers, which are used to resolve any roles
into role descriptors that are not found in the reserved or native
realms.  This feature enables the ability to define and provide roles
from other sources, without having to pre-define such roles in the security
config files.

relates elastic/x-pack-elasticsearch#77

Original commit: elastic/x-pack-elasticsearch@bbbe7a49bf
2017-03-01 12:20:34 -05:00
Areek Zillur ec4de10ee2 Tribe node security tests with external clusters (elastic/x-pack-elasticsearch#606)
* Tribe node security tests with external clusters

This PR adds a qa module for security tests with tribe node
using external clusters. Existing SecurityTribeIT tests
have been ported to use external clusters with tribe setup
as a first step.

Currently the ports to the external clusters are passed to the
integration tests through system properties and external clusters
are built on test setup (the code for building external clusters is
copied from ESIntegTestCase). This is a WIP as we need a
more generic way to facilitate testing tribe setup with external
clusters. thoughts welcome.

* incorporate feedback

* update to master

Original commit: elastic/x-pack-elasticsearch@686887ca91
2017-02-28 18:36:53 -05:00
Martijn van Groningen 6783f823a8 [ML] Change stop datafeed api delegate to node hosting datafeed task and execute cancel locally,
instead of only removing the persistent task from cluster state.

Original commit: elastic/x-pack-elasticsearch@3974b20827
2017-02-28 16:36:13 +01:00
javanna 33ccc3bd6c adapt to ObjectPath changes to support binary formats
Original commit: elastic/x-pack-elasticsearch@11782418e9
2017-02-27 12:28:04 +01:00
David Roberts e2f5715191 [TEST] Fix one last test that was missed in elastic/x-pack-elasticsearch#649
Original commit: elastic/x-pack-elasticsearch@1039197afd
2017-02-27 09:55:49 +00:00
Ali Beyad 4a001706e7 Simplifies security index state changes and template/mapping updates (elastic/x-pack-elasticsearch#551)
Currently, both the NativeUsersStore and NativeRolesStore can undergo
multiple state transitions. This is done primarily to check if the
security index is usable before it proceeds. However, such checks are
only needed for the tests, because if the security index is unavailable
when it is needed, the downstream actions invoked by the
NativeUsersStore and NativeRolesStore will throw the appropriate
exceptions notifying of that condition. In addition, both the
NativeUsersStore and NativeRolesStore had much duplicate code that
listened for cluster state changes and made the exact same state
transitions.

This commit removes the complicated state transitions in both classes
and enables both classes to use the SecurityTemplateService to monitor
all of the security index lifecycle changes they need to be aware of.

This commit also moves the logic for determining if the security index
needs template and/or mapping updates to the SecurityLifecycleService,
and makes the NativeRealmMigrator solely responsible for applying the
updates.

Original commit: elastic/x-pack-elasticsearch@b31d144597
2017-02-23 13:05:39 -05:00
Ryan Ernst 8527bc2415 Build: Convert integ test dsl to new split cluster/runner dsl
This is the xpack side of elastic/elasticsearch#23304

Original commit: elastic/x-pack-elasticsearch@8eddd7fb0d
2017-02-22 00:56:52 -08:00
Martijn van Groningen 8a6cea0350 [ml] For the data, flush and update apis check the job state on the node running the job task instead of the coordinating node.
[test] Also re-enabled multi node qa tests.

Original commit: elastic/x-pack-elasticsearch@efb24131f0
2017-02-18 19:18:29 +01:00
Jay Modi 68324f5a50 Remove usage of deprecated methods that do not accept a XContentType (elastic/x-pack-elasticsearch#549)
This commit removes the usages and definition of deprecated methods that do not accept the XContentType for the source.

Additionally, usages of *Entity classes from the http client library have been changed to always specify the content
type.

Original commit: elastic/x-pack-elasticsearch@29d336a008
2017-02-17 14:45:00 -05:00
Martijn van Groningen 92a4b97f6d [TEST] Add simple test, so that module has at least one working test.
Original commit: elastic/x-pack-elasticsearch@72441418ac
2017-02-17 19:44:04 +01:00
David Kyle d707a1c977 Mark class with awaitsfix
Is the test failing because all the methods are marked awaitsfix?

Original commit: elastic/x-pack-elasticsearch@de93bb83b7
2017-02-17 18:06:42 +00:00
David Kyle b9a4a2c621 Revert "Don’t wait on job close (elastic/x-pack-elasticsearch#574)"
This reverts commit elastic/x-pack-elasticsearch@99ed0e0dba and fixes the failing tests

Original commit: elastic/x-pack-elasticsearch@403e38316d
2017-02-17 17:05:41 +00:00
David Kyle 1a4b87454c Add AwaitsFix for elastic/x-pack-elasticsearch#592
Original commit: elastic/x-pack-elasticsearch@c5d2650acf
2017-02-17 14:05:25 +00:00
Martijn van Groningen 6739f84efb [test] change id between tests, to make reading log files easier.
Original commit: elastic/x-pack-elasticsearch@e633d7098e
2017-02-17 14:55:14 +01:00
Martijn van Groningen 6d79210f79 [ml] Registered named writable entry for DatafeedState.
Also added a test to multi node qa module that tests the datafeeder, which should have caught this.

Original commit: elastic/x-pack-elasticsearch@89e4875f6c
2017-02-15 21:38:19 +01:00
Ali Beyad b3d72af644 [TEST] bumps rolling upgrade bwc version to 5.4.0-snapshot
Original commit: elastic/x-pack-elasticsearch@2f321682d2
2017-02-15 11:00:11 -05:00
Zachary Tong 65cd119733 [ML] Fix injection of DomainSplit, and only inject for Painless
Was accidentally injecting the script object, not the string version of the code.  Also
added a check so we only inject for Painless scripts (and not groovy, etc).

Minor style tweaks too.

Original commit: elastic/x-pack-elasticsearch@58c7275bd8
2017-02-15 09:31:16 -05:00
Colin Goodheart-Smithe 6183015639 Adds licensing to machine learning feature (elastic/x-pack-elasticsearch#559)
This change adds licensing to the maching learning feature, and only allows access to machine learning if a trial or platinum license is installed.

Further, this change also renames `MlPlugin` to `MachineLearning` in line with the other feature plugin names and move the enabled setting to `XPackSettings`

Original commit: elastic/x-pack-elasticsearch@48ea9d781b
2017-02-14 15:47:37 +00:00
Ryan Ernst 2571921605 Rename x-pack project names to new names with split repo
Original commit: elastic/x-pack-elasticsearch@5a908f5dcc
2017-02-10 11:02:42 -08:00
Colin Goodheart-Smithe 1da752152a Migrates machine learning into X-Pack
Original commit: elastic/x-pack-elasticsearch@9ad22980b8
2017-02-08 16:58:56 +00:00
Colin Goodheart-Smithe e761b76765 Migrates QA files
Original commit: elastic/x-pack-elasticsearch@ac651f51d4
2017-02-08 16:58:55 +00:00
Simon Willnauer 0e779b41de Followup from elastic/elasticsearchelastic/elasticsearch#23042 (elastic/elasticsearch#4895)
This change accomodates for the renamings done in elastic/elasticsearchelastic/elasticsearch#23042

Original commit: elastic/x-pack-elasticsearch@c290c8ecc4
2017-02-08 14:40:17 +01:00
Tim Vernum 734a4ee66d Prevent default passwords in production mode (elastic/elasticsearch#4724)
Adds a new `xpack.security.authc.accept_default_password` setting that defaults to `true`. If it is set to false, then the default password is not accepted in the reserved realm.

Adds a bootstrap check that the above setting must be set to `false` if security is enabled.  

Adds docs for the new setting and bootstrap.

Changed `/_enable` and `/_disable`, to store a blank password if the user record did not previously exist, which is interpreted to mean "treat this user as having the default password". The previous functionality would explicitly set the user's password to `changeme`, which would then prevent the new configuration setting from doing its job.

For any existing reserved users that had their password set to `changeme`, migrates them to the blank password (per above paragraph)

Closes: elastic/elasticsearch#4333

Original commit: elastic/x-pack-elasticsearch@db64564093
2017-02-08 16:19:55 +11:00
Jay Modi bd04b30acd security: transport always uses TLS (elastic/elasticsearch#4738)
This commit brings back support an auto-generated certificate and private key for
transport traffic. The auto-generated certificate and key can only be used in development
mode; when moving to production a key and certificate must be provided.

For the edge case of a user not wanting to encrypt their traffic, the user can set
the cipher_suites setting to `TLS_RSA_WITH_NULL_SHA256` or a like cipher, but a key/cert
is still required.

Closes elastic/elasticsearch#4332

Original commit: elastic/x-pack-elasticsearch@b7a1e629f5
2017-02-07 11:39:31 -05:00
David Kyle 5a1cd69a6a More checkstyle fixes
Original commit: elastic/x-pack-elasticsearch@4c454c5061
2017-02-03 16:58:19 +00:00
Jason Tedor 02bcd9510f Enable disabled rolling upgrade tests
These tests were disabled due to a bug introduced when
sequence-number-based recovery was introduced. This commit enables these
tests again.

Relates elastic/elasticsearch#4801

Original commit: elastic/x-pack-elasticsearch@e555bfec1a
2017-02-03 08:12:52 -05:00
Jay Modi 1f32ef21a2 Update to pass content type from the rest APIs to transport requests (elastic/elasticsearch#4689)
This change adapts x-pack to pass on the parsed XContentType from rest requests to transport
requests and use this value in place of attempting to auto-detect the content type.

Original commit: elastic/x-pack-elasticsearch@57475fd403
2017-02-02 14:08:34 -05:00
polyfractal 3504608a1e [TEST] more robust regex for tests missing subdomains
Original commit: elastic/x-pack-elasticsearch@28e5d14c22
2017-02-01 11:04:55 -05:00
Zachary Tong a11ddd1e04 Integrate domainSplit function into datafeeds (elastic/elasticsearch#841)
If `domainSplit(` is detected in an inline script, the function and params are injected into
the script.

The majority of this PR is actually test-related.  Adds a unit test to check for the injected
script/params.  Also adds another QA test which -- through a very round-about mechanism --
confirms that the injected script compiles and functions correctly.  The QA test can
be simplified greatly once the Preview API is added.

Original commit: elastic/x-pack-elasticsearch@c7c35a982c
2017-02-01 10:20:00 -05:00
Ali Beyad 8cab4fec4b [TEST] adds AwaitsFix to failing test
Original commit: elastic/x-pack-elasticsearch@9e8f6ce047
2017-01-31 22:50:17 -05:00
Martijn van Groningen ce6dc4a506 Make job stats api task aware.
This will allow the job stats api to redirect the request to node where job is running.

Original commit: elastic/x-pack-elasticsearch@9f1d12dfcb
2017-01-31 22:35:54 +01:00
Jack Conradson 7f0ecc4b30 Change Namespace for Stored Script to Only Use Id (elastic/elasticsearch#4387)
Changes the behavior in x-pack necessary to support the elasticsearch change elastic/elasticsearch#22206.

Original commit: elastic/x-pack-elasticsearch@916e72e263
2017-01-31 13:28:03 -08:00
Colin Goodheart-Smithe f804bb1917 Removes ensureGreen from PlainlessDomainSplitIT (elastic/elasticsearch#839)
This shouldn’t be needed as the cluster no longer goes red when an index is created.

Original commit: elastic/x-pack-elasticsearch@b554ea9caf
2017-01-31 16:09:25 +00:00
Nik Everett cdfa20b83f Handle moving some classes from reindex to core
Original commit: elastic/x-pack-elasticsearch@cdf98a70f7
2017-01-27 16:08:32 -05:00
Zachary Tong 9395ef81b1 Painless DomainSplit tests in new Single-Node QA Module (elastic/elasticsearch#787)
This contains the Painless-based DomainSplit function, generated static maps and basic tests.  Due to cross-module complications, the tests are run by executing searches with script_fields and checking the response


Original commit: elastic/x-pack-elasticsearch@c6c2942e01
2017-01-27 08:52:48 -05:00
Colin Goodheart-Smithe 603fa47580 Adds an option to disable the ML plugin (elastic/elasticsearch#785)
Adds an `xpack.ml.enabled` node level setting that can be used to enable and disable the plugin. This will be important when we migrate to X-Pack

Closes elastic/elasticsearch#781

Original commit: elastic/x-pack-elasticsearch@e5c4969a96
2017-01-24 16:14:56 +00:00
Martijn van Groningen 5684caece0 [TEST] print hot threads when op job fails
Original commit: elastic/x-pack-elasticsearch@8820758495
2017-01-24 15:51:37 +01:00
David Roberts cd2332730b Move the named pipe no bootstrap test to a separate qa module (elastic/elasticsearch#769)
This matches the way tests that need to run without an Elasticsearch
bootstrap are run in core Elasticsearch.  This should make merging to
x-pack easier.

Note that the no bootstrap tests now run after the integration tests, but
this doesn't really matter.

Original commit: elastic/x-pack-elasticsearch@5547f457b6
2017-01-23 12:08:35 +00:00
Colin Goodheart-Smithe 4c6989212a Gets build to use elasticsearch-extras (elastic/elasticsearch#758)
* Gets build to use elasticsearch-extras

Also adds ci script for building repo on CI servers

To use this change you need to:
1. Clone elasticsearch: `git@github.com:elastic/elasticsearch.git`
2. create a directory at the same level as elasticsearch called `elasticsearch-extra`
3. Clone this repository into the `elasticsearch-extra` directory
4. Run `gradle build` from the `elasticsearch-extra/prelert-legacy` directory or run `gradle :prelert-legacy:build` from the `elasticsearch directory

* Adds USE_SSH option to ci script

* iter

Original commit: elastic/x-pack-elasticsearch@ea127dfef0
2017-01-20 15:11:21 +00:00
Martijn van Groningen 9665368755 Changed job lifecycle to be task oriented.
The job open api starts a task and ties that AutodetectCommunicator.
The job close api is a sugar api, that uses the list and cancel task api to close a AutodetectCommunicator instance.
The flush job and post data api redirect to the node holding the job task and then delegate the flush or data to the AutodetectCommunicator instance.

Also:
* Added basic multi node cluster test.
* Fixed cluster state diffs bugs, forgot to mark ml metadata diffs as named writeable.
* Moved waiting for open job logic into OpenJobAction.TransportAction and moved the logic that was original there to a new action named InternalOpenJobAction.

Original commit: elastic/x-pack-elasticsearch@194a058dd2
2017-01-19 23:15:00 +01:00
Ali Beyad db129051c8 [TEST] fix test to use the stored field instead of id for
stored_template_role

Original commit: elastic/x-pack-elasticsearch@18c9a78587
2017-01-18 12:34:12 -05:00
Lee Hinman a75fecb472 flush after deleting docs
Original commit: elastic/x-pack-elasticsearch@1cd245c398
2017-01-10 14:26:21 -07:00
Lee Hinman 760c960c98 Revert "Revert "Add a qa/rolling-upgrade test that does single-document index and deletes""
This reverts commit elastic/x-pack@94b65dcf52.

Original commit: elastic/x-pack-elasticsearch@abd90463f8
2017-01-10 12:12:09 -07:00
Lee Hinman 05f72c0b91 Revert "Add a qa/rolling-upgrade test that does single-document index and deletes"
This reverts commit elastic/x-pack@ff05f28b7e.

Original commit: elastic/x-pack-elasticsearch@a05fc1f9e6
2017-01-10 11:19:15 -07:00
Lee Hinman 320ec6716d Merge branch 'master' into enhancement/use_shard_bulk_for_single_ops
Original commit: elastic/x-pack-elasticsearch@23761f3e16
2017-01-09 16:27:42 -07:00
Lee Hinman 45e84cde6e Add PreventFailingBuildIT in qa/smoke-test-watcher
The watcher tests were recently marked with `@Network`, which prevents them from
normally being run. Unfortunately, this means no tests run by default and the
entire suite fails.

Original commit: elastic/x-pack-elasticsearch@40cfc75b26
2017-01-09 15:59:32 -07:00
Lee Hinman 8326b6d83b Merge branch 'master' into enhancement/use_shard_bulk_for_single_ops
Original commit: elastic/x-pack-elasticsearch@98f4e74d2e
2017-01-09 14:22:18 -07:00
Jay Modi e0f0b4b7b8 rename the kibana role to kibana_system
This commit renames the kibana role to kibana_system and provides a backwards compatibility
layer so that kibana access still works properly during a rolling upgrade.

Closes elastic/elasticsearch#4525

Original commit: elastic/x-pack-elasticsearch@5c5796e53a
2017-01-09 16:06:50 -05:00
Lee Hinman 93720505b8 Merge branch 'master' into enhancement/use_shard_bulk_for_single_ops
Original commit: elastic/x-pack-elasticsearch@089fa9977d
2017-01-09 11:39:37 -07:00
Lee Hinman 7387d04139 Add a qa/rolling-upgrade test that does single-document index and deletes
Original commit: elastic/x-pack-elasticsearch@5850439b22
2017-01-09 11:16:33 -07:00
Tanguy Leroux a86b112f1e [Watcher] Delete JIRA issues after integration tests execution (elastic/elasticsearch#4552)
This commit delete the JIRA issues after the integration test execution. All issues from the testing project XWT are deleted, even if they have not been created during this specific test execution.

closes elastic/elasticsearch#4535

Original commit: elastic/x-pack-elasticsearch@0362463633
2017-01-09 10:41:48 +01:00
Tanguy Leroux 19cbab4ac3 [Watcher] Update triggering interval in Jira integration tests
In Jira integration tests, some watches are triggered every second whereas they are executed using the watch execute API. This commit increases the triggering interval to 1d so that the watches are not executed on slow machines.

Original commit: elastic/x-pack-elasticsearch@4d0462bc00
2017-01-05 21:00:53 +01:00
Tanguy Leroux 3d2d1d49b6 [Watcher] Move Jira integration tests to smoke-test-watcher (elastic/elasticsearch#4534)
This commit moves the Jira rest integration tests from the smoke-test-watcher-with-mustache project to the smoke-test-watcher project.

Original commit: elastic/x-pack-elasticsearch@c6b03d557f
2017-01-05 15:22:59 +01:00
Alexander Reelsen 63f4bbba98 Watcher: Use Apache HttpClient for internal Watcher HttpClient (elastic/elasticsearch#4434)
Watcher: Use Apache HttpClient for internal Watcher HttpClient

The current implementation based on URLConnection has several drawbacks.

* If server returned HTTP header but then got stuck, no timeout would help, the connection remained stuck
* GET requests with a body were not supported, the method was silently changed to POST
* More complex handling of input/error stream handling, the body could not be read from a single input stream

NOTE: This is a BWC breaker. From now on every part of the URL needs to be encoded properly before it is configured in the requeust builder. This requires an upgrade of all watches.

Closes elastic/elasticsearch#1141

Original commit: elastic/x-pack-elasticsearch@bbc8f85dd8
2017-01-05 14:25:58 +01:00
Tanguy Leroux 125a2c9c03 [Watcher] Enable Jira integration tests (elastic/elasticsearch#4331)
This commit enables the Jira integration tests with the Jira project and account provided by Edward Sy.

closes elastic/infraelastic/elasticsearch#1498

Original commit: elastic/x-pack-elasticsearch@78d1005064
2017-01-05 13:44:04 +01:00
Nik Everett 93b16f0615 Fix another qa project
Original commit: elastic/x-pack-elasticsearch@9609036dde
2017-01-05 10:56:09 +01:00
Nik Everett f2ae490b32 Switch from standalone-test to standalone-rest-test
standalone-rest-test doesn't configure unit tests and for these
integTest only projects that is what we want.

Original commit: elastic/x-pack-elasticsearch@f576dfdfbb
2017-01-05 10:56:09 +01:00
Nik Everett 5b6bfffa9a Require either BuildPlugin or StandaloneTestBasePlugin to use RestTestPlugin
It used to be that RestTestPlugin "came with" StandaloneTestBasePlugin
but we'd like to use it with BuildPlugin for the high level rest client.

Also fix some license headers.

Original commit: elastic/x-pack-elasticsearch@3d5549d170
2017-01-05 10:56:09 +01:00
Jay Modi 18a2cf23d4 Build a single role that represents a user's permissions (elastic/elasticsearch#4449)
This PR changes how we use roles and how we look at the roles of a user. Previously we looked up each role individually, parsed each into their own `Role` object, and had a wrapper that essentially served as an iterator over the roles. The same pattern was also followed for the permissions that composed a role (ClusterPermission, IndicesPermission, and RunAsPermission). This resulted in a lot of code that was hard to follow and could be inefficient. 

Now, we look up the roles for a user in bulk and only get the RoleDescriptor for each role. Once all role descriptors have been retrieved, we build a single Role that represents the user's permissions and we also cache this combination for better performance as authorization can happen many times for a single top level request as we authorize the top level request and any sub requests, which could be a large number in the case of shard requests.

This change also enabled a large cleanup of our permission and privilege classes, which should reduce the footprint of what needs to be followed. Some of the notable changes are:

* Consolidation of GeneralPrivilege and AbstractAutomatonPrivilege into the Privilege class
* The DefaultRole class has been removed and the permissions it provided were moved into the AuthorizationService
* The GlobalPermission class was removed as there is a single role that represents a user's permissions
* The Global inner classes for the various permissions were removed
* The Core inner class was removed and ClusterPermission, IndexPermission, RunAsPermission became final classes instead of interfaces
* The Permission interface has been removed. The isEmpty() method defined by this interface is not needed as we can simply evaluate the permission to get the same effect
* The ClusterPermission#check method only takes the action name again
* The AutomatonPredicate class was removed and replaced by Automatons#predicate
* IndicesAccessControl objects no longer need to be merged when evaluating permissions
* MergedFieldPermissions has been removed
* The Name class that was used to hold an array of strings has been removed and replaced with the use of a Set
* Privilege resolution is more efficient by only combining automata once

Other items:
* NativeRolesStore no longer does caching, so the RoleAndVersion class could be removed
* FileRolesStore doesn't need to be an AbstractLifecycleComponent

Relates elastic/elasticsearch#4327

Original commit: elastic/x-pack-elasticsearch@c1901bc82e
2016-12-30 09:27:49 -05:00
Nik Everett 4ff6279865 Remove exception from client suite
Original commit: elastic/x-pack-elasticsearch@51e94561d9
2016-12-22 09:10:42 -05:00
Areek Zillur 84db1b8731 x-pack changes for elasticsearchelastic/elasticsearch#21964
In https://github.com/elastic/elasticsearch/pull/21964, index
and delete operations are executed as single item bulk requests
internally. This means index and delete operations use the
bulk transport endpoints (indices:data/write/bulk[s][p] and
indices:data/write/bulk[s][r]).

This PR adds bulk transport endpoint to 'write' and 'delete'
index privilages and adds index and delete action as composite
actions to delay the authentication to the shard level.

Original commit: elastic/x-pack-elasticsearch@2305fc9ca0
2016-12-22 02:35:35 -05:00
jaymode 880808c428 test: disable security manager for custom realm integration tests
Original commit: elastic/x-pack-elasticsearch@4bcd7675b2
2016-12-21 16:05:43 -05:00
jaymode 9288401767 test: do not use security manager for client tests
Original commit: elastic/x-pack-elasticsearch@1200a1b757
2016-12-21 14:46:18 -05:00
Ryan Ernst 789df7d2fa Internal: Update signatures for EnvironmentAwareCommand (elastic/elasticsearch#4367)
This is the xplugins side of elastic/elasticsearchelastic/elasticsearch#22175

Original commit: elastic/x-pack-elasticsearch@4359cb1947
2016-12-19 15:23:53 -08:00
Ryan Ernst 6d4d599f91 Build: Convert xplugins to use new extra projects setup (elastic/elasticsearch#4175)
* Build: Convert xplugins to use new extra projects setup

This change makes the gradle initialization for xplugins look in the
correct location for elasticsearch, which is now as a sibling of an
elasticsearch-extra directory, with x-plugins as a child of the extra
directory.

The elasticsearch side of this change is
elastic/elasticsearchelastic/elasticsearch#21773. This change will enable renaming x-plugins
to x-pack, see elastic/elasticsearch#3643.

Original commit: elastic/x-pack-elasticsearch@09398aea5a
2016-12-14 15:02:28 -08:00
Alexander Reelsen b57c4f6ebe Watcher: Remove in memory watch store (elastic/elasticsearch#4201)
In order to prepare to the distributed watch execution, this commit
removes the in memory watch store.

Whenever a watch is needed now, a get request is executed and the parsing
is done. This happens when

* Put
* Get
* Ack
* Activate/Deactivate
* Execute

Note: This also means there are no usage stats currently regarding
the watch count, because we would need to execute a query. This would
require the usage stats to be async, see elastic/elasticsearch#3569

Another advantage is, that there is no dirty flag in the watch itself
needed anymore, because the watch is always the latest. Also write
operations store immediately and dont leave anything in memory.

Also ActionListener.wrap() was used a lot instead of more verbose anonmyous
inner classes.

Original commit: elastic/x-pack-elasticsearch@c47465b47c
2016-12-13 08:54:03 +01:00
Jay Modi ac34c3c37f Remove deprecated methods from the Realm class
This change removes the deprecated methods from the realm class. These methods include
blocking authentication and lookup and the lookup supported method.

Original commit: elastic/x-pack-elasticsearch@cff21e21ee
2016-12-09 07:28:11 -05:00
jaymode 2b7c03848f test: disable xpack plugins on tribe client nodes
The license test for the tribe node disables xpack features but these settings
are not passed to the tribe client nodes, so they need to manually be added.

Original commit: elastic/x-pack-elasticsearch@7273817c47
2016-12-03 14:02:55 -05:00
Simon Willnauer 7f17896927 Revert "Followup for elastic/elasticsearchelastic/elasticsearch#21915 - removal of legacy BWC test infrastructure (elastic/elasticsearch#4247)"
This reverts commit elastic/x-pack@c6c0ffd5d9.

Original commit: elastic/x-pack-elasticsearch@8b7386fb99
2016-12-02 10:55:46 +01:00
Alexander Reelsen 7c04897392 Watcher: Compile scripts on each invocation (elastic/elasticsearch#4239)
Transform and condition scripts were only compiled on its initial creation, so
when a new watch is created or when the master node loads all the watches. However
changing a script (like a stored one) did not lead to any changes in the in memory
watch store and thus the old script was executed again.

We do however have a mechanism in Elasticsearch's ScriptService that already does some
caching, and should reuse that one.

Closes elastic/elasticsearch#4237

Original commit: elastic/x-pack-elasticsearch@477548e237
2016-12-02 10:36:05 +01:00
Simon Willnauer ace1a7e6af Followup for elastic/elasticsearchelastic/elasticsearch#21915 - removal of legacy BWC test infrastructure (elastic/elasticsearch#4247)
Original commit: elastic/x-pack-elasticsearch@07cecdbf00
2016-12-02 08:06:46 +01:00
Jay Modi e8836f99b0 test: add tests that exercise the security index during rolling upgrades
This commit adds basic tests that store a user and a role using the native API. The test checks
that the user and role can be used prior to starting the upgrade. The realm and roles caches are
also cleared to ensure the next authentication will require a read from the security index; this
ensures we are actually testing reads from the index.

Original commit: elastic/x-pack-elasticsearch@396862da94
2016-12-01 11:15:15 -05:00
Jay Modi 139073e8f7 security: improve migrate tool output and remove trappy config option
This commit improves the output of the migrate tool in cases when there are errors parsing entries
in the roles or users files. This is done through the use of a logger that delegates its output to
the terminal.

Additionally, the `-c` option has been removed. This option was used to set the configuration directory
but this should be handled one way only and that is through the use of the `-Epath.conf` setting.

Closes elastic/elasticsearch#3757
Closes elastic/elasticsearch#3758

Original commit: elastic/x-pack-elasticsearch@811e367766
2016-12-01 10:17:28 -05:00
javanna f19be18b20 [TEST] add render search template qa test
Original commit: elastic/x-pack-elasticsearch@7febccbfb2
2016-11-29 23:45:52 +01:00
Luca Cavanna 34d6dc1db1 Categorize search template action as a composite indices request (elastic/elasticsearch#4209)
When we encounter a composite request, we authorize at first without looking at the indices, to see whether the action can be executed at all. We then rely on the action to delegate to an inner action per sub-request, which will be authorized based on the indices it refers to. The first step works great for the simulate mode of search template, as it doesn't involve any index. The second step will make sure that when search template involves a search, it will be authorized as a normal search request would, based on the indices it reads from.

Note that the wildcard expansion happens now on the search side, it doesn't have to happen when executing the first authorization step, hence SearchTemplateRequest doesn't have to implement IndicesRequest, only SearchRequest has to (which it does already).

Closes elastic/elasticsearch#4171

Original commit: elastic/x-pack-elasticsearch@d586bd90cb
2016-11-29 20:53:01 +01:00
Yannick Welsch c7d7a2bafc [TEST] Wait in rolling_upgrade rest test for monitoring indices to be fully allocated
Fixes previous commit elastic/x-pack@8bb4e6a so that it also accounts for monitoring indices that are automatically created by x-pack while the test is running.

Original commit: elastic/x-pack-elasticsearch@e50e1a2717
2016-11-25 12:54:41 +01:00
Yannick Welsch 16b624b1e4 [TEST] Wait in rolling_upgrade rest test for old cluster to have all indices fully allocated
When one of the 2 nodes in the old cluster is shut down, shards that were on that node will become unassigned and be marked to be
delay-allocated, i.e. either a node with shard data for that shard must be available or the allocation of the shards will be delayed for a minute.
In the mixed cluster the replica shard might not be allocated as the primary is already on the node with the newer version and replicas are not allowed
then to be allocated to a node of an older version of ES. Once both nodes are upgraded, the delay might still be in place, and can only be nullified if there
is shard data available on the node. If there never was a shard on that node though, it will take a minute and run into the timeout checking for green.

This commit ensures that all shards are fully-allocated before we do the rolling restart scenario

Original commit: elastic/x-pack-elasticsearch@a0d9b1b043
2016-11-24 16:43:08 +01:00
Yannick Welsch b503f05916 [Test] Fix rolling upgrade test to only wait for yellow in a mixed cluster
If the primary shard of an index with (number_of_replicas > 0) ends up on a new node in a mixed cluster, the replica cannot be allocated to the old node as
the new node might have written segments that use a new postings format or codec that is not available on the older node.
As x-pack automatically creates indices with number_of_replicas > 0, for example monitoring-data-*, the test can only wait for yellow in a mixed cluster.

Original commit: elastic/x-pack-elasticsearch@945d9e3811
2016-11-23 14:36:48 +01:00
Tanguy Leroux a32f2096a6 Add mappings for Jira action (elastic/elasticsearch#4155)
This commit updates the watch_history.json file so that it includes mappings for the new Jira action. It also update the JiraIssue format so that it now includes the name of the account used to create the Jira issue. It also update the REST tests to check that Jira action result are searchable and hide the user's password.

Original commit: elastic/x-pack-elasticsearch@75888f7748
2016-11-23 11:53:06 +01:00
Ryan Ernst 1dc839bd98 Remove groovy scripting language (elastic/elasticsearch#4162)
This is the xplugins side of elastic/elasticsearchelastic/elasticsearch#21607

Original commit: elastic/x-pack-elasticsearch@125843e814
2016-11-22 22:45:15 -08:00
jaymode 9126600fc3 test: add timeout to cluster health call that we are waiting on
This commit adds a timeout to the cluster health call that we wait on so that we can
see the status of the health request instead of getting timeouts failures with no
information to go on.

Original commit: elastic/x-pack-elasticsearch@2f34d01e00
2016-11-22 15:17:21 -05:00
jaymode 6170f3d22c test: use the right number of bwc nodes in old cluster
Original commit: elastic/x-pack-elasticsearch@0afc045bbc
2016-11-22 15:14:29 -05:00
Jay Modi 4239ba5415 allow reads of native users and roles when template version hasn't been updated
This change allows reads of our native users and roles when the template version has not been updated to
match the current version. This is useful for rolling upgrades where the nodes are also being actively
queried and/or indexed into. Without this, we can wreak havoc on a cluster by causing exceptions during
replication, which leads to shard failures. On nodes that match the version defined in the template,
write operations are allowed since we know that we are backwards compatible in terms of format but we
may have added new fields and shouldn't index them until the mappings and template have been updated.

As part of this, the rolling upgrade tests from core were used as the basis for a very basic set of tests
for doing a rolling upgrade with x-pack.

Closes elastic/elasticsearch#4126

Original commit: elastic/x-pack-elasticsearch@9be518ef00
2016-11-22 12:00:09 -05:00
Areek Zillur a9f3619b5a Enable merging license in tribe node (elastic/elasticsearch#4147)
Currently, a tribe node ignored underlying cluster licenses
due to inablity to select an appropriate license from  multiple
licenses. Now that tribe node supports merging custom metadata
(elasticsearch#elastic/elasticsearch#21552), we can enable license support in tribe
node.

Now, tribe node chooses license with the highest operation
mode from underlying cluster licenses. This commit also
adds integration tests for licensing to verify that:
 - autogenerated trial license propagates to tribe node
 - tribe node chooses the highest operation mode license
 - removing a license from underlying cluster license is
   removed from tribe

closes elastic/elasticsearch#3212

Original commit: elastic/x-pack-elasticsearch@b5c003decd
2016-11-22 11:42:51 -05:00
Tanguy Leroux 18478d63c2 Watcher: Add JIRA action (elastic/elasticsearch#4014)
closes elastic/elasticsearch#493

Original commit: elastic/x-pack-elasticsearch@6b7387d3e4
2016-11-21 10:52:55 +01:00
Simon Willnauer 92040ef72e Remove netty_3 support from xpack (elastic/elasticsearch#4097)
This is a followup from elastic/elasticsearchelastic/elasticsearch#21590 and needs to be
committed first or at the same time since netty_3 is removed

Original commit: elastic/x-pack-elasticsearch@131d74dd6b
2016-11-17 12:44:24 +01:00
Tanguy Leroux 4badf28a8d Add Vagrant Gradle plugin (elastic/elasticsearch#3993)
This commit adds a new Gradle sub project that makes use of the VagrantTestPlugin in order to test the installation of X-Pack.

Original commit: elastic/x-pack-elasticsearch@e09db6602c
2016-11-15 15:30:13 +01:00
Yannick Welsch 7b165504dc Use project-defined Java installation for keytool (elastic/elasticsearch#4066)
Companion commit for elastic/elasticsearchelastic/elasticsearch#21540

Original commit: elastic/x-pack-elasticsearch@a1c21ece25
2016-11-15 09:32:04 +01:00
Ryan Ernst 23e6cab7f1 Merge pull request elastic/elasticsearch#4044 from rjernst/realm_sig
Extensions: Make resource watcher available to custom realms

Original commit: elastic/x-pack-elasticsearch@3cb494e98d
2016-11-14 12:38:27 -08:00
Yannick Welsch 9d5ebe9e2a Use project-defined Java installation for keytool
Companion commit for elastic/elasticsearchelastic/elasticsearch#21540

Original commit: elastic/x-pack-elasticsearch@aa4e2df5bf
2016-11-14 15:43:11 +01:00
Nik Everett 700467c3a7 Use index_patterns in templates
And skip a REST test that won't pass.

Original commit: elastic/x-pack-elasticsearch@e297add6c1
2016-11-10 21:42:59 -05:00
Ryan Ernst bcd32ada4f Extensions: Make resource watcher available to custom realms
This simply adds ResourceWatcherService as an arg for getting custom
realms from xpack extensions.

closes elastic/elasticsearch#4038

Original commit: elastic/x-pack-elasticsearch@fe58d8a7ee
2016-11-10 12:43:28 -08:00
Ryan Ernst fa97a806ca Remove unneeded rest test params
This is the xplugins side of elastic/elasticsearchelastic/elasticsearch#21391

Original commit: elastic/x-pack-elasticsearch@30d36e340a
2016-11-07 14:46:39 -08:00
Jay Modi bd522191b2 test: use toJSON inside a role
This commit adds a test that uses toJSON templating inside of a role with
an array defined in the user's metadata.

Originates from user discussion at:
https://discuss.elastic.co/t/x-pack-security-role-definition-query-template-with-terms/62790

Original commit: elastic/x-pack-elasticsearch@196f7f597c
2016-11-07 07:21:41 -05:00
Jack Conradson 7dd4188299 Cleanup ScriptType (elastic/elasticsearch#3922)
Refactored ScriptType to clean up some of the variable and method names. Added more documentation. Deprecated the 'in' ParseField in favor of 'stored' to match the indexed scripts being replaced by stored scripts.

Original commit: elastic/x-pack-elasticsearch@d7c7bd7362
2016-10-31 13:49:10 -07:00
Nik Everett 0f54f9524d Handle reindex's change to whitelist
Reindex has grown support for simple whitelist patterns like
`localhost:*` and lost support for `myself`.

Original commit: elastic/x-pack-elasticsearch@31d9c4dc5c
2016-10-18 21:46:30 -04:00
javanna 4bb6e856f3 Authorize composite actions based on their action name only, subrequests and their indices will be later authorized individually
Eagerly authorizing CompositeIndicesRequests allowed the security plugin to fail fast up until now, but it makes it very hard to reason about each specific item in a multi items request. Either all items fail, or none do. We would rather want to adopt a similar behaviour to es core, where individual items fail without affecting other items that are part of the same request. We can rely on the fact that es core always authorizes both main action and every subaction too, and skip authorization for the main action. By subaction we mean either all sub search requests in msearch, as well as each shard level get in mget or shard level bulk request for bulk.

 BulkRequestInterceptor was converted to intercept BulkShardRequests rather than BulkRequest as that is where bulk is authorized after this change.

 Split IndicesAndAliasesResolverIntegrationTests into ReadActionsTests and WriteActionsTests as they require different set of permissions, lots of tests added.

Explicitly listing the composite actions makes sure that the actions that can bypass security are known, somebody adding a similar action must to add it to the list, so we know it doesn't happen by mistake. At this point the CompositeIndicesRequest can be used as a marker interface only (it is not really needed but can be used to verify that composite actions use a request that implements such interface).

Given that we don't authorize composite actions based on their indices anymore, but only their sub-requests which implement IndicesRequest, printing out the indices names in the audit log for requests like bulk and msearch is confusing. Removed support for that.

Authorize composite indices actions based on their name only, their indices will be authorized at the sub-request/shard level

Rather than simply granting bulk, mget, msearch etc. and relying on authorization at the sub-request/shard level, we check that the current user can at least execute the action. This justifies the grant line that gets written in the audit log, the action is potentially possible without looking at the indices. Each specific item will fail or succeed later and will yield its own specific audit log entry.

Original commit: elastic/x-pack-elasticsearch@4570caf019
2016-10-13 16:05:02 +02:00
javanna d27c4bee82 Support allowNoIndices option in security plugin
Supporting allowNoIndices means that the security plugin has a behaviour much more similar to vanilla es when dealing with wildcard expressions that match no indices, or empty clusters. The default for most request is to allow no indices, but security plugin could only disallow no indices all the time up until now.

The technical problem was that when anything gets resolved to an empty set of indices, we couldn't let that go through to es core, as that would become resolved to all indices by es core, which would be a security hole. We have now found a way though to replace an empty set of indices with something that es core will for sure resolve to no indices, so we can let the request through. We simply replace empty indices with '-*'.

Multi apis requests (e.g. _msearch) have yet to be fixed, as all their indices end up in the same bucket while they should each be authorized separately, so that every specific item can fail or be let through.

Original commit: elastic/x-pack-elasticsearch@0f67a0bfea
2016-10-13 16:05:02 +02:00
javanna 9b46b34bed Honour ignore_unavailable option when resolving indices
For all the requests that support multiple indices and wildcards, hence implementing IndicesRequest.Replaceable, we replace the wildcard expressions with the explicit names of the authorized indices they match. _all or empty indices is treated as a wildcard expression. We can also honour the ignore_unavailable option by going over all the explicit names and filter out the non authorized ones when ignore_unavailable is set to true. If ignore_unavailable is set to false, we leave everything as-is, which will cause an authorization exception to be thrown if only one of those explicit indices is not authorized for the current user.

This is the first step towards resolving elastic/elasticsearch#1250. The remaining issue is that in case we are left with no indices after stripping out the ones that the user is not authorized for, we throw an authorization exception rather than returning an empty response. That will require honouring the allow_no_indices option, which will also change the behaviour when a cluster is empty.

Relates to elastic/elasticsearch#1250

Original commit: elastic/x-pack-elasticsearch@e4ca940d05
2016-10-13 16:05:02 +02:00
Alexander Reelsen 8b83cf067c Watcher: Ensure awesome painless exceptions are propagated to the user (elastic/elasticsearch#3707)
When adding a watch which has a painless component, the scriptexception
was wrapped into a deprecated exception which means, that the awesome
painless descriptions were lost. This wrapping has been removed.

Closes elastic/elasticsearch#3161

Original commit: elastic/x-pack-elasticsearch@1703fe4eb6
2016-10-12 08:14:06 +02:00
Nik Everett 6e31ab8d99 Skip a new template rest test
It isn't compatible with security.

Relates to elastic/elasticsearch#20658

Original commit: elastic/x-pack-elasticsearch@33df690341
2016-10-11 12:30:45 -04:00
Tanguy Leroux 2e7b7be25c Watcher: Re enable array compare test (elastic/elasticsearch#3708)
This test has been blacklisted and deactivated months ago. This commit reenables this test and moves it at the right place. It also change the test to use the Execute Watch API instead of being sleep based.

Original commit: elastic/x-pack-elasticsearch@e7a9689375
2016-10-11 10:25:40 +02:00
Simon Willnauer 2f70ae92b6 Cut over to MockTcpTransport since LocalTransport is remove in core (elastic/elasticsearch#3684)
This is a followup commit to elastic/elasticsearchelastic/elasticsearch#20695

Original commit: elastic/x-pack-elasticsearch@27cd454ba6
2016-10-07 11:28:05 +02:00
Ryan Ernst 1fa0f835fe Build: Reorganize src roots
This change flattens the directory structure, both for the elasticsearch
specific directories, as well as within the elasticsearch x-pack plugin.

closes elastic/elasticsearch#2957

Original commit: elastic/x-pack-elasticsearch@45891a4632
2016-10-01 09:46:43 +02:00
Ryan Ernst 905237a56f Moved directories around
Original commit: elastic/x-pack-elasticsearch@2018bb5f9f
2016-09-29 12:03:14 +02:00
jaymode e5b0e7f5cb reorganize directory layout
See elastic/elasticsearch#1022

Original commit: elastic/x-pack-elasticsearch@3ee8761312
2015-12-03 16:22:37 +01:00
Robert Muir 9df905ff19 Simplify SSL test to not use openssl.
I think the intent here is to just test that our SSL layers work,
not invoke a long chain of keytool + openssl commands.

This simplifies the build and will work on windows.

Original commit: elastic/x-pack-elasticsearch@af07d0d4f7
2015-11-30 09:28:16 -05:00
Robert Muir 8a22ba0a08 smoke-test-plugins-ssl shoudl check if openssl is available
Today some jenkins servers dont have it (e.g. windows), and it constantly fails...

Original commit: elastic/x-pack-elasticsearch@6b561c73e0
2015-11-30 08:42:25 -05:00