Commit Graph

202 Commits

Author SHA1 Message Date
Colin Goodheart-Smithe 99db6013ea [ML] Adds a test for ml transport client (elastic/x-pack-elasticsearch#851)
This change also adds synchronous methods to `MachineLearningClient`.

relates elastic/x-pack-elasticsearch#567

Original commit: elastic/x-pack-elasticsearch@b3a4b38a51
2017-03-28 09:05:55 +01:00
Martijn van Groningen fc29fd159a [TEST] Don't lose the actual cause of the exception.
Original commit: elastic/x-pack-elasticsearch@8720288ef5
2017-03-27 15:33:04 +02:00
David Roberts ecbfaace38 [TEST] Improve ML security test comments and account for new close assertion
Original commit: elastic/x-pack-elasticsearch@004845bf2f
2017-03-27 10:24:05 +01:00
David Roberts c0445fac4d [TEST] Silence failing ML security tests
Original commit: elastic/x-pack-elasticsearch@af4ed2019d
2017-03-24 17:58:30 +00:00
Colin Goodheart-Smithe caa9afad2b [TEST] fixed content type in ML QA tests
Original commit: elastic/x-pack-elasticsearch@b7749e1319
2017-03-24 14:08:39 +00:00
Hendrik Muhs a8b1b3e863 Revert "disable integration test, see elastic/x-pack-elasticsearch#823 for details (elastic/x-pack-elasticsearch#824)"
This reverts commit elastic/x-pack-elasticsearch@2338b3c972.

Original commit: elastic/x-pack-elasticsearch@a9c7168762
2017-03-23 20:10:58 +01:00
Hendrik Muhs 27dce8669c disable integration test, see elastic/x-pack-elasticsearch#823 for details (elastic/x-pack-elasticsearch#824)
[ML] disable failing integration test, see elastic/x-pack-elasticsearch#823 for details

Original commit: elastic/x-pack-elasticsearch@2338b3c972
2017-03-23 19:54:02 +01:00
Hendrik Muhs 6f7f466fa3 [ML] move DataStreamDiagnostics into DataCountsReporter (elastic/x-pack-elasticsearch#775)
repair DataStreamDiagnostics

Moves DataStreamDiagnostics into DataCountsReporter to survive if job is opened/closed/fed in chunks.

relates elastic/x-pack-elasticsearch#764

Original commit: elastic/x-pack-elasticsearch@29c221a451
2017-03-23 16:43:51 +01:00
Jay Modi 8ba6e8b3eb Fix merging of field level security rules (elastic/x-pack-elasticsearch#796)
This commit fixes the merging of field level security rules from multiple roles. Prior to 5.2, the
merging was treated as the merging of languages whereas after 5.2, this incorrectly became a merge
of all rules meaning a single wildcard could cause denials to be ignored.

Original commit: elastic/x-pack-elasticsearch@42f9e6d8b0
2017-03-22 05:10:30 -07:00
Martijn van Groningen 60c79ccdbb [TEST] Set job close timeout to 20s.
Original commit: elastic/x-pack-elasticsearch@9b92c940cb
2017-03-16 21:57:36 +01:00
Martijn van Groningen 73c5c1576f [TEST] Added logging and removed unneeded jackson-databind test dependency.
Original commit: elastic/x-pack-elasticsearch@58c247d696
2017-03-16 21:17:14 +01:00
Dimitris Athanasiou 16efd4e474 [ML] Use TimeValue in job and datafeed configs (elastic/x-pack-elasticsearch#732)
relates elastic/x-pack-elasticsearch#679

Original commit: elastic/x-pack-elasticsearch@891edd5bfe
2017-03-16 14:07:47 +00:00
Ryan Ernst baaad36c5e Fix tribe test setup to look for a single node
numNodes isn't available because it is not in the normal context of
cluster configuration.

relates elastic/x-pack-elasticsearch#740

Original commit: elastic/x-pack-elasticsearch@336147f20c
2017-03-15 15:55:10 -07:00
Ryan Ernst 534584d525 Tests: Use cluster health api for wait condition in ml integ test
This was forgotten in elastic/x-pack-elasticsearch#740

Original commit: elastic/x-pack-elasticsearch@47c10dc543
2017-03-15 10:27:38 -07:00
Ryan Ernst 8c01d6ea69 Tests: Add cluster health check to xpack integ wait conditions (elastic/x-pack-elasticsearch#740)
The wait condition used for integ tests by default calls the cluster
health api with wait_for_nodes nd wait_for_status. However, xpack
overrides the wait condition to add auth, but most of these conditions
still looked at the root ES url, which means the tests are susceptible
to race conditions with the check and node startup. This change modifies
the url for the authenticated wait condtion to check the health api,
with the appropriate wait_for_nodes and wait_for_status.

Original commit: elastic/x-pack-elasticsearch@0b23ef528f
2017-03-15 10:23:26 -07:00
David Roberts 0b7c735aec [ML] Add machine learning privileges/roles (elastic/x-pack-elasticsearch#673)
* Changed ML action names to allow distinguishing of admin and read-only actions
  using wildcards
* Added manage_ml and monitor_ml built-in privileges as subsets of the existing
  manage and monitor privileges
* Added out-of-the-box machine_learning_admin and machine_learning_user roles
* Changed machine learning results endpoints to use a NodeClient rather than an
  InternalClient when searching for results so that index/document level permissions
  applied to ML results are respected

Original commit: elastic/x-pack-elasticsearch@eee800aaa8
2017-03-14 16:13:41 +00:00
Dimitris Athanasiou a7a36245c9 [ML] Add end-point for deleting expired results (elastic/x-pack-elasticsearch#670)
This commit adds an end-point to force deletion of expired data:

DELETE /_xpack/ml/_delete_expired_data

A few other things are changed too:

- Delete expired results from now rather than start of day
- Rename MlDaily{Management -> Maintenance}Service
- Refresh job indices when job is closing to ensure latest result
  visibility
- Commit results when quantiles are persisted to ensure they are visible
  for renormalization


Original commit: elastic/x-pack-elasticsearch@8ca5272a94
2017-03-03 11:33:22 +00:00
Dimitrios Athanasiou 0542a9eb92 [TEST] Disable ml in qa modules where necessary
Original commit: elastic/x-pack-elasticsearch@bb311b44d7
2017-03-02 17:01:35 +00:00
Martijn van Groningen 948a8594bb [TEST] Disable ml in audit qa test
Original commit: elastic/x-pack-elasticsearch@34b7243f69
2017-03-02 16:02:15 +01:00
Luca Cavanna 70a4a69e39 Fix indentation in some yaml tests (elastic/x-pack-elasticsearch#686)
The yaml test runner now throws error when skip or do sections are malformed, such as they don't start with the proper token (START_OBJECT). That signals bad indentation, which was previously ignored. Thanks (or due to) our pull parsing code, we were still able to properly parse the sections, yet other runners weren't able to.

Original commit: elastic/x-pack-elasticsearch@920201207c
2017-03-02 12:43:43 +01:00
Nik Everett dcec4bbf4b Handle Mustache not using Settings any more
It didn't need them so now it doesn't take them.

Original commit: elastic/x-pack-elasticsearch@f047c3926b
2017-03-01 14:59:14 -05:00
Ali Beyad b20578b9f6 Enables X-Pack extensions to implement custom roles providers (elastic/x-pack-elasticsearch#603)
This commit adds the ability for x-pack extensions to optionally
provide custom roles providers, which are used to resolve any roles
into role descriptors that are not found in the reserved or native
realms.  This feature enables the ability to define and provide roles
from other sources, without having to pre-define such roles in the security
config files.

relates elastic/x-pack-elasticsearch#77

Original commit: elastic/x-pack-elasticsearch@bbbe7a49bf
2017-03-01 12:20:34 -05:00
Areek Zillur ec4de10ee2 Tribe node security tests with external clusters (elastic/x-pack-elasticsearch#606)
* Tribe node security tests with external clusters

This PR adds a qa module for security tests with tribe node
using external clusters. Existing SecurityTribeIT tests
have been ported to use external clusters with tribe setup
as a first step.

Currently the ports to the external clusters are passed to the
integration tests through system properties and external clusters
are built on test setup (the code for building external clusters is
copied from ESIntegTestCase). This is a WIP as we need a
more generic way to facilitate testing tribe setup with external
clusters. thoughts welcome.

* incorporate feedback

* update to master

Original commit: elastic/x-pack-elasticsearch@686887ca91
2017-02-28 18:36:53 -05:00
Martijn van Groningen 6783f823a8 [ML] Change stop datafeed api delegate to node hosting datafeed task and execute cancel locally,
instead of only removing the persistent task from cluster state.

Original commit: elastic/x-pack-elasticsearch@3974b20827
2017-02-28 16:36:13 +01:00
javanna 33ccc3bd6c adapt to ObjectPath changes to support binary formats
Original commit: elastic/x-pack-elasticsearch@11782418e9
2017-02-27 12:28:04 +01:00
David Roberts e2f5715191 [TEST] Fix one last test that was missed in elastic/x-pack-elasticsearch#649
Original commit: elastic/x-pack-elasticsearch@1039197afd
2017-02-27 09:55:49 +00:00
Ali Beyad 4a001706e7 Simplifies security index state changes and template/mapping updates (elastic/x-pack-elasticsearch#551)
Currently, both the NativeUsersStore and NativeRolesStore can undergo
multiple state transitions. This is done primarily to check if the
security index is usable before it proceeds. However, such checks are
only needed for the tests, because if the security index is unavailable
when it is needed, the downstream actions invoked by the
NativeUsersStore and NativeRolesStore will throw the appropriate
exceptions notifying of that condition. In addition, both the
NativeUsersStore and NativeRolesStore had much duplicate code that
listened for cluster state changes and made the exact same state
transitions.

This commit removes the complicated state transitions in both classes
and enables both classes to use the SecurityTemplateService to monitor
all of the security index lifecycle changes they need to be aware of.

This commit also moves the logic for determining if the security index
needs template and/or mapping updates to the SecurityLifecycleService,
and makes the NativeRealmMigrator solely responsible for applying the
updates.

Original commit: elastic/x-pack-elasticsearch@b31d144597
2017-02-23 13:05:39 -05:00
Ryan Ernst 8527bc2415 Build: Convert integ test dsl to new split cluster/runner dsl
This is the xpack side of elastic/elasticsearch#23304

Original commit: elastic/x-pack-elasticsearch@8eddd7fb0d
2017-02-22 00:56:52 -08:00
Martijn van Groningen 8a6cea0350 [ml] For the data, flush and update apis check the job state on the node running the job task instead of the coordinating node.
[test] Also re-enabled multi node qa tests.

Original commit: elastic/x-pack-elasticsearch@efb24131f0
2017-02-18 19:18:29 +01:00
Jay Modi 68324f5a50 Remove usage of deprecated methods that do not accept a XContentType (elastic/x-pack-elasticsearch#549)
This commit removes the usages and definition of deprecated methods that do not accept the XContentType for the source.

Additionally, usages of *Entity classes from the http client library have been changed to always specify the content
type.

Original commit: elastic/x-pack-elasticsearch@29d336a008
2017-02-17 14:45:00 -05:00
Martijn van Groningen 92a4b97f6d [TEST] Add simple test, so that module has at least one working test.
Original commit: elastic/x-pack-elasticsearch@72441418ac
2017-02-17 19:44:04 +01:00
David Kyle d707a1c977 Mark class with awaitsfix
Is the test failing because all the methods are marked awaitsfix?

Original commit: elastic/x-pack-elasticsearch@de93bb83b7
2017-02-17 18:06:42 +00:00
David Kyle b9a4a2c621 Revert "Don’t wait on job close (elastic/x-pack-elasticsearch#574)"
This reverts commit elastic/x-pack-elasticsearch@99ed0e0dba and fixes the failing tests

Original commit: elastic/x-pack-elasticsearch@403e38316d
2017-02-17 17:05:41 +00:00
David Kyle 1a4b87454c Add AwaitsFix for elastic/x-pack-elasticsearch#592
Original commit: elastic/x-pack-elasticsearch@c5d2650acf
2017-02-17 14:05:25 +00:00
Martijn van Groningen 6739f84efb [test] change id between tests, to make reading log files easier.
Original commit: elastic/x-pack-elasticsearch@e633d7098e
2017-02-17 14:55:14 +01:00
Martijn van Groningen 6d79210f79 [ml] Registered named writable entry for DatafeedState.
Also added a test to multi node qa module that tests the datafeeder, which should have caught this.

Original commit: elastic/x-pack-elasticsearch@89e4875f6c
2017-02-15 21:38:19 +01:00
Ali Beyad b3d72af644 [TEST] bumps rolling upgrade bwc version to 5.4.0-snapshot
Original commit: elastic/x-pack-elasticsearch@2f321682d2
2017-02-15 11:00:11 -05:00
Zachary Tong 65cd119733 [ML] Fix injection of DomainSplit, and only inject for Painless
Was accidentally injecting the script object, not the string version of the code.  Also
added a check so we only inject for Painless scripts (and not groovy, etc).

Minor style tweaks too.

Original commit: elastic/x-pack-elasticsearch@58c7275bd8
2017-02-15 09:31:16 -05:00
Colin Goodheart-Smithe 6183015639 Adds licensing to machine learning feature (elastic/x-pack-elasticsearch#559)
This change adds licensing to the maching learning feature, and only allows access to machine learning if a trial or platinum license is installed.

Further, this change also renames `MlPlugin` to `MachineLearning` in line with the other feature plugin names and move the enabled setting to `XPackSettings`

Original commit: elastic/x-pack-elasticsearch@48ea9d781b
2017-02-14 15:47:37 +00:00
Ryan Ernst 2571921605 Rename x-pack project names to new names with split repo
Original commit: elastic/x-pack-elasticsearch@5a908f5dcc
2017-02-10 11:02:42 -08:00
Colin Goodheart-Smithe 1da752152a Migrates machine learning into X-Pack
Original commit: elastic/x-pack-elasticsearch@9ad22980b8
2017-02-08 16:58:56 +00:00
Colin Goodheart-Smithe e761b76765 Migrates QA files
Original commit: elastic/x-pack-elasticsearch@ac651f51d4
2017-02-08 16:58:55 +00:00
Simon Willnauer 0e779b41de Followup from elastic/elasticsearchelastic/elasticsearch#23042 (elastic/elasticsearch#4895)
This change accomodates for the renamings done in elastic/elasticsearchelastic/elasticsearch#23042

Original commit: elastic/x-pack-elasticsearch@c290c8ecc4
2017-02-08 14:40:17 +01:00
Tim Vernum 734a4ee66d Prevent default passwords in production mode (elastic/elasticsearch#4724)
Adds a new `xpack.security.authc.accept_default_password` setting that defaults to `true`. If it is set to false, then the default password is not accepted in the reserved realm.

Adds a bootstrap check that the above setting must be set to `false` if security is enabled.  

Adds docs for the new setting and bootstrap.

Changed `/_enable` and `/_disable`, to store a blank password if the user record did not previously exist, which is interpreted to mean "treat this user as having the default password". The previous functionality would explicitly set the user's password to `changeme`, which would then prevent the new configuration setting from doing its job.

For any existing reserved users that had their password set to `changeme`, migrates them to the blank password (per above paragraph)

Closes: elastic/elasticsearch#4333

Original commit: elastic/x-pack-elasticsearch@db64564093
2017-02-08 16:19:55 +11:00
Jay Modi bd04b30acd security: transport always uses TLS (elastic/elasticsearch#4738)
This commit brings back support an auto-generated certificate and private key for
transport traffic. The auto-generated certificate and key can only be used in development
mode; when moving to production a key and certificate must be provided.

For the edge case of a user not wanting to encrypt their traffic, the user can set
the cipher_suites setting to `TLS_RSA_WITH_NULL_SHA256` or a like cipher, but a key/cert
is still required.

Closes elastic/elasticsearch#4332

Original commit: elastic/x-pack-elasticsearch@b7a1e629f5
2017-02-07 11:39:31 -05:00
David Kyle 5a1cd69a6a More checkstyle fixes
Original commit: elastic/x-pack-elasticsearch@4c454c5061
2017-02-03 16:58:19 +00:00
Jason Tedor 02bcd9510f Enable disabled rolling upgrade tests
These tests were disabled due to a bug introduced when
sequence-number-based recovery was introduced. This commit enables these
tests again.

Relates elastic/elasticsearch#4801

Original commit: elastic/x-pack-elasticsearch@e555bfec1a
2017-02-03 08:12:52 -05:00
Jay Modi 1f32ef21a2 Update to pass content type from the rest APIs to transport requests (elastic/elasticsearch#4689)
This change adapts x-pack to pass on the parsed XContentType from rest requests to transport
requests and use this value in place of attempting to auto-detect the content type.

Original commit: elastic/x-pack-elasticsearch@57475fd403
2017-02-02 14:08:34 -05:00
polyfractal 3504608a1e [TEST] more robust regex for tests missing subdomains
Original commit: elastic/x-pack-elasticsearch@28e5d14c22
2017-02-01 11:04:55 -05:00
Zachary Tong a11ddd1e04 Integrate domainSplit function into datafeeds (elastic/elasticsearch#841)
If `domainSplit(` is detected in an inline script, the function and params are injected into
the script.

The majority of this PR is actually test-related.  Adds a unit test to check for the injected
script/params.  Also adds another QA test which -- through a very round-about mechanism --
confirms that the injected script compiles and functions correctly.  The QA test can
be simplified greatly once the Preview API is added.

Original commit: elastic/x-pack-elasticsearch@c7c35a982c
2017-02-01 10:20:00 -05:00