Commit Graph

233 Commits

Author SHA1 Message Date
Ryan Ernst ef3d3b51a4 Move integ test runner deps to cluster deps (elastic/x-pack-elasticsearch#1096)
This is the xpack side of elastic/elasticsearch#24142

Original commit: elastic/x-pack-elasticsearch@d502f06cea
2017-04-17 16:04:09 -07:00
Jay Modi b59b6bbdd4 Remove SecuredString and use SecureString from elasticsearch core (elastic/x-pack-elasticsearch#1092)
This commit removes the SecuredString class that was previously used throughout the security code
and replaces it with the SecureString class from core that was added as part of the new secure
settings infrastructure.

relates elastic/x-pack-elasticsearch#421

Original commit: elastic/x-pack-elasticsearch@e9cd117ca1
2017-04-17 13:28:46 -04:00
Martijn van Groningen d114a55b99 [ML] Make open job and start datafeed apis master node actions and
let close job and stop datafeed apis redirect to elected master node.

This is for cluster state observation purposes, so that a subsequent open and then close job or
start and then stop datafeed see the same local cluster state and sanity validation doesn't fail.

Original commit: elastic/x-pack-elasticsearch@21a63184b9
2017-04-14 10:43:21 +02:00
David Kyle 17a8c9b9e8 Revert "Muted test."
This reverts commit elastic/x-pack-elasticsearch@da69d049ad.

Original commit: elastic/x-pack-elasticsearch@e676c4267c
2017-04-13 14:15:24 +01:00
Martijn van Groningen fc06f09b91 [TEST] Only run qa cluster with ml enabled.
Relates to elastic/x-pack-elasticsearch#997

Original commit: elastic/x-pack-elasticsearch@863bf9abfe
2017-04-13 10:50:03 +02:00
Martijn van Groningen 4f34af20ba Muted test.
Original commit: elastic/x-pack-elasticsearch@da69d049ad
2017-04-11 20:51:52 +02:00
Simon Willnauer a7dfbcd2cb [TEST] Use SecurityClusterClientYamlTestCase to ensure mappings and templates are present
Original commit: elastic/x-pack-elasticsearch@742633c93c
2017-04-11 18:11:31 +02:00
Ali Beyad c468a4d3a3 [TEST] Security index template/mappings check on tests (elastic/x-pack-elasticsearch#1031)
Extends the security index check that ensure the template/mappings
are up-to-date to other security integration tests that depend on this
check.

relates elastic/x-pack-elasticsearch#794

Original commit: elastic/x-pack-elasticsearch@ec8e5b37bd
2017-04-11 11:22:41 -04:00
Simon Willnauer 9c542173b7 [TEST] Make test more robust against slow connection establishing
Original commit: elastic/x-pack-elasticsearch@6f918a44c9
2017-04-11 14:43:44 +02:00
Dimitris Athanasiou 4da1c5b9dc [ML] Remove batch_span and period from analysis config (elastic/x-pack-elasticsearch#1043)
relates elastic/x-pack-elasticsearch#1040

Original commit: elastic/x-pack-elasticsearch@aed5e9912c
2017-04-11 12:57:58 +01:00
Hendrik Muhs 7ef9a16f45 [ML] implement '_all' for stopping datafeeds (elastic/x-pack-elasticsearch#995)
Add a '_all' functionality for stopping ML datafeeds.

For cluster shutdown due to maintenance and major upgrades we recommend the user to stop all datafeeds and jobs. This change add the ability to stop all datafeeds at once where previously it was required to iterate over all feeds and do a explicit stop. This is part two of elastic/x-pack-elasticsearch#795, part one can be found in elastic/x-pack-elasticsearch#962 .

relates elastic/x-pack-elasticsearch#795

Original commit: elastic/x-pack-elasticsearch@ed1eff83d5
2017-04-11 13:39:22 +02:00
Simon Willnauer 617c3ead5c Add tests for _remote/info API (elastic/x-pack-elasticsearch#1009)
Relates to elastic/elasticsearch#23925
Depends on elastic/elasticsearch#23969

Original commit: elastic/x-pack-elasticsearch@d1e8754a57
2017-04-11 11:24:22 +02:00
Tim Vernum 8840042751 Only log LDAP errors if authentication failed overall (elastic/x-pack-elasticsearch#960)
Changes the logging of LDAP authentication failures from "always" to "only if the user failed to be authenticated"

Previously there were cases (such has having 2 AD realms) where successful user authentication would still cause an INFO message to be written to the log for every request. 
Now that message is suppressed, but a WARN message is added _if-and-only-if_ the user cannot be authenticated by any realm.

This is implemented via a new value stored in the ThreadContext that the AuthenticationService choses to log (or not log) depending on the result of the authenticate process.
  
Closes: elastic/x-pack-elasticsearch#887

Original commit: elastic/x-pack-elasticsearch@b81b363729
2017-04-11 09:37:21 +10:00
Tanguy Leroux 6396edc6a7 [Test] Add Monitoring Bulk API REST tests with privileges (elastic/x-pack-elasticsearch#968)
This commit removes the smoke-test-monitoring-with-security project
and replaces it with a REST test.

Original commit: elastic/x-pack-elasticsearch@f1665815c2
2017-04-10 15:08:19 +02:00
Ryan Ernst e8399cfccf Build: Build 5.x version of xpack for bwc tests to use (elastic/x-pack-elasticsearch#885)
This is analagous of the bwc-zip for elasticsearch. The one caveat is
due to the structure of how ES+xpack must be checked out, we end up with
a third clone of elasticsearch (the second being in :distribution:bwc-zip).
But the rolling upgrade integ test passes with this change.

relates elastic/x-pack-elasticsearch#870

Original commit: elastic/x-pack-elasticsearch@34bdce6e99
2017-04-07 22:19:30 -07:00
Hendrik Muhs c7fd1aacff [ML] implement _all for closing jobs (elastic/x-pack-elasticsearch#962)
Add a '_all' functionality for closing ML jobs.

For cluster shutdown due to maintenance and major upgrades we recommend the user to stop all datafeeds and jobs. This change add the ability to close all jobs at once where previously it was required to iterate over all jobs and do a explicit close. This is part one of elastic/x-pack-elasticsearch#795, part two can be found in elastic/x-pack-elasticsearch#995.

relates elastic/x-pack-elasticsearch#795

Original commit: elastic/x-pack-elasticsearch@9b251ed7e1
2017-04-07 14:51:13 +02:00
Martijn van Groningen 2b5c0faebd [TEST] wait for cs version to be the same on all nodes before stopping datafeed
Original commit: elastic/x-pack-elasticsearch@6fd6224047
2017-04-05 11:37:36 +02:00
Martijn van Groningen 4a57c92e6f [TEST] unmuted test
Original commit: elastic/x-pack-elasticsearch@5dd80c32c7
2017-04-05 09:46:02 +02:00
Jason Tedor dee1bd1825 Respond to rename random ASCII helper methods
This commit is response to the renaming of the random ASCII helper
methods in ESTestCase. The name of this method was changed because these
methods only produce random strings generated from [a-zA-Z], not from
all ASCII characters.

Relates elastic/x-pack-elasticsearch#942

Original commit: elastic/x-pack-elasticsearch@a6085964d3
2017-04-04 11:04:37 -04:00
Yannick Welsch e52fbdf63c Use random http port for watcher REST tests (elastic/x-pack-elasticsearch#936)
Adapts the Watcher REST tests so that they don't require a fixed http port anymore.

Original commit: elastic/x-pack-elasticsearch@14919b16e7
2017-04-04 09:02:23 +02:00
David Kyle 5b66c7a7ba [ML] Set job create time on server (elastic/x-pack-elasticsearch#910)
* [ML] Set job create time on server

* Job.Builder serialisation tests

* Make setCreateTime package private

Original commit: elastic/x-pack-elasticsearch@d2d75e0d7b
2017-04-03 18:30:47 +01:00
Simon Willnauer 5e6bfb9a82 Fix cross cluster search with security (elastic/x-pack-elasticsearch#904)
This commit adds an integration test that runs basic cross cluster search
actions across 2 clusters with security installed. This commit also fixes
several issues with respect to internal actions and proxy actions in the context
of cross cluster search.

Relates to elastic/elasticsearch#23830
relates elastic/x-pack-elasticsearch#892

Original commit: elastic/x-pack-elasticsearch@2e5486c259
2017-04-03 09:53:48 +02:00
Ryan Ernst e1949ee362 Test: Use separate wrapper tasks for tribe clusters setup (elastic/x-pack-elasticsearch#905)
This commit adds separate tasks for tribe clusteres which the
cluster formation tasks build their own tasks off. This ensures each
cluster will have its own wait task, so that the tribe node will be able
to wait on the other clusters being up before even trying to start.

relates elastic/x-pack-elasticsearch#877

Original commit: elastic/x-pack-elasticsearch@1e4c729372
2017-03-31 14:30:44 -07:00
David Roberts 95fd6777a0 [ML] Add 'xpack' into action names to match what the rest of X-Pack does (elastic/x-pack-elasticsearch#909)
Original commit: elastic/x-pack-elasticsearch@36be904ad9
2017-03-31 11:22:53 +01:00
Jason Tedor f52760d2ac Revert "Silence UpgradeClusterClientYamlTestSuiteIT"
This reverts commit elastic/x-pack-elasticsearch@e29f94b24c.

Original commit: elastic/x-pack-elasticsearch@d7d27b20ef
2017-03-30 08:39:10 -04:00
David Roberts 72f5e92972 [ML] Adjust error to make sense for FAILED as well as CLOSED jobs (elastic/x-pack-elasticsearch#888)
Previously trying to perform an operation on a FAILED job would tell
you that it hadn't been opened.  This was very confusing if you knew
very well you'd just opened the job.

Relates elastic/x-pack-elasticsearch#879

Original commit: elastic/x-pack-elasticsearch@2dbe41caaa
2017-03-30 10:33:56 +01:00
David Roberts c335e79508 [TEST] Enable two blacklisted tests that now work
Previously force closing a job required extra privileges.  Following
the full discussion about what privileges should be required.

Original commit: elastic/x-pack-elasticsearch@4d85314b35
2017-03-30 10:26:15 +01:00
Martijn van Groningen 61b4f6480e [TEST] muted test
Original commit: elastic/x-pack-elasticsearch@550b55739e
2017-03-30 11:24:27 +02:00
Martijn van Groningen f2654b5872 [ML] Changed job and datafeed lifecycle management
* Removed OPENING and CLOSING job states. Instead when persistent task has been created and
   status hasn't been set then this means we haven't yet started, when the executor changes it to STARTED we have.
   The coordinating node will monitor cs for a period of time until that happens and then returns or times out.
* Refactored job close api to go to node running job task and close job there.
* Changed unexpected job and datafeed exception messages to not mention the state and instead mention that job/datafeed haven't yet started/stopped.

Original commit: elastic/x-pack-elasticsearch@37e778b585
2017-03-29 20:35:10 +02:00
Daniel Mitterdorfer 2355338625 Silence UpgradeClusterClientYamlTestSuiteIT
Relates elastic/x-pack-elasticsearch#870
Relates elastic/x-pack-kibana#690

Original commit: elastic/x-pack-elasticsearch@e29f94b24c
2017-03-29 10:38:28 +02:00
Dimitrios Athanasiou 991e5d8833 [TEST] Set required euery on delete-by-query request
Original commit: elastic/x-pack-elasticsearch@b9b6234f3d
2017-03-28 17:15:23 +01:00
Colin Goodheart-Smithe 99db6013ea [ML] Adds a test for ml transport client (elastic/x-pack-elasticsearch#851)
This change also adds synchronous methods to `MachineLearningClient`.

relates elastic/x-pack-elasticsearch#567

Original commit: elastic/x-pack-elasticsearch@b3a4b38a51
2017-03-28 09:05:55 +01:00
Martijn van Groningen fc29fd159a [TEST] Don't lose the actual cause of the exception.
Original commit: elastic/x-pack-elasticsearch@8720288ef5
2017-03-27 15:33:04 +02:00
David Roberts ecbfaace38 [TEST] Improve ML security test comments and account for new close assertion
Original commit: elastic/x-pack-elasticsearch@004845bf2f
2017-03-27 10:24:05 +01:00
David Roberts c0445fac4d [TEST] Silence failing ML security tests
Original commit: elastic/x-pack-elasticsearch@af4ed2019d
2017-03-24 17:58:30 +00:00
Colin Goodheart-Smithe caa9afad2b [TEST] fixed content type in ML QA tests
Original commit: elastic/x-pack-elasticsearch@b7749e1319
2017-03-24 14:08:39 +00:00
Hendrik Muhs a8b1b3e863 Revert "disable integration test, see elastic/x-pack-elasticsearch#823 for details (elastic/x-pack-elasticsearch#824)"
This reverts commit elastic/x-pack-elasticsearch@2338b3c972.

Original commit: elastic/x-pack-elasticsearch@a9c7168762
2017-03-23 20:10:58 +01:00
Hendrik Muhs 27dce8669c disable integration test, see elastic/x-pack-elasticsearch#823 for details (elastic/x-pack-elasticsearch#824)
[ML] disable failing integration test, see elastic/x-pack-elasticsearch#823 for details

Original commit: elastic/x-pack-elasticsearch@2338b3c972
2017-03-23 19:54:02 +01:00
Hendrik Muhs 6f7f466fa3 [ML] move DataStreamDiagnostics into DataCountsReporter (elastic/x-pack-elasticsearch#775)
repair DataStreamDiagnostics

Moves DataStreamDiagnostics into DataCountsReporter to survive if job is opened/closed/fed in chunks.

relates elastic/x-pack-elasticsearch#764

Original commit: elastic/x-pack-elasticsearch@29c221a451
2017-03-23 16:43:51 +01:00
Jay Modi 8ba6e8b3eb Fix merging of field level security rules (elastic/x-pack-elasticsearch#796)
This commit fixes the merging of field level security rules from multiple roles. Prior to 5.2, the
merging was treated as the merging of languages whereas after 5.2, this incorrectly became a merge
of all rules meaning a single wildcard could cause denials to be ignored.

Original commit: elastic/x-pack-elasticsearch@42f9e6d8b0
2017-03-22 05:10:30 -07:00
Martijn van Groningen 60c79ccdbb [TEST] Set job close timeout to 20s.
Original commit: elastic/x-pack-elasticsearch@9b92c940cb
2017-03-16 21:57:36 +01:00
Martijn van Groningen 73c5c1576f [TEST] Added logging and removed unneeded jackson-databind test dependency.
Original commit: elastic/x-pack-elasticsearch@58c247d696
2017-03-16 21:17:14 +01:00
Dimitris Athanasiou 16efd4e474 [ML] Use TimeValue in job and datafeed configs (elastic/x-pack-elasticsearch#732)
relates elastic/x-pack-elasticsearch#679

Original commit: elastic/x-pack-elasticsearch@891edd5bfe
2017-03-16 14:07:47 +00:00
Ryan Ernst baaad36c5e Fix tribe test setup to look for a single node
numNodes isn't available because it is not in the normal context of
cluster configuration.

relates elastic/x-pack-elasticsearch#740

Original commit: elastic/x-pack-elasticsearch@336147f20c
2017-03-15 15:55:10 -07:00
Ryan Ernst 534584d525 Tests: Use cluster health api for wait condition in ml integ test
This was forgotten in elastic/x-pack-elasticsearch#740

Original commit: elastic/x-pack-elasticsearch@47c10dc543
2017-03-15 10:27:38 -07:00
Ryan Ernst 8c01d6ea69 Tests: Add cluster health check to xpack integ wait conditions (elastic/x-pack-elasticsearch#740)
The wait condition used for integ tests by default calls the cluster
health api with wait_for_nodes nd wait_for_status. However, xpack
overrides the wait condition to add auth, but most of these conditions
still looked at the root ES url, which means the tests are susceptible
to race conditions with the check and node startup. This change modifies
the url for the authenticated wait condtion to check the health api,
with the appropriate wait_for_nodes and wait_for_status.

Original commit: elastic/x-pack-elasticsearch@0b23ef528f
2017-03-15 10:23:26 -07:00
David Roberts 0b7c735aec [ML] Add machine learning privileges/roles (elastic/x-pack-elasticsearch#673)
* Changed ML action names to allow distinguishing of admin and read-only actions
  using wildcards
* Added manage_ml and monitor_ml built-in privileges as subsets of the existing
  manage and monitor privileges
* Added out-of-the-box machine_learning_admin and machine_learning_user roles
* Changed machine learning results endpoints to use a NodeClient rather than an
  InternalClient when searching for results so that index/document level permissions
  applied to ML results are respected

Original commit: elastic/x-pack-elasticsearch@eee800aaa8
2017-03-14 16:13:41 +00:00
Dimitris Athanasiou a7a36245c9 [ML] Add end-point for deleting expired results (elastic/x-pack-elasticsearch#670)
This commit adds an end-point to force deletion of expired data:

DELETE /_xpack/ml/_delete_expired_data

A few other things are changed too:

- Delete expired results from now rather than start of day
- Rename MlDaily{Management -> Maintenance}Service
- Refresh job indices when job is closing to ensure latest result
  visibility
- Commit results when quantiles are persisted to ensure they are visible
  for renormalization


Original commit: elastic/x-pack-elasticsearch@8ca5272a94
2017-03-03 11:33:22 +00:00
Dimitrios Athanasiou 0542a9eb92 [TEST] Disable ml in qa modules where necessary
Original commit: elastic/x-pack-elasticsearch@bb311b44d7
2017-03-02 17:01:35 +00:00
Martijn van Groningen 948a8594bb [TEST] Disable ml in audit qa test
Original commit: elastic/x-pack-elasticsearch@34b7243f69
2017-03-02 16:02:15 +01:00