Commit Graph

5576 Commits

Author SHA1 Message Date
Dimitrios Athanasiou 1f9ddb6937 [TEST] Increase ml datafeed logging to understand test failure
DatafeedJobsIT has been failing. This change increases logging
to investigate those failures.

Original commit: elastic/x-pack-elasticsearch@0c7c29ac29
2017-04-20 17:18:11 +01:00
Lisa Cawley bf110ba05e [DOCS] Document machine_learning_admin and machine_learning_user roles (elastic/x-pack-elasticsearch#1132)
* [DOCS] Document machine_learning_admin and machine_learning_user roles

* [DOCS] Fix auth requrements for ML result APIs

* [DOCS] Update authorization.asciidoc based on elastic/x-pack-elasticsearch#1132

Original commit: elastic/x-pack-elasticsearch@1bf563e8d7
2017-04-20 08:45:30 -07:00
Alexander Reelsen 50dff91a3a Watcher: Fix resetting of ack status on unmet condition (elastic/x-pack-elasticsearch#1141)
When a condition is unmet, the ack status of the actions needs to be
resetted again, so that new alerts can be triggered.

Due to a bugfix this functionality was removed from ES 5.0.0-alpha5
onwards.

relates elastic/x-pack-elasticsearch#1123

Original commit: elastic/x-pack-elasticsearch@83db2cecf9
2017-04-20 15:19:25 +01:00
Dimitris Athanasiou 13d3b353c6 [TEST] Wait for pending tasks on test clean up... (elastic/x-pack-elasticsearch#1137)
.. in MlNativeAutodetectIntegTestCase.

We wait for pending tasks to finish from REST tests but not
from the ones using a native autodetect. This commit adds the
waiting in those tests too.

Relates elastic/x-pack-elasticsearch#1136

Original commit: elastic/x-pack-elasticsearch@a7a5455c78
2017-04-20 14:03:38 +01:00
Dimitris Athanasiou a2124b68e1 [ML] Invert interim result inclusion strategy (elastic/x-pack-elasticsearch#1139)
This changes the get-result actions to include interim
results by default. The former include_interim param is
now changed into an exclude_interim param.

relates elastic/x-pack-elasticsearch#1091

Original commit: elastic/x-pack-elasticsearch@c55a9a89d7
2017-04-20 13:52:35 +01:00
David Roberts bf543f0735 [TEST] Report more detail if waitForPendingTasks times out
Original commit: elastic/x-pack-elasticsearch@63ca54b851
2017-04-20 13:10:48 +01:00
David Roberts bd58856e6f [ML] Remove obsolete comment
Original commit: elastic/x-pack-elasticsearch@5637f95311
2017-04-20 11:00:27 +01:00
David Roberts caa080ec02 [TEST] Fix exception specification
Caused by a clash between two PRs

Original commit: elastic/x-pack-elasticsearch@47910636eb
2017-04-20 10:04:30 +01:00
David Roberts 99def2bd33 [ML] Increase the wait time for AWS credentials to propagate
This was timing out a lot yesterday from the London office

Original commit: elastic/x-pack-elasticsearch@a0989c2a0f
2017-04-20 09:54:00 +01:00
David Roberts b03147bea9 [TEST] Don't duplicate the MlRestTestStateCleaner class (elastic/x-pack-elasticsearch#1127)
We didn't realise it was possible for a qa module to depend on the
test classes of the plugin module, so we duplicated a test class.
But it turns out it IS possible to declare this dependency and avoid
the duplication.

Original commit: elastic/x-pack-elasticsearch@b6a21cda28
2017-04-20 09:13:04 +01:00
lcawley 7c3a3cce80 [DOCS] Add results_index_name to ML put job API
Original commit: elastic/x-pack-elasticsearch@d319a968f4
2017-04-19 23:02:45 -07:00
lcawley b0bfca6a11 [DOCS] Fix watcher typo
Original commit: elastic/x-pack-elasticsearch@f970206d1d
2017-04-19 15:58:09 -07:00
Lisa Cawley ded8edcc3d [DOCS] Fixes build and typo issues in ML API update elastic/x-pack-elasticsearch#1118
Original commit: elastic/x-pack-elasticsearch@3ef20792ac
2017-04-19 13:31:07 -07:00
Igor Motov 7656e4a67b Persistent tasks: require allocation id on task completion (elastic/x-pack-elasticsearch#1107)
Persistent tasks should verify that completion notification is done for correct version of the task, otherwise a delayed notification from an old node can accidentally close a newly reassigned task.

Original commit: elastic/x-pack-elasticsearch@478bb6e730
2017-04-19 15:42:55 -04:00
Dimitrios Athanasiou a0099cace6 [TEST] Call get on update job action from DetectionRulesIT
This was the reason for the intermittent failures of this test.

Original commit: elastic/x-pack-elasticsearch@e8605590a1
2017-04-19 19:52:16 +01:00
Sophie Chang e2cc00ab8e [DOCS] Review of API docs part 1 (elastic/x-pack-elasticsearch#1118)
* [DOCS] Review of close job and job stats

* [DOCS] Add force close

* [DOCS] Remove invalid params from get records

* [DOCS] Remove invalid params from get buckets

* [DOCS] Job resource corrections

Original commit: elastic/x-pack-elasticsearch@bc68d05097
2017-04-19 11:43:30 -07:00
Martijn van Groningen 7ee48846ec [ML] Auto job close should use the current node as coordinating node.
Original commit: elastic/x-pack-elasticsearch@4f3f8f9915
2017-04-19 20:23:16 +02:00
Shaunak Kashyap 1f75dec642 Adding limitations of Watch Edit page (elastic/x-pack-elasticsearch#1014)
* Adding limitations of Watch Edit page

* Removing image reference for now until image dir referencing is sorted out

* More end-user-friendly phrasing

* Language changes + bringing back image

Original commit: elastic/x-pack-elasticsearch@06bf93ee99
2017-04-19 11:12:54 -07:00
Colin Goodheart-Smithe 7cb2be2e5c Adds a check to wait for active tasks for XPackRestIT (elastic/x-pack-elasticsearch#964)
* Adds a check to wait for active tasks for XPackRestIT

* uses test logger

* Change to use assertBusy instead of awaitBusy

* fixes failures with active tasks remaining

* Moves wait for pending tasks into MlRestTestStateCleaner

* remove unecessary log line

Original commit: elastic/x-pack-elasticsearch@1f098dbb64
2017-04-19 17:36:30 +01:00
Ali Beyad 7def5ac01d [DOCS] Documentation for Custom Roles Providers
relates elastic/x-pack-elasticsearch#721

Original commit: elastic/x-pack-elasticsearch@67fdba706c
2017-04-19 11:10:33 -04:00
Martijn van Groningen 06c4a3223b [ML] Having no datafeed tasks can happen.
Original commit: elastic/x-pack-elasticsearch@b752ce4f0f
2017-04-19 16:49:19 +02:00
David Roberts 4255f6dae6 [ML] Further reduce log file noise (elastic/x-pack-elasticsearch#1126)
Original commit: elastic/x-pack-elasticsearch@65301281d2
2017-04-19 15:19:15 +01:00
Dimitrios Athanasiou 8e7fd1421c [ML] Fix typo: decending -> descending
Original commit: elastic/x-pack-elasticsearch@d0b3a6a4b9
2017-04-19 15:18:01 +01:00
Dimitris Athanasiou 96cfbc8f4c [ML] Remove partition_field param from GetBucketsAction (elastic/x-pack-elasticsearch#1122)
Relates elastic/x-pack-elasticsearch#1119


Original commit: elastic/x-pack-elasticsearch@c87fdbc031
2017-04-19 14:07:51 +01:00
Dimitrios Athanasiou 03170d870b [ML] Remove expand from get-records REST spec
Relates elastic/x-pack-elasticsearch#1119

Original commit: elastic/x-pack-elasticsearch@d034a7ee5d
2017-04-19 13:30:34 +01:00
Dimitrios Athanasiou 02d1350778 [TEST] Change the serialization order back to keep tests happy
Original commit: elastic/x-pack-elasticsearch@2b83697395
2017-04-19 13:27:48 +01:00
Dimitris Athanasiou 9865d5b955 [ML] Fix detector rules and add integ test (elastic/x-pack-elasticsearch#1084)
Relates elastic/x-pack-elasticsearch#882

Original commit: elastic/x-pack-elasticsearch@fd1cc0d402
2017-04-19 12:23:38 +01:00
Shaunak Kashyap 618341db6c Amending docs on security privileges required for Watcher (elastic/x-pack-elasticsearch#1086)
* Amending docs on security privileges required for Watcher

Previously, the watcher_user and watcher_admin roles did not exist so we documented the actual security privileges necessary. Now that these roles exist and encapsulate the security privileges, we update the documentation to refer to the roles instead.

* Breaking up sentences. Putting main content up front.

* Include triggered watches as well

* Emphasize read-only operations

Original commit: elastic/x-pack-elasticsearch@720d84557c
2017-04-19 04:21:43 -07:00
David Kyle d8a70138cd [TEST] Give jobs meaningful names in ML integ tests (elastic/x-pack-elasticsearch#1103)
Original commit: elastic/x-pack-elasticsearch@aaf1807172
2017-04-19 12:00:53 +01:00
Lisa Cawley 5223acdd9f [DOCS] Multiple fixes related to privileges in ML documentation (elastic/x-pack-elasticsearch#1110)
* [DOCS] Add privilege requirements to ML API docs

* [DOCS] Document ML cluster-level privileges

Original commit: elastic/x-pack-elasticsearch@221c67d395
2017-04-18 15:13:21 -07:00
Ryan Ernst b86cdd6c8e Test: Update rest base class parameters signature (elastic/x-pack-elasticsearch#1101)
This is the xpack side of elastic/elasticsearch#21392

Original commit: elastic/x-pack-elasticsearch@b760815f54
2017-04-18 15:07:14 -07:00
Chris Earle b1c0631e46 [TEST] Test explicitly for the watch we're interested in
Original commit: elastic/x-pack-elasticsearch@71bd0aee7e
2017-04-18 17:00:30 -04:00
debadair ac441fab57 [DOCS] Migrating images to separate x-pack repos.
Original commit: elastic/x-pack-elasticsearch@80317c063b
2017-04-18 13:13:12 -07:00
Chris Earle 165291a23c Check for specific watch in watch test
Original commit: elastic/x-pack-elasticsearch@e778f14714
2017-04-18 15:57:52 -04:00
Lisa Cawley 178d259315 [DOCS] Clarify missing_field_count for elastic/x-pack-elasticsearch#987 (elastic/x-pack-elasticsearch#1105)
Original commit: elastic/x-pack-elasticsearch@91e867e6b5
2017-04-18 11:30:30 -07:00
Chris Earle c27bb16141 [Monitoring] Use Exporter to create watches in Monitoring Cluster (elastic/x-pack-elasticsearch#994)
By creating the watches via the exporter, we get to afford ourselves
with a much more automatic and simpler set of security permissions.

This does limit us in a few ways (e.g., every exporter has to deal with
cluster alerts itself, which means that newer releases of Kibana cannot
help by adding newer cluster alerts for older, still-monitored
clusters).

Original commit: elastic/x-pack-elasticsearch@448ef313c3
2017-04-18 12:59:46 -04:00
Chris Earle a11f77aea0 [Monitoring] Convert "" _id to null (elastic/x-pack-elasticsearch#1098)
When Logstash 5.2 - 5.3 submit documents via the `_xpack/monitoring/_bulk`
endpoint, it sends its time-based documents with an explicit `_id` of
`""`.

This used to be automatically ignored by Monitoring, but we now accept the
_id that we are given (including `null`). ES, prior to 5.3.1, accepted
`""` as a valid `_id` through the `_bulk` endpoint, which means that it
blindly accepted and overwrote documents given that ID, meaning that all
Logstash instances "shared" the exact same document and therefore the UI
becomes useless.

This change allows `""` to be used and it simply replaces that value, and
only that value, with `null`. This enables backwards compatibility with LS
5.2 - 5.3.0.

Original commit: elastic/x-pack-elasticsearch@889578e61e
2017-04-18 12:24:15 -04:00
Adrien Grand 98c4dc6a92 Upgrade to a snapshot of Lucene 7. (elastic/x-pack-elasticsearch#1102)
Original commit: elastic/x-pack-elasticsearch@2c145e4160
2017-04-18 15:35:55 +02:00
Colin Goodheart-Smithe 51de15dcf1 Fixes compile errors in Eclipse due to generics
PersistentTasksCustomMetadata was using a generic param named `Params`. This conflicted with the imported interface `ToXContent.Params`. The java compiler was preferring the generic param over the interface so everything was fine but Eclipse apparently prefers the interface int his case which was screwing up the Hierarchy and causing compile errors in Eclipse. This changes fixes it by renaming the Generic param to `P`

Original commit: elastic/x-pack-elasticsearch@8528870684
2017-04-18 12:50:05 +01:00
Tim Vernum 4d557afaa4 Improvements in preparation for multiple security indices (elastic/x-pack-elasticsearch#1074)
- Mark all security indices (that is all indices managed by SecurityLifecycleService) as "superuser only" (only superuser role can have direct permissions)
- Add unit tests for IndexLifecycleManager

Original commit: elastic/x-pack-elasticsearch@e4478825e0
2017-04-18 15:22:19 +10:00
Chris Earle 585f70f641 Add instructions for creating Vault Secret ID / GitHub Token (elastic/x-pack-elasticsearch#761)
The instructions over in elastic/infra are more geared toward the infra team.

Original commit: elastic/x-pack-elasticsearch@1c5125d9bc
2017-04-17 19:30:31 -04:00
Lisa Cawley 18111e8617 [DOCS] First draft of ML getting started tutorial (elastic/x-pack-elasticsearch#1099)
* [DOCS] First draft of ML getting started tutorial

* [DOCS] More ML getting started content

* [DOCS] Getting started content for data feeds

* [DOCS] Added ML getting started screenshot

Original commit: elastic/x-pack-elasticsearch@73174d27e8
2017-04-17 16:28:28 -07:00
Ryan Ernst ef3d3b51a4 Move integ test runner deps to cluster deps (elastic/x-pack-elasticsearch#1096)
This is the xpack side of elastic/elasticsearch#24142

Original commit: elastic/x-pack-elasticsearch@d502f06cea
2017-04-17 16:04:09 -07:00
Lisa Cawley e3ee3fb581 [DOCS] Add ML settings page (elastic/x-pack-elasticsearch#1069)
* [DOCS] Add ML settings page

* [DOCS] Add ML node to introductory concepts

* [DOCS] ML node clarifications for elastic/x-pack-elasticsearch#1069

Original commit: elastic/x-pack-elasticsearch@ebbd3b31c7
2017-04-17 11:53:31 -07:00
Jay Modi b59b6bbdd4 Remove SecuredString and use SecureString from elasticsearch core (elastic/x-pack-elasticsearch#1092)
This commit removes the SecuredString class that was previously used throughout the security code
and replaces it with the SecureString class from core that was added as part of the new secure
settings infrastructure.

relates elastic/x-pack-elasticsearch#421

Original commit: elastic/x-pack-elasticsearch@e9cd117ca1
2017-04-17 13:28:46 -04:00
jaymode 453bf907ca Test: expand wildcards when testing wildcard expression
Original commit: elastic/x-pack-elasticsearch@f5b74383a9
2017-04-14 10:05:25 -04:00
jaymode 492439fc70 Test: fix test bug with incorrect date math expressions
Original commit: elastic/x-pack-elasticsearch@ff88e947cd
2017-04-14 08:19:38 -04:00
Jay Modi 6f352260d6 Fix resolution of indices names with both date math and wildcards (elastic/x-pack-elasticsearch#1067)
When a index name pattern contains both date math and wildcards, the name resolution does not
return the expected result. This change moves the date math resolution to before our attempts to
match wildcards so that both can be used in the same pattern.

relates elastic/x-pack-elasticsearch#1065

Original commit: elastic/x-pack-elasticsearch@9f48b42fad
2017-04-14 07:44:52 -04:00
Martijn van Groningen d114a55b99 [ML] Make open job and start datafeed apis master node actions and
let close job and stop datafeed apis redirect to elected master node.

This is for cluster state observation purposes, so that a subsequent open and then close job or
start and then stop datafeed see the same local cluster state and sanity validation doesn't fail.

Original commit: elastic/x-pack-elasticsearch@21a63184b9
2017-04-14 10:43:21 +02:00
Martijn van Groningen e93b447b9c [ML] Only check licensing in the transport action of the open job and start datafeed apis.
Original commit: elastic/x-pack-elasticsearch@a59ef8884c
2017-04-14 08:57:44 +02:00