Commit Graph

445 Commits

Author SHA1 Message Date
Ali Beyad 4ec819706e [TEST] AwaitsFix annotation on periodically failing ML tests
Relates elastic/x-pack-elasticsearch#1079

Original commit: elastic/x-pack-elasticsearch@bc10e87869
2017-04-13 11:48:33 -04:00
David Kyle 1dd6207ce7 Revert "[TEST] mute failing ML test for now"
This reverts commit elastic/x-pack-elasticsearch@e04c3d0fa4.

Original commit: elastic/x-pack-elasticsearch@5c5240144a
2017-04-13 16:40:46 +01:00
David Kyle 00cfa78056 [TEST] Remove out of date tests. (elastic/x-pack-elasticsearch#1078)
The path has changed so it’s no longer possible to distinguish between data feed and job tasks.
The preceding test get_datafeed provides ample coverage anyway.

Original commit: elastic/x-pack-elasticsearch@780b1beb6b
2017-04-13 16:40:38 +01:00
Ali Beyad 8f61530dba [TEST] mute failing ML test for now
Original commit: elastic/x-pack-elasticsearch@26ca6bd65d
2017-04-13 11:20:40 -04:00
Alexander Reelsen 32a46f0595 Tests: Remove broken tests for now
Original commit: elastic/x-pack-elasticsearch@62fdbfc996
2017-04-13 15:32:19 +01:00
Alexander Reelsen 80e7babe3d Watcher: Provide real watch status on execute watch API (elastic/x-pack-elasticsearch#1076)
When the execute watch API is called without recording the execution
in the watch history, the watch status is not updated, in order to not
divert the in-memory object status and the one persisted on disk.

In order to work around this issue, the execute watch API can simply
clone a new watch status and a new watch, which means the object in
the watch store is never updated. This allows for execution and changing
of the watch status, before it is returned to the client.

relates elastic/x-pack-elasticsearch#889

Original commit: elastic/x-pack-elasticsearch@6a0d9c9a78
2017-04-13 14:56:14 +01:00
Igor Motov 450d47d1f5 Persistent Tasks: force writeable name of params and status to be the same as their task (elastic/x-pack-elasticsearch#1072)
Changes persistent task serialization and forces params and status to have the same writeable name as the task itself.

Original commit: elastic/x-pack-elasticsearch@59cf3dca39
2017-04-13 09:49:55 -04:00
David Kyle 468507e788 Revert "Muted test."
This reverts commit elastic/x-pack-elasticsearch@a4d8a72023.

Original commit: elastic/x-pack-elasticsearch@a2d5100ed2
2017-04-13 14:16:32 +01:00
David Kyle ac0e2a72cc Prevent the stop data feed listener from being called multiple times (elastic/x-pack-elasticsearch#1075)
Original commit: elastic/x-pack-elasticsearch@464d5ae6ba
2017-04-13 14:02:03 +01:00
Martijn van Groningen 911cfc9623 [ML] Made `max_running_jobs` a cluster wide setting and
remove `node.attr.max_running_jobs` node attribute and use `node.attr.ml.enabled` node attribute instead to know whether a node is a ml node or not.

Also renamed `max_running_jobs` setting to `xpack.ml.max_running_jobs`.

Original commit: elastic/x-pack-elasticsearch@798732886b
2017-04-13 15:00:50 +02:00
Dimitrios Athanasiou 2385619158 [TEST] Add missing clean up in UpdateInterimResultsIT
Original commit: elastic/x-pack-elasticsearch@bb1a495c8e
2017-04-13 12:53:03 +01:00
Martijn van Groningen f1b529291a [TEST] Use cluster state api to check whether job state is closed.
Original commit: elastic/x-pack-elasticsearch@27c9aa0c41
2017-04-13 13:39:43 +02:00
Martijn van Groningen 81abd29598 [ML] Empty task response is ok.
Original commit: elastic/x-pack-elasticsearch@9da321c531
2017-04-13 12:12:44 +02:00
Dimitrios Athanasiou 783a552b13 [TEST] Add integ test for updating interim results
Relates elastic/x-pack-elasticsearch#882

Original commit: elastic/x-pack-elasticsearch@e05b566188
2017-04-13 01:10:23 +01:00
Dimitrios Athanasiou 0c23d5174e [TEST] Extract base integ test case for ml with native autodetect
Original commit: elastic/x-pack-elasticsearch@7bf192527e
2017-04-12 22:50:32 +01:00
Martijn van Groningen 5461c87345 [ML] use correct id
Original commit: elastic/x-pack-elasticsearch@2801ef9f35
2017-04-12 20:32:17 +02:00
David Kyle 647c3aef53 [ML] Delete model snapshot and state before results (elastic/x-pack-elasticsearch#1066)
Original commit: elastic/x-pack-elasticsearch@65152e9735
2017-04-12 16:07:09 +01:00
Martijn van Groningen f72967eb7f [TEST] Use task builder instead of creating persistent tasks directly.
Original commit: elastic/x-pack-elasticsearch@f74792b23b
2017-04-12 16:21:13 +02:00
Jay Modi 666e87c29b Resolve paths from the current working directory instead of the config directory (elastic/x-pack-elasticsearch#637)
This commit changes the resolution of the output and input files so that relative paths will be resolved from the
current working directory instead of the x-pack config directory.

relates elastic/x-pack-elasticsearch#621

Original commit: elastic/x-pack-elasticsearch@bbfd83c2d5
2017-04-12 10:18:56 -04:00
Igor Motov 253340a597 Persistent Tasks: PersistentTaskRequest -> PersistTaskParams (elastic/x-pack-elasticsearch#1057)
Removes the last pieces of ActionRequest from PersistentTaskRequest and renames it into PersistTaskParams, which is now just an interface that extends NamedWriteable and ToXContent.

Original commit: elastic/x-pack-elasticsearch@5a298b924f
2017-04-12 09:58:15 -04:00
David Kyle a0dcae97db Add more checks for debugging test failure (elastic/x-pack-elasticsearch#1061)
Original commit: elastic/x-pack-elasticsearch@8dde009549
2017-04-12 12:32:38 +01:00
David Roberts 17d9d6aea3 [ML] Be nicer on platforms that don't have the ML binaries (elastic/x-pack-elasticsearch#1060)
Following this change, if the user runs on a platform that we don't ship
ML binaries for:

* If ML is enabled the node still refuses to start, but clearly says why
* If ML is disabled the node starts up without logging any errors

Original commit: elastic/x-pack-elasticsearch@af4fb8c411
2017-04-12 10:02:26 +01:00
Martijn van Groningen 9d683dfe13 [ML] Use datafeed and job id as task id.
Now that task id are strings instead of longs (elastic/x-pack-elasticsearch#1035), ml can use the job and datafeed as task id.
This removes logic that would otherwise iterate over all tasks and check if the task's request id was equal to the provided id and instead just do lookup in the task map.
Job and datafeed task ids are prefixed with either 'job-' or 'datafeed-', because job and datafeed ids don't have to be unique as they are stored separately from each other.

Original commit: elastic/x-pack-elasticsearch@b48c2b368a
2017-04-12 10:15:18 +02:00
Martijn van Groningen f8f7454130 [TEST] Actually perform the intended close...
Original commit: elastic/x-pack-elasticsearch@1333d78509
2017-04-12 09:23:48 +02:00
Martijn van Groningen 272d1b269e Muted test.
Original commit: elastic/x-pack-elasticsearch@a4d8a72023
2017-04-12 09:08:06 +02:00
Igor Motov 428af93f7b Persistent Tasks: switch from long task ids to string task ids (elastic/x-pack-elasticsearch#1035)
This commit switches from long persistent task ids to caller-supplied string persistent task ids.

Original commit: elastic/x-pack-elasticsearch@2dff985df7
2017-04-11 12:24:54 -04:00
Zachary Tong 6f643ffba5 [ML] If snapshot list is empty, return after signaling failure
Original commit: elastic/x-pack-elasticsearch@a89f88da11
2017-04-11 11:37:25 -04:00
Alexander Reelsen 71852c7215 Security: Add watcher_user and watcher_admin role (elastic/x-pack-elasticsearch#983)
This built-in watcher_admin role is able to execute all watcher actions,
read the watch history indices and read the watches index
index. The watcher_user role allows to GET a watch and to get the stats and thats it.

relates elastic/x-pack-elasticsearch#978

Original commit: elastic/x-pack-elasticsearch@11b33a413b
2017-04-11 16:28:55 +01:00
Dimitrios Athanasiou a5f0a74b67 [ML] Remove unused imports in GetDatafeedsStatsAction
Original commit: elastic/x-pack-elasticsearch@83abaae58e
2017-04-11 16:26:13 +01:00
Ali Beyad c468a4d3a3 [TEST] Security index template/mappings check on tests (elastic/x-pack-elasticsearch#1031)
Extends the security index check that ensure the template/mappings
are up-to-date to other security integration tests that depend on this
check.

relates elastic/x-pack-elasticsearch#794

Original commit: elastic/x-pack-elasticsearch@ec8e5b37bd
2017-04-11 11:22:41 -04:00
Martijn van Groningen 0e508d1bfb [TEST] Adjusted log levels to better track failure elastic/x-pack-elasticsearch#1048
Original commit: elastic/x-pack-elasticsearch@8f8b9fe17e
2017-04-11 16:58:49 +02:00
Dimitris Athanasiou 3fa43093eb [ML] Improve handling of job errors from datafeed (elastic/x-pack-elasticsearch#1049)
- stops the datafeed when post/flush throw a conflict exception.
A conflict exception signifies the job state is not opened, thus
we are better off stopping the datafeed.
- handles flushing the job the same way as posting to the job.

relates elastic/x-pack-elasticsearch#855

Original commit: elastic/x-pack-elasticsearch@49a54912c2
2017-04-11 15:55:05 +01:00
David Roberts 9525cb4784 [ML] Replace UTF-8 infinity symbol with ASCII word (elastic/x-pack-elasticsearch#1041)
Makes the log more readable in editors not set to UTF-8.
Customers may well be in this situation on Linux/Windows.

Original commit: elastic/x-pack-elasticsearch@4e59fc90cf
2017-04-11 15:25:50 +01:00
David Kyle 78a3c32ec4 [ML] Reduce frequency of data counts log message (elastic/x-pack-elasticsearch#1030)
* Reduce data counts log message frequency

* Start logging at 10,000 records

Original commit: elastic/x-pack-elasticsearch@af6c791b6e
2017-04-11 15:12:41 +01:00
David Kyle f11df2c0c7 [ML] Give the jobs in index_layout.yaml more meaningful names (elastic/x-pack-elasticsearch#1047)
For debugging the CI failures

Original commit: elastic/x-pack-elasticsearch@5f6a3d1c6d
2017-04-11 14:58:48 +01:00
Martijn van Groningen e1aac25f2d [TEST] Ensure all nodes indicate job has been stopped
Original commit: elastic/x-pack-elasticsearch@39184b6c69
2017-04-11 15:14:41 +02:00
jaymode cb735b5f69 Test: add missng action cluster:monitor/remote/info
Original commit: elastic/x-pack-elasticsearch@11429d81a7
2017-04-11 09:04:34 -04:00
Tanguy Leroux 1243c64671 [TEST] Remove log traces and change how LocalExporterTests stops
The commit changes how LocalExporterTests stops: it now uses the
node_stats document collected on each node and check if it's older
than a given number of seconds (10). It also removes log traces.

Original commit: elastic/x-pack-elasticsearch@0384690b41
2017-04-11 14:50:44 +02:00
Dimitris Athanasiou 4da1c5b9dc [ML] Remove batch_span and period from analysis config (elastic/x-pack-elasticsearch#1043)
relates elastic/x-pack-elasticsearch#1040

Original commit: elastic/x-pack-elasticsearch@aed5e9912c
2017-04-11 12:57:58 +01:00
Hendrik Muhs 7ef9a16f45 [ML] implement '_all' for stopping datafeeds (elastic/x-pack-elasticsearch#995)
Add a '_all' functionality for stopping ML datafeeds.

For cluster shutdown due to maintenance and major upgrades we recommend the user to stop all datafeeds and jobs. This change add the ability to stop all datafeeds at once where previously it was required to iterate over all feeds and do a explicit stop. This is part two of elastic/x-pack-elasticsearch#795, part one can be found in elastic/x-pack-elasticsearch#962 .

relates elastic/x-pack-elasticsearch#795

Original commit: elastic/x-pack-elasticsearch@ed1eff83d5
2017-04-11 13:39:22 +02:00
David Roberts d2c90e798f [ML] Make open/close job timeouts consistent (elastic/x-pack-elasticsearch#1039)
Before this change the persistent task operations related to opening
and closing jobs would time out a long time before the operations
related to native processes.

Original commit: elastic/x-pack-elasticsearch@23076b773b
2017-04-11 11:45:48 +01:00
Dimitrios Athanasiou 1760677b0e [ML] Remove redundant logging
Relates elastic/x-pack-elasticsearch#1027

Original commit: elastic/x-pack-elasticsearch@ddbe390e6f
2017-04-11 10:44:00 +01:00
Simon Willnauer 617c3ead5c Add tests for _remote/info API (elastic/x-pack-elasticsearch#1009)
Relates to elastic/elasticsearch#23925
Depends on elastic/elasticsearch#23969

Original commit: elastic/x-pack-elasticsearch@d1e8754a57
2017-04-11 11:24:22 +02:00
Colin Goodheart-Smithe 29c64d83e1 Removes version 2.x constants from Version (elastic/x-pack-elasticsearch#1022)
Original commit: elastic/x-pack-elasticsearch@39ef909aec
2017-04-11 08:30:59 +01:00
Tim Vernum 8840042751 Only log LDAP errors if authentication failed overall (elastic/x-pack-elasticsearch#960)
Changes the logging of LDAP authentication failures from "always" to "only if the user failed to be authenticated"

Previously there were cases (such has having 2 AD realms) where successful user authentication would still cause an INFO message to be written to the log for every request. 
Now that message is suppressed, but a WARN message is added _if-and-only-if_ the user cannot be authenticated by any realm.

This is implemented via a new value stored in the ThreadContext that the AuthenticationService choses to log (or not log) depending on the result of the authenticate process.
  
Closes: elastic/x-pack-elasticsearch#887

Original commit: elastic/x-pack-elasticsearch@b81b363729
2017-04-11 09:37:21 +10:00
Zachary Tong 43f1fb2bb1 [ML] Correctly parse stream of SMILE documents in JsonDataToProcessWriter (elastic/x-pack-elasticsearch#982)
The PR detects if SMILE is being provided, then correctly slices the stream such that each document is parsed individually. This is required because jackson's SMILE parser is stricter than it's JSON parser and will stop parsing when it hits a streamSeparator (unlike JSON, which will eagerly try to find more objects to parse).

Removes the forced-headers from the various REST tests.

relates elastic/x-pack-elasticsearch#642 

Original commit: elastic/x-pack-elasticsearch@c0e97cd545
2017-04-10 17:55:15 -04:00
Igor Motov 49223a8782 Persistent Tasks: remove listener from PersistentTasksExecutor#nodeOperation (elastic/x-pack-elasticsearch#1032)
Instead of having a separate listener for indicating that the current task is finished, this commit is switching to use allocated object itself.

Original commit: elastic/x-pack-elasticsearch@7ad5362121
2017-04-10 17:32:30 -04:00
Martijn van Groningen 018a3d197d [ML] Index job and datafeed assignment notifications via a separate cluster state listener.
`PersistentTasksExecutor#getAssignment(...)` should be a cheap and side-effect free method,
but in case of `OpenJobPersistentTasksExecutor` and `StartDatafeedPersistentTasksExecutor` before this change it would index a document each time `getAssignment(...)` was invoked

Original commit: elastic/x-pack-elasticsearch@5ca5890baf
2017-04-10 20:20:48 +02:00
Martijn van Groningen ecc19d9948 [TEST] Fixed typos in @TestLogging annotations
Original commit: elastic/x-pack-elasticsearch@3152765d4c
2017-04-10 20:13:50 +02:00
Martijn van Groningen 9201a5d640 [TEST] Changed test log levels
Original commit: elastic/x-pack-elasticsearch@73bd1251fa
2017-04-10 19:48:56 +02:00