Commit Graph

98 Commits

Author SHA1 Message Date
Igor Motov db88cc458d Add persistent task assignment explanations.
This commit allows persistent actions to indicate why a task was or wasn't assigned to a certain node.

Original commit: elastic/x-pack-elasticsearch@cdacf9b10f
2017-02-23 11:22:08 -05:00
Martijn van Groningen d87926ab86 [ML] Never stop the datafeeder when no data can be found
Changed `ProblemTracker#updateEmptyDataCount(boolean)` into `ProblemTracker#reportEmptyDataCount()`.
and added `ProblemTracker#reportNoneEmptyCount()` to reset the empty count counter.

Original commit: elastic/x-pack-elasticsearch@c1a44d6fd3
2017-02-23 16:32:08 +01:00
Martijn van Groningen e7d56e92f8 [Ml] Rename `useNativeProcess` setting to `xpack.ml.autodetect_process` and un-deprecate it.
Original commit: elastic/x-pack-elasticsearch@eb2143e6f0
2017-02-23 15:29:44 +01:00
Dimitris Athanasiou 412d5e35b1 [ML] model_debug_config: add enabled and remove bounds_percentile (elastic/x-pack-elasticsearch#627)
This allows it to be disabled/enabled via job updates.
It also simplifies it by removing bounds_percentile as it currently
rarely makes sense to set the bounds to another value than 95.0.

Original commit: elastic/x-pack-elasticsearch@c27fce2d86
2017-02-23 14:11:23 +00:00
David Roberts 33b10970ee [ML] Temporary workaround to make some non-ml integration tests work (elastic/x-pack-elasticsearch#630)
This will be replaced with a better solution when elastic/x-pack-elasticsearch#566 is implemented

Original commit: elastic/x-pack-elasticsearch@bf0d9e0176
2017-02-23 11:13:58 +00:00
Dimitris Athanasiou d3d73ca095 [ML] Remove data that expired their respective retention period (elastic/x-pack-elasticsearch#610)
This PR introduces a service component that is scheduled to run
once per day, around 30 minutes past midnight, local time.
Listeners can be subscribed to the service in order to get notified
when the next service trigger occurs so that they can execute
management tasks.

Currently, there are 2 such listeners: one is performing removal
of results and the other of model snapshots that have expired
their retention period.

Finally, note that the service is only installed on the master
node to ensure there is only one running on the cluster.

Original commit: elastic/x-pack-elasticsearch@c8edb6016a
2017-02-23 11:06:14 +00:00
Martijn van Groningen a03e9d36c2 [TEST] mute test
Original commit: elastic/x-pack-elasticsearch@3989e5a074
2017-02-23 11:38:44 +01:00
Martijn van Groningen b40d5dde9b [ML] Before selecting an executor node for running job task, make sure that all required indices exist and are accessible.
Original commit: elastic/x-pack-elasticsearch@5459a0d5f2
2017-02-23 10:55:16 +01:00
Alexander Reelsen 43a09a2fa7 Watcher: Add license files (elastic/x-pack-elasticsearch#619)
Add license, notice and SHA sums for watcher compile dependencies
(excluding test deps).

Relates elastic/x-pack-elasticsearch#607

Original commit: elastic/x-pack-elasticsearch@06ce510436
2017-02-23 10:24:16 +01:00
Martijn van Groningen 2eba41cfac [TEST] Fix mocking bug
Original commit: elastic/x-pack-elasticsearch@61268c1180
2017-02-23 09:59:45 +01:00
Martijn van Groningen 6195000b8a [TEST] Change ml cleanup workaround.
Original commit: elastic/x-pack-elasticsearch@793d6cfb78
2017-02-23 09:08:10 +01:00
Tim Vernum 9f6285f362 [TEST] Waiting longer for native realm migration
Let LicensingTests.testNativeRealmMigratorWorksUnderBasicLicense wait for 3s (instead of 1s) as it is occasionally failing.
Also added logging so that if/when it does fail we can see how long we were waiting and what was taking place.

Original commit: elastic/x-pack-elasticsearch@22882ec412
2017-02-23 14:15:17 +11:00
Tim Vernum 672819f038 LICENSE files for security (elastic/x-pack-elasticsearch#614)
* License information for unboundid-ldapsdk
   From: https://docs.ldap.com/ldap-sdk/docs/LICENSE-UnboundID-LDAPSDK.txt

* License information for Bouncy Castle
   From:  java -cp ~/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.55/bcprov-jdk15on-1.55.jar org.bouncycastle.LICENSE


Original commit: elastic/x-pack-elasticsearch@b32d5aade3
2017-02-23 10:43:34 +11:00
Ryan Ernst bca7d87ac4 Rename threadpool time method
See elastic/elasticsearch#23175

Original commit: elastic/x-pack-elasticsearch@0e185c0a44
2017-02-22 15:11:42 -08:00
Dimitris Athanasiou eda2f15b44 [ML] Remove deprecated use of TransportDeleteAction (elastic/x-pack-elasticsearch#628)
Original commit: elastic/x-pack-elasticsearch@b5fdb32a0c
2017-02-22 22:14:32 +00:00
Martijn van Groningen 783a2cf01b [TEST] Added more test logging
Original commit: elastic/x-pack-elasticsearch@d68a1e3510
2017-02-22 21:47:01 +01:00
Ryan Ernst 2cd3979654 Merge pull request elastic/x-pack-elasticsearch#617 from rjernst/integ_test_order
Build: Convert integ test dsl to new split cluster/runner dsl

Original commit: elastic/x-pack-elasticsearch@454edd48d1
2017-02-22 12:43:25 -08:00
Martijn van Groningen 124fe0ad04 [ML] Fix line length
Original commit: elastic/x-pack-elasticsearch@2fba2434b8
2017-02-22 21:37:25 +01:00
polyfractal 4041545cfa [ML] Put back check for inconsistent job id
Lost during the refactor, this checks to make sure an ID in both the URL and the
body match.

Original commit: elastic/x-pack-elasticsearch@ef96275c99
2017-02-22 15:21:03 -05:00
Martijn van Groningen 9da18a14c5 [ML] Set createTime if not specified in validate api too
Original commit: elastic/x-pack-elasticsearch@477fb0fc52
2017-02-22 21:03:55 +01:00
polyfractal f856a79ef3 Fix method signature in test
Original commit: elastic/x-pack-elasticsearch@4340952f77
2017-02-22 14:08:19 -05:00
Zachary Tong 41811efcf1 [ML] Job's CreateTime must be non-null (elastic/x-pack-elasticsearch#616)
Also simplifies Job.Builder to only have one build() method.

Original commit: elastic/x-pack-elasticsearch@50e4b4c210
2017-02-22 13:45:13 -05:00
Martijn van Groningen 9a5b410110 [TEST] Add test that verifies that machine learning works correctly after full cluster restart.
Added missing named xcontent entries, tweaked start datafeed validation logic and fixed status serialization issue that were uncovered by this test.

Original commit: elastic/x-pack-elasticsearch@8b3fb012df
2017-02-22 17:25:08 +01:00
Dimitrios Athanasiou 4bb57137a9 fix unit test
Original commit: elastic/x-pack-elasticsearch@af9276f02f
2017-02-22 14:42:39 +00:00
Dimitrios Athanasiou a6c62f0717 [ML] Search model state docs using correct id
Original commit: elastic/x-pack-elasticsearch@70400101ad
2017-02-22 13:54:18 +00:00
David Kyle f06b4fbbaf [ML] remove duplicate code
Original commit: elastic/x-pack-elasticsearch@340480477c
2017-02-22 13:52:38 +00:00
Dimitrios Athanasiou b233c4729d [ML] Correctly fetch persistent tasks in JobUpdateAction
Original commit: elastic/x-pack-elasticsearch@440bfdd06e
2017-02-22 11:26:31 +00:00
David Roberts 614aeeeb53 [ML] Add back SuperCSV license/notice (elastic/x-pack-elasticsearch#618)
These were lost during the migration from prelert-legacy to x-pack

Original commit: elastic/x-pack-elasticsearch@da338d4a44
2017-02-22 10:49:26 +00:00
David Kyle f82634b6b1 [ML] Use ExceptionsHelper.convertToElastic to wrap exceptions in TransportJobTaskAction (elastic/x-pack-elasticsearch#577)
* Wrap exceptions if they are not ElasticsearchExceptions in TransportJobTaskAction

* Review comments

Original commit: elastic/x-pack-elasticsearch@07e900ffaa
2017-02-22 10:03:22 +00:00
Ryan Ernst 8527bc2415 Build: Convert integ test dsl to new split cluster/runner dsl
This is the xpack side of elastic/elasticsearch#23304

Original commit: elastic/x-pack-elasticsearch@8eddd7fb0d
2017-02-22 00:56:52 -08:00
polyfractal 45ecebac20 [Watcher] Fix tests due to upstream method signature change
Caused by elastic/elasticsearch#23288

Original commit: elastic/x-pack-elasticsearch@6f041708f9
2017-02-21 13:48:53 -05:00
Martijn van Groningen 9100fa6333 [ML] Rename `xpack.ml.allocation_enabled` boolean setting to `node.ml`.
Remove `node.attr.xpack.ml.allocation_enabled` node attribute and instead use `node.attr.max_running_jobs` attribute to indicate a node is a ml node.
(in addition to indicating how many jobs can hosted on a node)

Original commit: elastic/x-pack-elasticsearch@7979bc55b4
2017-02-21 14:10:50 +01:00
Dimitris Athanasiou 165c0d0e4f [ML] Move metadata package classes at the top (elastic/x-pack-elasticsearch#605)
Original commit: elastic/x-pack-elasticsearch@579f86e7b9
2017-02-21 13:08:16 +00:00
Dimitrios Athanasiou b9f83da5ac [ML] Include all fields in Job copy constructor
Fields were missing from the job's copy constructor.
This was the cause of intermittent test failures.

Original commit: elastic/x-pack-elasticsearch@3cd705ab3c
2017-02-21 12:49:08 +00:00
Martijn van Groningen 0e44683de6 [TEST] Change default close timeout for close job requests in tests.
Original commit: elastic/x-pack-elasticsearch@240bb0561c
2017-02-21 13:07:25 +01:00
Dimitrios Athanasiou c49a17e7ca [ML] Fix model snapshot YAML tests
Original commit: elastic/x-pack-elasticsearch@e29663da35
2017-02-21 11:42:24 +00:00
Dimitris Athanasiou 6c97ac7691 [ML] Remove model snapshot restore_priority and... (elastic/x-pack-elasticsearch#598)
...correctly update the job's model_snapshot_id whenever
a new model snapshot is persisted. Load the snapshot that
corresponds to the job's model_snapshot_id when a job is
opened.

This PR makes a series of changes that completes the change
of having job's active model snapshot persisted on the job.

Original commit: elastic/x-pack-elasticsearch@c2e23fa1ee
2017-02-20 17:45:28 +00:00
Alexander Reelsen b58b92dc79 Merge pull request elastic/x-pack-elasticsearch#591 from spinscale/1702-rest-hijack-delete-by-query
Delete/Update by query was not disabled for the watcher index
on HTTP.

Test suite was renamed from hijack to `block_direct_index_access` in the watcher tests

relates elastic/x-pack-elasticsearch#586

Original commit: elastic/x-pack-elasticsearch@5984ee3624
2017-02-20 13:24:00 +01:00
David Roberts 782f97a60f Relax regex for getting C++ build so it works for developers with uncommitted changes
Original commit: elastic/x-pack-elasticsearch@a6faea0cc6
2017-02-20 11:25:19 +00:00
Alexander Reelsen 7e8b43bfb1 Merge branch 'master' into 1702-rest-hijack-delete-by-query
Original commit: elastic/x-pack-elasticsearch@c206f84300
2017-02-20 12:06:55 +01:00
David Roberts 5233ab9831 Don't try to get the native code version in the transport client (elastic/x-pack-elasticsearch#600)
The native controller process won't be running

Original commit: elastic/x-pack-elasticsearch@d848e86cd5
2017-02-20 10:56:20 +00:00
Alexander Reelsen d2c705206c renamed test suite
Original commit: elastic/x-pack-elasticsearch@784d5ee68a
2017-02-20 11:43:03 +01:00
Martijn van Groningen 4b1ed5b453 [ML] Add a setting (`xpack.ml.node_concurrent_job_allocations`) to control the maximum concurrent job allocations.
A job allocation is either a job task in OPENING state or a job task that has been assigned to an executor node, but the executor node hasn't had the opportunity to set the job task status to OPENING.

In order to keep track of restarted tasks, `allocationIdOnLastStatusUpdate` field was added to `PersistentTaskInProgress` class.
This will allow persistent task implementors to detect whether the executor node has changed or has been unset since the last status update has occured.

Original commit: elastic/x-pack-elasticsearch@b7b85a8274
2017-02-20 11:22:19 +01:00
David Roberts 0d1181eabb [ML] Show C++ process info in X-Pack's info response (elastic/x-pack-elasticsearch#581)
The /_xpack endpoint now has a "native_code_info" value in the feature
info for ml

Original commit: elastic/x-pack-elasticsearch@6b4b408f4a
2017-02-18 20:14:52 +00:00
Martijn van Groningen 8a6cea0350 [ml] For the data, flush and update apis check the job state on the node running the job task instead of the coordinating node.
[test] Also re-enabled multi node qa tests.

Original commit: elastic/x-pack-elasticsearch@efb24131f0
2017-02-18 19:18:29 +01:00
Martijn van Groningen a74d1329ba [test] changed default close timeout
Original commit: elastic/x-pack-elasticsearch@f07cca869f
2017-02-18 15:40:13 +01:00
Martijn van Groningen 31c88ae1e5 [ml] Set job state to failed when fetching required information for opening a job failed.
Relates to elastic/x-pack-elasticsearch#596

Original commit: elastic/x-pack-elasticsearch@6ad699df86
2017-02-18 13:13:57 +01:00
Jay Modi 68324f5a50 Remove usage of deprecated methods that do not accept a XContentType (elastic/x-pack-elasticsearch#549)
This commit removes the usages and definition of deprecated methods that do not accept the XContentType for the source.

Additionally, usages of *Entity classes from the http client library have been changed to always specify the content
type.

Original commit: elastic/x-pack-elasticsearch@29d336a008
2017-02-17 14:45:00 -05:00
Martijn van Groningen 22022ea25f [ml] Fix start datafeed validation
relates elastic/x-pack-elasticsearch#596

Original commit: elastic/x-pack-elasticsearch@103fd7f2b4
2017-02-17 20:14:08 +01:00
David Kyle b9a4a2c621 Revert "Don’t wait on job close (elastic/x-pack-elasticsearch#574)"
This reverts commit elastic/x-pack-elasticsearch@99ed0e0dba and fixes the failing tests

Original commit: elastic/x-pack-elasticsearch@403e38316d
2017-02-17 17:05:41 +00:00