Commit Graph

73 Commits

Author SHA1 Message Date
Dimitrios Athanasiou a6c62f0717 [ML] Search model state docs using correct id
Original commit: elastic/x-pack-elasticsearch@70400101ad
2017-02-22 13:54:18 +00:00
David Kyle f06b4fbbaf [ML] remove duplicate code
Original commit: elastic/x-pack-elasticsearch@340480477c
2017-02-22 13:52:38 +00:00
Dimitrios Athanasiou b233c4729d [ML] Correctly fetch persistent tasks in JobUpdateAction
Original commit: elastic/x-pack-elasticsearch@440bfdd06e
2017-02-22 11:26:31 +00:00
David Roberts 614aeeeb53 [ML] Add back SuperCSV license/notice (elastic/x-pack-elasticsearch#618)
These were lost during the migration from prelert-legacy to x-pack

Original commit: elastic/x-pack-elasticsearch@da338d4a44
2017-02-22 10:49:26 +00:00
David Kyle f82634b6b1 [ML] Use ExceptionsHelper.convertToElastic to wrap exceptions in TransportJobTaskAction (elastic/x-pack-elasticsearch#577)
* Wrap exceptions if they are not ElasticsearchExceptions in TransportJobTaskAction

* Review comments

Original commit: elastic/x-pack-elasticsearch@07e900ffaa
2017-02-22 10:03:22 +00:00
polyfractal 45ecebac20 [Watcher] Fix tests due to upstream method signature change
Caused by elastic/elasticsearch#23288

Original commit: elastic/x-pack-elasticsearch@6f041708f9
2017-02-21 13:48:53 -05:00
Martijn van Groningen 9100fa6333 [ML] Rename `xpack.ml.allocation_enabled` boolean setting to `node.ml`.
Remove `node.attr.xpack.ml.allocation_enabled` node attribute and instead use `node.attr.max_running_jobs` attribute to indicate a node is a ml node.
(in addition to indicating how many jobs can hosted on a node)

Original commit: elastic/x-pack-elasticsearch@7979bc55b4
2017-02-21 14:10:50 +01:00
Dimitris Athanasiou 165c0d0e4f [ML] Move metadata package classes at the top (elastic/x-pack-elasticsearch#605)
Original commit: elastic/x-pack-elasticsearch@579f86e7b9
2017-02-21 13:08:16 +00:00
Dimitrios Athanasiou b9f83da5ac [ML] Include all fields in Job copy constructor
Fields were missing from the job's copy constructor.
This was the cause of intermittent test failures.

Original commit: elastic/x-pack-elasticsearch@3cd705ab3c
2017-02-21 12:49:08 +00:00
Martijn van Groningen 0e44683de6 [TEST] Change default close timeout for close job requests in tests.
Original commit: elastic/x-pack-elasticsearch@240bb0561c
2017-02-21 13:07:25 +01:00
Dimitrios Athanasiou c49a17e7ca [ML] Fix model snapshot YAML tests
Original commit: elastic/x-pack-elasticsearch@e29663da35
2017-02-21 11:42:24 +00:00
Dimitris Athanasiou 6c97ac7691 [ML] Remove model snapshot restore_priority and... (elastic/x-pack-elasticsearch#598)
...correctly update the job's model_snapshot_id whenever
a new model snapshot is persisted. Load the snapshot that
corresponds to the job's model_snapshot_id when a job is
opened.

This PR makes a series of changes that completes the change
of having job's active model snapshot persisted on the job.

Original commit: elastic/x-pack-elasticsearch@c2e23fa1ee
2017-02-20 17:45:28 +00:00
Alexander Reelsen b58b92dc79 Merge pull request elastic/x-pack-elasticsearch#591 from spinscale/1702-rest-hijack-delete-by-query
Delete/Update by query was not disabled for the watcher index
on HTTP.

Test suite was renamed from hijack to `block_direct_index_access` in the watcher tests

relates elastic/x-pack-elasticsearch#586

Original commit: elastic/x-pack-elasticsearch@5984ee3624
2017-02-20 13:24:00 +01:00
David Roberts 782f97a60f Relax regex for getting C++ build so it works for developers with uncommitted changes
Original commit: elastic/x-pack-elasticsearch@a6faea0cc6
2017-02-20 11:25:19 +00:00
Alexander Reelsen 7e8b43bfb1 Merge branch 'master' into 1702-rest-hijack-delete-by-query
Original commit: elastic/x-pack-elasticsearch@c206f84300
2017-02-20 12:06:55 +01:00
David Roberts 5233ab9831 Don't try to get the native code version in the transport client (elastic/x-pack-elasticsearch#600)
The native controller process won't be running

Original commit: elastic/x-pack-elasticsearch@d848e86cd5
2017-02-20 10:56:20 +00:00
Alexander Reelsen d2c705206c renamed test suite
Original commit: elastic/x-pack-elasticsearch@784d5ee68a
2017-02-20 11:43:03 +01:00
Martijn van Groningen 4b1ed5b453 [ML] Add a setting (`xpack.ml.node_concurrent_job_allocations`) to control the maximum concurrent job allocations.
A job allocation is either a job task in OPENING state or a job task that has been assigned to an executor node, but the executor node hasn't had the opportunity to set the job task status to OPENING.

In order to keep track of restarted tasks, `allocationIdOnLastStatusUpdate` field was added to `PersistentTaskInProgress` class.
This will allow persistent task implementors to detect whether the executor node has changed or has been unset since the last status update has occured.

Original commit: elastic/x-pack-elasticsearch@b7b85a8274
2017-02-20 11:22:19 +01:00
David Roberts 0d1181eabb [ML] Show C++ process info in X-Pack's info response (elastic/x-pack-elasticsearch#581)
The /_xpack endpoint now has a "native_code_info" value in the feature
info for ml

Original commit: elastic/x-pack-elasticsearch@6b4b408f4a
2017-02-18 20:14:52 +00:00
Martijn van Groningen 8a6cea0350 [ml] For the data, flush and update apis check the job state on the node running the job task instead of the coordinating node.
[test] Also re-enabled multi node qa tests.

Original commit: elastic/x-pack-elasticsearch@efb24131f0
2017-02-18 19:18:29 +01:00
Martijn van Groningen a74d1329ba [test] changed default close timeout
Original commit: elastic/x-pack-elasticsearch@f07cca869f
2017-02-18 15:40:13 +01:00
Martijn van Groningen 31c88ae1e5 [ml] Set job state to failed when fetching required information for opening a job failed.
Relates to elastic/x-pack-elasticsearch#596

Original commit: elastic/x-pack-elasticsearch@6ad699df86
2017-02-18 13:13:57 +01:00
Jay Modi 68324f5a50 Remove usage of deprecated methods that do not accept a XContentType (elastic/x-pack-elasticsearch#549)
This commit removes the usages and definition of deprecated methods that do not accept the XContentType for the source.

Additionally, usages of *Entity classes from the http client library have been changed to always specify the content
type.

Original commit: elastic/x-pack-elasticsearch@29d336a008
2017-02-17 14:45:00 -05:00
Martijn van Groningen 22022ea25f [ml] Fix start datafeed validation
relates elastic/x-pack-elasticsearch#596

Original commit: elastic/x-pack-elasticsearch@103fd7f2b4
2017-02-17 20:14:08 +01:00
David Kyle b9a4a2c621 Revert "Don’t wait on job close (elastic/x-pack-elasticsearch#574)"
This reverts commit elastic/x-pack-elasticsearch@99ed0e0dba and fixes the failing tests

Original commit: elastic/x-pack-elasticsearch@403e38316d
2017-02-17 17:05:41 +00:00
Zachary Tong 3fd1c90707 [ML] Move all Job validation to builder and make Job private (elastic/x-pack-elasticsearch#585)
All validation should be done in the builder and the Job object should just be a dumb, immutable blob.

Original commit: elastic/x-pack-elasticsearch@a78e6edbe9
2017-02-17 11:39:41 -05:00
Dimitrios Athanasiou fa9f4e3bca [ML] Remove leftoever debug statement
Original commit: elastic/x-pack-elasticsearch@a3f529b656
2017-02-17 16:16:48 +00:00
David Kyle 45d228dd41 [ML] Synchronise updates to running C++ process (elastic/x-pack-elasticsearch#557)
* Update running process from cluster state update

* Put a big lock on the UpdateProcessAction

* Protect concurrent update of the process with a semaphore

* Review comments

Original commit: elastic/x-pack-elasticsearch@d08b53247a
2017-02-17 16:00:46 +00:00
Martijn van Groningen 75470b4036 [ML] No need to remove the job task manually, when job task is cancelled the persistent task will be removed too.
Original commit: elastic/x-pack-elasticsearch@e0e3947a53
2017-02-17 14:58:00 +01:00
Alexander Reelsen 7d7da4262f Watcher: Hijack REST delete/udpate by query operations
Delete/Update by query was not disabled for the watcher index
on HTTP.

relates elastic/x-pack-elasticsearch#586

Original commit: elastic/x-pack-elasticsearch@b41b375379
2017-02-17 14:32:31 +01:00
Dimitris Athanasiou d970869e00 [ML] Add builder to JobUpdate (elastic/x-pack-elasticsearch#589)
Original commit: elastic/x-pack-elasticsearch@07a69eb407
2017-02-17 11:45:16 +00:00
David Kyle 48ee5e021d Don’t wait on job close (elastic/x-pack-elasticsearch#574)
Original commit: elastic/x-pack-elasticsearch@99ed0e0dba
2017-02-17 09:25:13 +00:00
Dimitris Athanasiou 4f6146cecc [ML] Improve logging on job closing (elastic/x-pack-elasticsearch#587)
Original commit: elastic/x-pack-elasticsearch@58d814fd71
2017-02-17 09:20:23 +00:00
Igor Motov d733aa684d Make persistent task persist full cluster restart
This commit moves persistent tasks from ClusterState.Custom to MetaData.Custom and adds ability for the task to remain in the metadata after completion.

Original commit: elastic/x-pack-elasticsearch@751b40de2a
2017-02-16 19:35:06 -05:00
Zach f4e2830ab1 [ML] Fix MlJobIT#testCreateJobInSharedIndex: "index_name" is now "results_index_name"
relates elastic/x-pack-elasticsearch#584

Original commit: elastic/x-pack-elasticsearch@fbf9fdd9b0
2017-02-16 18:21:13 -05:00
Igor Motov 11a46dd938 [ML] Suppress MlJobIT.testCreateJobInSharedIndexUpdatesMapping
Original commit: elastic/x-pack-elasticsearch@214ff88ac3
2017-02-16 14:24:32 -05:00
Dimitris Athanasiou 1ae8ed495e [ML] Add keyword mapping to audit message to enable aggs (elastic/x-pack-elasticsearch#583)
The messages stored in the .ml-notifications index include a
message field that is mapped to text. It makes sense to keep
that mapping as it can improve search experience in messages.
However, the ML kibana app needs to aggregate over messages,
thus we also need to provide a keyword mapping.

Original commit: elastic/x-pack-elasticsearch@5a5df53d85
2017-02-16 17:53:26 +00:00
Dimitris Athanasiou 6edc051e8d [ML] Close job after a lookback-only datafeed stops (elastic/x-pack-elasticsearch#580)
Original commit: elastic/x-pack-elasticsearch@febf026ba9
2017-02-16 17:48:57 +00:00
David Kyle 15987f49c0 [ML] Update index mapping for jobs in shared indices (elastic/x-pack-elasticsearch#572)
* Update input field mappings if job results index exists

* Add field names to the index mapping in shared indices

* Throw error if creating job will violate the index.mapping.total_fields.limit

* Review comments

Original commit: elastic/x-pack-elasticsearch@d7712d9263
2017-02-16 17:34:47 +00:00
jaymode 6ec6db8544 Test: handle case where multiple state_uuids are found in monitoring shards docs
Original commit: elastic/x-pack-elasticsearch@8d95776081
2017-02-16 11:34:17 -05:00
David Kyle c5506b7440 Consistent defaults for _open job (elastic/x-pack-elasticsearch#578)
Matches OpenJobAction

Original commit: elastic/x-pack-elasticsearch@24c07913a1
2017-02-16 15:41:20 +00:00
David Kyle 545921698c Update in use messages (elastic/x-pack-elasticsearch#576)
[ML] Update in use messages

Original commit: elastic/x-pack-elasticsearch@629961f1ed
2017-02-16 14:44:52 +00:00
jaymode ff96939c5f Authentication should set the proper version on the stream
This commit ensures that the authentication sets the correct version on the stream when it is serialized over the wire
so that there is not a version mismatch between the authentication and the connection it came from.

Original commit: elastic/x-pack-elasticsearch@267d7068f4
2017-02-16 08:44:29 -05:00
Martijn van Groningen 3572cff0a8 [TEST] slightly changed workaround cleanup for ml integ tests.
Original commit: elastic/x-pack-elasticsearch@3eba1e76ab
2017-02-16 14:28:06 +01:00
Martijn van Groningen 55b2111bdf [TEST] wait number of nodes instead of green
Original commit: elastic/x-pack-elasticsearch@d5e47d4e28
2017-02-16 13:54:30 +01:00
Tim Vernum de10d26bfc [Security] Treat LDAP exceptions as auth failures (elastic/x-pack-elasticsearch#565)
Any failure to communicate with an LDAP server should be treated as if authentication failed rather than a fatal "stop the world" error.

Because the AD realm validates the user's password by attempting to bind to the directory with that username/password combination, connection failures are to be expected and must be handled gracefully because the AD realm may not be the last realm in the order-chain, and any exception would prevent later realms from being tested.

Closes: elastic/x-pack-elasticsearch#564

Original commit: elastic/x-pack-elasticsearch@5282290b00
2017-02-16 22:57:12 +11:00
David Kyle e77a71827d Fix integ test after field rename
Original commit: elastic/x-pack-elasticsearch@4ebb93fb44
2017-02-16 11:19:15 +00:00
David Kyle 26df31a3d0 Rename Job.index_name -> Job.results_index_name (elastic/x-pack-elasticsearch#570)
Original commit: elastic/x-pack-elasticsearch@50fc6515e7
2017-02-16 10:26:38 +00:00
Martijn van Groningen d03c6befd5 [ML] Don't set node attributes when ml is disabled.
Original commit: elastic/x-pack-elasticsearch@d450750f41
2017-02-16 10:52:30 +01:00
Colin Goodheart-Smithe 19fc532961 Restricts certain ml endpoints if license forbids ml (elastic/x-pack-elasticsearch#568)
This change adds checks to the following machine learning actions to check that machine learning is permitted by the license before executing the request and throw and exception if the license forbids it:

* Create job
* Create data feed
* Open job
* Start data feed

This is not a final list of the restrictions we want to have in place if the license forbids ML access but it serves as a starting point and can be easily updated following further discussions.

The change also moves the `transportClientMode` check to `createComponents` so we don’t try to start the server parts of machine learning (job manager, connection to named pipes etc.) if we are running in a transport client.

Original commit: elastic/x-pack-elasticsearch@6c19ebd3bc
2017-02-16 09:40:53 +00:00