Commit Graph

4922 Commits

Author SHA1 Message Date
Martijn van Groningen 22022ea25f [ml] Fix start datafeed validation
relates elastic/x-pack-elasticsearch#596

Original commit: elastic/x-pack-elasticsearch@103fd7f2b4
2017-02-17 20:14:08 +01:00
Martijn van Groningen 92a4b97f6d [TEST] Add simple test, so that module has at least one working test.
Original commit: elastic/x-pack-elasticsearch@72441418ac
2017-02-17 19:44:04 +01:00
David Kyle d707a1c977 Mark class with awaitsfix
Is the test failing because all the methods are marked awaitsfix?

Original commit: elastic/x-pack-elasticsearch@de93bb83b7
2017-02-17 18:06:42 +00:00
David Kyle b9a4a2c621 Revert "Don’t wait on job close (elastic/x-pack-elasticsearch#574)"
This reverts commit elastic/x-pack-elasticsearch@99ed0e0dba and fixes the failing tests

Original commit: elastic/x-pack-elasticsearch@403e38316d
2017-02-17 17:05:41 +00:00
Zachary Tong 3fd1c90707 [ML] Move all Job validation to builder and make Job private (elastic/x-pack-elasticsearch#585)
All validation should be done in the builder and the Job object should just be a dumb, immutable blob.

Original commit: elastic/x-pack-elasticsearch@a78e6edbe9
2017-02-17 11:39:41 -05:00
Dimitrios Athanasiou fa9f4e3bca [ML] Remove leftoever debug statement
Original commit: elastic/x-pack-elasticsearch@a3f529b656
2017-02-17 16:16:48 +00:00
David Kyle 45d228dd41 [ML] Synchronise updates to running C++ process (elastic/x-pack-elasticsearch#557)
* Update running process from cluster state update

* Put a big lock on the UpdateProcessAction

* Protect concurrent update of the process with a semaphore

* Review comments

Original commit: elastic/x-pack-elasticsearch@d08b53247a
2017-02-17 16:00:46 +00:00
David Kyle 1a4b87454c Add AwaitsFix for elastic/x-pack-elasticsearch#592
Original commit: elastic/x-pack-elasticsearch@c5d2650acf
2017-02-17 14:05:25 +00:00
Martijn van Groningen 75470b4036 [ML] No need to remove the job task manually, when job task is cancelled the persistent task will be removed too.
Original commit: elastic/x-pack-elasticsearch@e0e3947a53
2017-02-17 14:58:00 +01:00
Martijn van Groningen 6739f84efb [test] change id between tests, to make reading log files easier.
Original commit: elastic/x-pack-elasticsearch@e633d7098e
2017-02-17 14:55:14 +01:00
Dimitris Athanasiou d970869e00 [ML] Add builder to JobUpdate (elastic/x-pack-elasticsearch#589)
Original commit: elastic/x-pack-elasticsearch@07a69eb407
2017-02-17 11:45:16 +00:00
David Kyle 48ee5e021d Don’t wait on job close (elastic/x-pack-elasticsearch#574)
Original commit: elastic/x-pack-elasticsearch@99ed0e0dba
2017-02-17 09:25:13 +00:00
Dimitris Athanasiou 4f6146cecc [ML] Improve logging on job closing (elastic/x-pack-elasticsearch#587)
Original commit: elastic/x-pack-elasticsearch@58d814fd71
2017-02-17 09:20:23 +00:00
Igor Motov d733aa684d Make persistent task persist full cluster restart
This commit moves persistent tasks from ClusterState.Custom to MetaData.Custom and adds ability for the task to remain in the metadata after completion.

Original commit: elastic/x-pack-elasticsearch@751b40de2a
2017-02-16 19:35:06 -05:00
Zach f4e2830ab1 [ML] Fix MlJobIT#testCreateJobInSharedIndex: "index_name" is now "results_index_name"
relates elastic/x-pack-elasticsearch#584

Original commit: elastic/x-pack-elasticsearch@fbf9fdd9b0
2017-02-16 18:21:13 -05:00
Igor Motov 11a46dd938 [ML] Suppress MlJobIT.testCreateJobInSharedIndexUpdatesMapping
Original commit: elastic/x-pack-elasticsearch@214ff88ac3
2017-02-16 14:24:32 -05:00
Dimitris Athanasiou 1ae8ed495e [ML] Add keyword mapping to audit message to enable aggs (elastic/x-pack-elasticsearch#583)
The messages stored in the .ml-notifications index include a
message field that is mapped to text. It makes sense to keep
that mapping as it can improve search experience in messages.
However, the ML kibana app needs to aggregate over messages,
thus we also need to provide a keyword mapping.

Original commit: elastic/x-pack-elasticsearch@5a5df53d85
2017-02-16 17:53:26 +00:00
Dimitris Athanasiou 6edc051e8d [ML] Close job after a lookback-only datafeed stops (elastic/x-pack-elasticsearch#580)
Original commit: elastic/x-pack-elasticsearch@febf026ba9
2017-02-16 17:48:57 +00:00
David Kyle 15987f49c0 [ML] Update index mapping for jobs in shared indices (elastic/x-pack-elasticsearch#572)
* Update input field mappings if job results index exists

* Add field names to the index mapping in shared indices

* Throw error if creating job will violate the index.mapping.total_fields.limit

* Review comments

Original commit: elastic/x-pack-elasticsearch@d7712d9263
2017-02-16 17:34:47 +00:00
jaymode 6ec6db8544 Test: handle case where multiple state_uuids are found in monitoring shards docs
Original commit: elastic/x-pack-elasticsearch@8d95776081
2017-02-16 11:34:17 -05:00
David Kyle c5506b7440 Consistent defaults for _open job (elastic/x-pack-elasticsearch#578)
Matches OpenJobAction

Original commit: elastic/x-pack-elasticsearch@24c07913a1
2017-02-16 15:41:20 +00:00
David Kyle 545921698c Update in use messages (elastic/x-pack-elasticsearch#576)
[ML] Update in use messages

Original commit: elastic/x-pack-elasticsearch@629961f1ed
2017-02-16 14:44:52 +00:00
jaymode ff96939c5f Authentication should set the proper version on the stream
This commit ensures that the authentication sets the correct version on the stream when it is serialized over the wire
so that there is not a version mismatch between the authentication and the connection it came from.

Original commit: elastic/x-pack-elasticsearch@267d7068f4
2017-02-16 08:44:29 -05:00
Martijn van Groningen 3572cff0a8 [TEST] slightly changed workaround cleanup for ml integ tests.
Original commit: elastic/x-pack-elasticsearch@3eba1e76ab
2017-02-16 14:28:06 +01:00
Martijn van Groningen 55b2111bdf [TEST] wait number of nodes instead of green
Original commit: elastic/x-pack-elasticsearch@d5e47d4e28
2017-02-16 13:54:30 +01:00
Tim Vernum de10d26bfc [Security] Treat LDAP exceptions as auth failures (elastic/x-pack-elasticsearch#565)
Any failure to communicate with an LDAP server should be treated as if authentication failed rather than a fatal "stop the world" error.

Because the AD realm validates the user's password by attempting to bind to the directory with that username/password combination, connection failures are to be expected and must be handled gracefully because the AD realm may not be the last realm in the order-chain, and any exception would prevent later realms from being tested.

Closes: elastic/x-pack-elasticsearch#564

Original commit: elastic/x-pack-elasticsearch@5282290b00
2017-02-16 22:57:12 +11:00
David Kyle e77a71827d Fix integ test after field rename
Original commit: elastic/x-pack-elasticsearch@4ebb93fb44
2017-02-16 11:19:15 +00:00
David Kyle 26df31a3d0 Rename Job.index_name -> Job.results_index_name (elastic/x-pack-elasticsearch#570)
Original commit: elastic/x-pack-elasticsearch@50fc6515e7
2017-02-16 10:26:38 +00:00
Martijn van Groningen d03c6befd5 [ML] Don't set node attributes when ml is disabled.
Original commit: elastic/x-pack-elasticsearch@d450750f41
2017-02-16 10:52:30 +01:00
Colin Goodheart-Smithe 19fc532961 Restricts certain ml endpoints if license forbids ml (elastic/x-pack-elasticsearch#568)
This change adds checks to the following machine learning actions to check that machine learning is permitted by the license before executing the request and throw and exception if the license forbids it:

* Create job
* Create data feed
* Open job
* Start data feed

This is not a final list of the restrictions we want to have in place if the license forbids ML access but it serves as a starting point and can be easily updated following further discussions.

The change also moves the `transportClientMode` check to `createComponents` so we don’t try to start the server parts of machine learning (job manager, connection to named pipes etc.) if we are running in a transport client.

Original commit: elastic/x-pack-elasticsearch@6c19ebd3bc
2017-02-16 09:40:53 +00:00
Martijn van Groningen 54d57f6398 [ml] When trying to find a node to open a job take `max_running_jobs` into account instead of just trying and then failing.
Also before even creating a persistent task for the job, check if there ml nodes available to run the job, otherwise fail quickly.

relates elastic/x-pack-elasticsearch#545

Original commit: elastic/x-pack-elasticsearch@63fd72e56a
2017-02-16 09:58:44 +01:00
Zach e441e0dc4e [ML] Fix order that DBQ is constructed
Also use Job.ID constant instead of string literal.

Kudos to @dimitris-athanasiou for this one :)

Original commit: elastic/x-pack-elasticsearch@906b080110
2017-02-15 15:53:45 -05:00
Martijn van Groningen 6d79210f79 [ml] Registered named writable entry for DatafeedState.
Also added a test to multi node qa module that tests the datafeeder, which should have caught this.

Original commit: elastic/x-pack-elasticsearch@89e4875f6c
2017-02-15 21:38:19 +01:00
jaymode c8b5be186d remove signing of authentication information
Now that TLS is required for node to node communication, we no longer need to sign the
authentication information to prevent tampering.

Original commit: elastic/x-pack-elasticsearch@1f86cf2395
2017-02-15 12:55:24 -05:00
Ali Beyad b3d72af644 [TEST] bumps rolling upgrade bwc version to 5.4.0-snapshot
Original commit: elastic/x-pack-elasticsearch@2f321682d2
2017-02-15 11:00:11 -05:00
Zachary Tong 65cd119733 [ML] Fix injection of DomainSplit, and only inject for Painless
Was accidentally injecting the script object, not the string version of the code.  Also
added a check so we only inject for Painless scripts (and not groovy, etc).

Minor style tweaks too.

Original commit: elastic/x-pack-elasticsearch@58c7275bd8
2017-02-15 09:31:16 -05:00
Jim Ferenczi 87563e96f2 Fix default path for bwc index
Original commit: elastic/x-pack-elasticsearch@72bcf992f4
2017-02-15 12:44:15 +01:00
Brandon Kobel e39ab93a3d Fixing the x-pack path (elastic/x-pack-elasticsearch#562)
Original commit: elastic/x-pack-elasticsearch@dc311d4a78
2017-02-14 17:16:18 -05:00
Martijn van Groningen 76ea6444cf [test] if enabled always specify setting other randomly specify setting to false.
Original commit: elastic/x-pack-elasticsearch@eacdb979e2
2017-02-14 22:28:43 +01:00
Zachary Tong cb29cbd8a9 [ML] Allow result indices to be shared. (elastic/x-pack-elasticsearch#555)
Essentially an update to https://github.com/elastic/prelert-legacy/pull/736 .  Still does not
default to using shared indices, but adds the capability for two jobs to share the same one
without conflict

Still does not default to using shared indices, just adds the capability for two jobs to share the same one without conflict.

Original commit: elastic/x-pack-elasticsearch@60d93a06ea
2017-02-14 14:59:28 -05:00
Zachary Tong 72ef1acdc0 Move domainSplit injection to ScrollDataExtractor (elastic/x-pack-elasticsearch#554)
By injecting immediately before use, we don't have to store it in the config and return
huge blobs to the user when they request the config

Original commit: elastic/x-pack-elasticsearch@2e53c930ea
2017-02-14 12:26:21 -05:00
Adrien Grand ac8f3ffec4 Fix security source filtering. (elastic/x-pack-elasticsearch#561)
Source filtering is currently written as if the fact that an object is accepted
means that all sub fields and objects are implicitly accepted, which is not
true.

Original commit: elastic/x-pack-elasticsearch@afc4440e7a
2017-02-14 17:44:20 +01:00
Martijn van Groningen 6e411c6118 [ml] Added allocation enabled node attribute setting
This node attribute setting defaults to true when ml is enabled and jobs are only opened on nodes where the allocation enabled node attribute is set to true.

Original commit: elastic/x-pack-elasticsearch@fd3f1b3058
2017-02-14 16:58:08 +01:00
Colin Goodheart-Smithe 6183015639 Adds licensing to machine learning feature (elastic/x-pack-elasticsearch#559)
This change adds licensing to the maching learning feature, and only allows access to machine learning if a trial or platinum license is installed.

Further, this change also renames `MlPlugin` to `MachineLearning` in line with the other feature plugin names and move the enabled setting to `XPackSettings`

Original commit: elastic/x-pack-elasticsearch@48ea9d781b
2017-02-14 15:47:37 +00:00
Martijn van Groningen 67fe584e0f [ml] Also allow closing failed jobs
Relates to elastic/x-pack-elasticsearch#545

Original commit: elastic/x-pack-elasticsearch@1c814afbf0
2017-02-14 15:38:46 +01:00
Dimitris Athanasiou 4976feb53b [ML] Extract MlInitializationService steps into respective methods (elastic/x-pack-elasticsearch#560)
Original commit: elastic/x-pack-elasticsearch@c43d02e752
2017-02-14 14:05:39 +00:00
David Kyle 27a90f4f0d Remove the job ignore_downtime option (elastic/x-pack-elasticsearch#550)
Original commit: elastic/x-pack-elasticsearch@5a7c4fdfea
2017-02-14 09:43:25 +00:00
Jason Tedor c4be485514 Add support for dispatching bad requests
This commit is a response to an upstream change in Elasticsearch which
added a bad dispatch request handler.

Relates elastic/x-pack-elasticsearch#553

Original commit: elastic/x-pack-elasticsearch@c28baee799
2017-02-13 17:39:44 -05:00
jaymode 82a80dbf42 Update seriailization after backport of content type changes
Original commit: elastic/x-pack-elasticsearch@0ef4dda41d
2017-02-13 13:18:56 -05:00
Martijn van Groningen 5eb8414215 [ml] Specify the minimum supported version,
so that we don't send ml metadata to older nodes.

Original commit: elastic/x-pack-elasticsearch@f196b5c461
2017-02-13 18:15:48 +01:00