Commit Graph

229 Commits

Author SHA1 Message Date
Zachary Tong c680b79641 Remove norelease
Issue elastic/x-pack-elasticsearch#820 filed to track progress of better error handling, but not a blocker so removing
the norelease

Original commit: elastic/x-pack-elasticsearch@82cf6eebe2
2017-03-23 13:07:50 -04:00
Zachary Tong cf6bca96db Remove norelease and adjust audit logging
Audit logging in wrong location.  Norelease removed because the proper solution
will take some more work, and this edge-case is not likely to be run into frequently.

More details in elastic/x-pack-elasticsearch#818

Original commit: elastic/x-pack-elasticsearch@ab0d731abf
2017-03-23 13:01:37 -04:00
Igor Motov fba5c09e3d Persistent Tasks: remove task restart on failure (elastic/x-pack-elasticsearch#815)
If a persistent task throws an exception, the persistent tasks framework will no longer try to restart the task. This is a temporary measure to prevent threshing the cluster with endless restart attempt. We will revisit this in the future version to make the restart process more robust. Please note, however, that if node executing the task goes down, the task will still be restarted on another node.

Original commit: elastic/x-pack-elasticsearch@30712e0fbf
2017-03-23 12:56:48 -04:00
Hendrik Muhs 6f7f466fa3 [ML] move DataStreamDiagnostics into DataCountsReporter (elastic/x-pack-elasticsearch#775)
repair DataStreamDiagnostics

Moves DataStreamDiagnostics into DataCountsReporter to survive if job is opened/closed/fed in chunks.

relates elastic/x-pack-elasticsearch#764

Original commit: elastic/x-pack-elasticsearch@29c221a451
2017-03-23 16:43:51 +01:00
Martijn van Groningen bb9befcdcb [ML] Include MlMetadata in all xcontent contexts.
Original commit: elastic/x-pack-elasticsearch@e870b5bf6d
2017-03-23 16:10:24 +01:00
Colin Goodheart-Smithe dde0570296 [ML] Removes direct dependency on Jackson Parser (elastic/x-pack-elasticsearch#810)
* [ML] Removes direct dependency on Jackson Parser

The classes that read data from the post data action ready to write to the autodetect process had a direct dependency on Jackson’s `Parser` class. This changes makes those classes depend on XContent instead making them consistent with the way we parse requests and data across Elasticsearch and X-Pack.

* Simplify json record reader

This commit removes the unnecessary `AbstractJsonRecordReader` and `JsonRecordReader` interfaces/classes. These are not required as we do and should only have one implementation of reading json records.

Original commit: elastic/x-pack-elasticsearch@366b8af943
2017-03-23 13:56:32 +00:00
David Roberts 7756067e5d [ML] Check underlying index permissions on datafeed PUT (elastic/x-pack-elasticsearch#809)
Also added a test to ensure underlying index permissions are required
for datafeed preview.

Relates elastic/x-pack-elasticsearch#648

Original commit: elastic/x-pack-elasticsearch@6edadbb401
2017-03-23 13:56:04 +00:00
Martijn van Groningen 93d7b8c14b [ML] State check doesn't need to know if task is stale now that task validation is only done at create time.
Original commit: elastic/x-pack-elasticsearch@d19858240b
2017-03-23 14:40:20 +01:00
Martijn van Groningen df86125a7d [TEST] Adjust assertions.
Original commit: elastic/x-pack-elasticsearch@39b289a4d9
2017-03-23 11:34:39 +01:00
Colin Goodheart-Smithe 795a00ddab Removes unnecessary NORELEASE for ML endpoint
Original commit: elastic/x-pack-elasticsearch@bbb203665b
2017-03-23 08:47:37 +00:00
Tim Vernum 3e0f785f0c Add ldap_dn and ldap_groups to user meta-data (elastic/x-pack-elasticsearch#739)
In the LdapRealm, include the user's DN and the list of groups (the DN for each group) in the User object
This allows this information to be referenced in templated roles.

Closes: elastic/x-pack-elasticsearch#729

Original commit: elastic/x-pack-elasticsearch@065f337109
2017-03-23 15:38:57 +11:00
Zachary Tong cc7a1a821f [ML] Load resource and split into keys manually (elastic/x-pack-elasticsearch#786)
Manually loads the resources, splits the lines and fills the map instead of using the ResourceBundle stuff. 

relates elastic/x-pack-elasticsearch#750 

Original commit: elastic/x-pack-elasticsearch@25ff4d5d19
2017-03-22 15:24:47 -04:00
Igor Motov 5c4933f5ea Make PersistentAction independent from TransportActions (elastic/x-pack-elasticsearch#742)
Removes the transport layer dependency from PersistentActions, makes PersistentActionRegistry immutable and rename actions into tasks in class and variable names.

Original commit: elastic/x-pack-elasticsearch@e3e5b79c28
2017-03-22 14:02:30 -04:00
Martijn van Groningen d779bf66a5 [ML] Also serialize the job id in a update job request.
Relates to elastic/x-pack-elasticsearch#787 and elastic/x-pack-elasticsearch#799

Original commit: elastic/x-pack-elasticsearch@fc64d25bcb
2017-03-22 18:09:10 +01:00
Martijn van Groningen 6ddc626c17 [ML] Never restart job locally upon cancel
Relates to elastic/x-pack-elasticsearch#749

Original commit: elastic/x-pack-elasticsearch@6943d75178
2017-03-22 17:09:19 +01:00
Martijn van Groningen f20efa7152 [ML] Renamed action names from */anomaly_detectors/* to */job/*
The action name is currently inconsistent with the class name.
Also as more types of jobs will be added, they should reuse the existing actions,
so it make more sense to gave a generic job name for the action name. Note that
it is ok to have additional rest handlers for new type of jobs.

Original commit: elastic/x-pack-elasticsearch@a70d46fb8a
2017-03-22 16:47:27 +01:00
Martijn van Groningen a59badf842 [ML] Restore model state as part of opening a job.
Before the model state was restored in the background when the job was already opened.

 relates elastic/x-pack-elasticsearch#751

Original commit: elastic/x-pack-elasticsearch@44a3d98365
2017-03-22 16:33:03 +01:00
Jay Modi 85aedb6776 Cancel LDAP runnables if they are pending execution too long (elastic/x-pack-elasticsearch#727)
The LdapRealm submits runnables to the generic thread pool when a bind is necessary as a bind must
be a synchronous operation and we do not want to block network threads on this. However, the
generic threadpool could be full and this runnable could get queued. When this happens requests
appear to hang; to prevent this we submit a delayed executable that will stop the runnable from
attempting to connect to Ldap and provide an exceptional response to the listener.

relates elastic/x-pack-elasticsearch#716

Original commit: elastic/x-pack-elasticsearch@3e43b17f1d
2017-03-22 05:27:46 -07:00
Jay Modi 8ba6e8b3eb Fix merging of field level security rules (elastic/x-pack-elasticsearch#796)
This commit fixes the merging of field level security rules from multiple roles. Prior to 5.2, the
merging was treated as the merging of languages whereas after 5.2, this incorrectly became a merge
of all rules meaning a single wildcard could cause denials to be ignored.

Original commit: elastic/x-pack-elasticsearch@42f9e6d8b0
2017-03-22 05:10:30 -07:00
Martijn van Groningen 3f5d9850ae [ML] Removed public constructors for testing.
Original commit: elastic/x-pack-elasticsearch@a5d45f2b66
2017-03-22 10:52:18 +01:00
Martijn van Groningen c12d487c01 [ML] Make fields final
Original commit: elastic/x-pack-elasticsearch@49c7d79835
2017-03-22 10:46:59 +01:00
Ryan Ernst 8a8ed1cfbb Plugins: Update jarhell calls with new set based api (elastic/x-pack-elasticsearch#743)
This is the xpack side of elastic/elasticsearch#23596

Original commit: elastic/x-pack-elasticsearch@d1654b1ccd
2017-03-21 12:13:37 -07:00
Dimitris Athanasiou f7c4c754c2 [ML] Accept 'now' in start/end params of start datafeed API (elastic/x-pack-elasticsearch#784)
The params start/end of the start datafeed API now accept
'now' as a value.

Also adds a validation that start must be earlier than end.

relates elastic/x-pack-elasticsearch#781

Original commit: elastic/x-pack-elasticsearch@5396dcb5e8
2017-03-21 17:05:36 +00:00
Jay Modi 72248adcbb Asynchronous export of monitoring data (elastic/x-pack-elasticsearch#718)
This commit removes the blocking invocation of bulk requests in monitoring as in some cases this
can lead to exhaustion of the generic threadpool, which effectively prevents the node from
operating normally.

One behavior change that is made by this commit is the MonitoringService will no longer wait
indefinitely when closing as this can lead to a node blocking forever and never shutting down.
Instead a wait of 10 seconds is added, which aligns with the security index audit trail's behavior
on shutdown.

relates elastic/x-pack-elasticsearch#715

Original commit: elastic/x-pack-elasticsearch@5ba7f49aab
2017-03-21 12:15:13 -04:00
Dimitris Athanasiou 58827dd433 [ML] Do not advance time when datafeed encounters extraction problems (elastic/x-pack-elasticsearch#782)
Original commit: elastic/x-pack-elasticsearch@fcc9af6869
2017-03-21 14:50:10 +00:00
Martijn van Groningen cb5d44c9c4 [ML] Don't assign cancel holders to node tasks, but use the services that stop datafeed and jobs directly.
Relates to elastic/x-pack-elasticsearch#749

Original commit: elastic/x-pack-elasticsearch@ff074b0442
2017-03-21 15:30:13 +01:00
Martijn van Groningen 6fae1867ec [ML] Uses AbstractRunnable for running datafeed, so that we handle rejected execution exceptions.
Original commit: elastic/x-pack-elasticsearch@d04d3fa401
2017-03-21 14:18:09 +01:00
Martijn van Groningen dd91ac0cd3 [ML] Don't use default locale.
Original commit: elastic/x-pack-elasticsearch@f1e5016008
2017-03-21 13:28:11 +01:00
Martijn van Groningen 4f5b3dfea3 [ML] Set ml' tp queue sizes to a reasonable level.
Original commit: elastic/x-pack-elasticsearch@7f7b2defcd
2017-03-21 13:09:46 +01:00
Dimitris Athanasiou 62721c78ea [ML] Make ModelSnapshot immutable (elastic/x-pack-elasticsearch#770)
In preparation of extending the model snapshot update API,
it makes sense to make it immutable to avoid any confusion
that could be caused by having setters on it.

I also like immutable stuff :-)

Original commit: elastic/x-pack-elasticsearch@6b2ee527a8
2017-03-21 12:00:54 +00:00
Martijn van Groningen 7cf959cb66 [ML] renamed getter as it overwrote super method that was used for a different purpose.
Original commit: elastic/x-pack-elasticsearch@a8b70e769b
2017-03-21 11:24:41 +01:00
Alexander Reelsen ce0391f3c7 Watcher: Update dependencies (elastic/x-pack-elasticsearch#613)
Updated dependencies for javax.activation and mail to latest versions.

Original commit: elastic/x-pack-elasticsearch@d73529f98f
2017-03-21 10:28:20 +01:00
Alexander Reelsen 0c7c2f521c Watcher: Remove scheduler based trigger engine (elastic/x-pack-elasticsearch#724)
The scheduler based trigger engine is not enabled by default
as the ticker based trigger engine is used. As we dont use it
in production, this commit removes this specific implementation.

It also removes some uneeded abstractions like AbstractTriggerEngine, TriggerEngine.Listener and TriggerEngine.Job

Original commit: elastic/x-pack-elasticsearch@b17a2e9d62
2017-03-21 10:27:41 +01:00
Jay Modi 1a7e842c15 Make XPackFeatureSet#usage calls asynchronous (elastic/x-pack-elasticsearch#738)
This commit makes the XPackFeatureSet#usage calls asynchronous. Previously these were synchronous
calls that would execute a multi-search request from the currently elected master node in a blocking
fashion. The multi-search request is now executed asynchronously.

relates elastic/x-pack-elasticsearch#213

Original commit: elastic/x-pack-elasticsearch@a0cb988442
2017-03-20 14:23:04 -04:00
Boaz Leskes e3d244164c testSendMessage: log message request/response on failures
Original commit: elastic/x-pack-elasticsearch@591284dced
2017-03-20 19:09:16 +01:00
Dimitris Athanasiou 025461c3c8 [ML] Correctly collect datafeed stats for _all (elastic/x-pack-elasticsearch#776)
Datafeed state was always stopped when the stats
endpoint was called with _all. The reason is that
_all was not being expanded into all datafeed IDs.

This commit fixes the issue by expanding _all into
all datafeed IDs in the cluster. Stats are then fetched
for all of them.

relates elastic/x-pack-elasticsearch#693

Original commit: elastic/x-pack-elasticsearch@ccbdb35b6e
2017-03-20 17:51:36 +00:00
David Roberts f96a40c61a [ML] Rename score fields (elastic/x-pack-elasticsearch#769)
* Rename normalized_probability to record_score
* Rename anomaly_score on influencers to influencer_score
* Remove record level anomaly_score
* Remove bucket level max_normalized_probability

Relates elastic/x-pack-elasticsearch#678

Original commit: elastic/x-pack-elasticsearch@c2c05e73d8
2017-03-20 16:19:52 +00:00
Martijn van Groningen 47e28669b3 [TEST] Fix expected error message
Original commit: elastic/x-pack-elasticsearch@5a60b66b4a
2017-03-20 15:07:13 +01:00
Martijn van Groningen 2c8545f38f [TEST] Don't rely on `new Date()` (2)
Relates elastic/x-pack-elasticsearch#746

Original commit: elastic/x-pack-elasticsearch@f1ccdf89f0
2017-03-20 13:19:30 +01:00
Dimitrios Athanasiou 3e8b45f2a2 Revert "[TEST] Muted failing tests, see elastic/x-pack-elasticsearch#771"
This reverts commit elastic/x-pack-elasticsearch@014ae38d5b.

Original commit: elastic/x-pack-elasticsearch@c34a5bcc60
2017-03-20 12:06:14 +00:00
Martijn van Groningen 67e4544267 [TEST] making failing restrictions tighter.
Original commit: elastic/x-pack-elasticsearch@b22352a5ba
2017-03-20 12:27:53 +01:00
Martijn van Groningen 06277c3677 [ML] Upon job close validate if datafeed has been stopped
and fail otherwise.

Original commit: elastic/x-pack-elasticsearch@d13bfe1041
2017-03-20 12:03:52 +01:00
Martijn van Groningen 06ce981fed [TEST] Don't rely on `new Date()`
relates elastic/x-pack-elasticsearch#746

Original commit: elastic/x-pack-elasticsearch@11879958b6
2017-03-20 09:20:05 +01:00
David Roberts aa02fa85da [ML] Reduce caching time for native code dependency to 0 (elastic/x-pack-elasticsearch#772)
Prior to this change the integration tests could fail for up to 24
hours after a change to the C++ output format

Original commit: elastic/x-pack-elasticsearch@b8d9fa0adb
2017-03-19 15:58:19 +00:00
Hendrik Muhs 7832783122 [ML] Rename model plot elastic/x-pack-elasticsearch#284 (elastic/x-pack-elasticsearch#753)
Rename model_debug_output to model_plot

The model debug output is 1 result type of the autodetect (anomaly detection) process reporting bounds/statistics/state - not anomalies - It's main usage as of time of writing is visualization in the UI, naming the 'blue shadow' under the result graph. Rename in order to remove 'debug' in the name

Original commit: elastic/x-pack-elasticsearch@86de428512
2017-03-19 11:55:06 +01:00
Martijn van Groningen a76232f5e4 [TEST] Muted failing tests, see elastic/x-pack-elasticsearch#771
Original commit: elastic/x-pack-elasticsearch@014ae38d5b
2017-03-18 12:26:07 +01:00
Dimitris Athanasiou ca931f4b39 [ML] Default model_snapshots_retention_days to 1 (elastic/x-pack-elasticsearch#759)
relates elastic/x-pack-elasticsearch#722

Original commit: elastic/x-pack-elasticsearch@e68aaf229c
2017-03-17 17:56:46 +00:00
David Kyle 30ea37943c [ML] Change delimiter in the names of state docs (elastic/x-pack-elasticsearch#768)
Original commit: elastic/x-pack-elasticsearch@34acda3ffe
2017-03-17 17:18:52 +00:00
Martijn van Groningen 9d4e3efad4 [ML] Fix deleteCategorizerState() by using delete by query instead of individual blocking delete requests.
Original commit: elastic/x-pack-elasticsearch@f28cc9fc5b
2017-03-17 11:54:31 +01:00
David Kyle a7d42f3732 Delete state in the JobStorageDeletionTask (elastic/x-pack-elasticsearch#702)
Original commit: elastic/x-pack-elasticsearch@272b0894c5
2017-03-17 09:37:28 +00:00