Commit Graph

207 Commits

Author SHA1 Message Date
Dimitris Athanasiou f7c4c754c2 [ML] Accept 'now' in start/end params of start datafeed API (elastic/x-pack-elasticsearch#784)
The params start/end of the start datafeed API now accept
'now' as a value.

Also adds a validation that start must be earlier than end.

relates elastic/x-pack-elasticsearch#781

Original commit: elastic/x-pack-elasticsearch@5396dcb5e8
2017-03-21 17:05:36 +00:00
Jay Modi 72248adcbb Asynchronous export of monitoring data (elastic/x-pack-elasticsearch#718)
This commit removes the blocking invocation of bulk requests in monitoring as in some cases this
can lead to exhaustion of the generic threadpool, which effectively prevents the node from
operating normally.

One behavior change that is made by this commit is the MonitoringService will no longer wait
indefinitely when closing as this can lead to a node blocking forever and never shutting down.
Instead a wait of 10 seconds is added, which aligns with the security index audit trail's behavior
on shutdown.

relates elastic/x-pack-elasticsearch#715

Original commit: elastic/x-pack-elasticsearch@5ba7f49aab
2017-03-21 12:15:13 -04:00
Dimitris Athanasiou 58827dd433 [ML] Do not advance time when datafeed encounters extraction problems (elastic/x-pack-elasticsearch#782)
Original commit: elastic/x-pack-elasticsearch@fcc9af6869
2017-03-21 14:50:10 +00:00
Martijn van Groningen cb5d44c9c4 [ML] Don't assign cancel holders to node tasks, but use the services that stop datafeed and jobs directly.
Relates to elastic/x-pack-elasticsearch#749

Original commit: elastic/x-pack-elasticsearch@ff074b0442
2017-03-21 15:30:13 +01:00
Martijn van Groningen 6fae1867ec [ML] Uses AbstractRunnable for running datafeed, so that we handle rejected execution exceptions.
Original commit: elastic/x-pack-elasticsearch@d04d3fa401
2017-03-21 14:18:09 +01:00
Martijn van Groningen dd91ac0cd3 [ML] Don't use default locale.
Original commit: elastic/x-pack-elasticsearch@f1e5016008
2017-03-21 13:28:11 +01:00
Martijn van Groningen 4f5b3dfea3 [ML] Set ml' tp queue sizes to a reasonable level.
Original commit: elastic/x-pack-elasticsearch@7f7b2defcd
2017-03-21 13:09:46 +01:00
Dimitris Athanasiou 62721c78ea [ML] Make ModelSnapshot immutable (elastic/x-pack-elasticsearch#770)
In preparation of extending the model snapshot update API,
it makes sense to make it immutable to avoid any confusion
that could be caused by having setters on it.

I also like immutable stuff :-)

Original commit: elastic/x-pack-elasticsearch@6b2ee527a8
2017-03-21 12:00:54 +00:00
Martijn van Groningen 7cf959cb66 [ML] renamed getter as it overwrote super method that was used for a different purpose.
Original commit: elastic/x-pack-elasticsearch@a8b70e769b
2017-03-21 11:24:41 +01:00
Alexander Reelsen ce0391f3c7 Watcher: Update dependencies (elastic/x-pack-elasticsearch#613)
Updated dependencies for javax.activation and mail to latest versions.

Original commit: elastic/x-pack-elasticsearch@d73529f98f
2017-03-21 10:28:20 +01:00
Alexander Reelsen 0c7c2f521c Watcher: Remove scheduler based trigger engine (elastic/x-pack-elasticsearch#724)
The scheduler based trigger engine is not enabled by default
as the ticker based trigger engine is used. As we dont use it
in production, this commit removes this specific implementation.

It also removes some uneeded abstractions like AbstractTriggerEngine, TriggerEngine.Listener and TriggerEngine.Job

Original commit: elastic/x-pack-elasticsearch@b17a2e9d62
2017-03-21 10:27:41 +01:00
Jay Modi 1a7e842c15 Make XPackFeatureSet#usage calls asynchronous (elastic/x-pack-elasticsearch#738)
This commit makes the XPackFeatureSet#usage calls asynchronous. Previously these were synchronous
calls that would execute a multi-search request from the currently elected master node in a blocking
fashion. The multi-search request is now executed asynchronously.

relates elastic/x-pack-elasticsearch#213

Original commit: elastic/x-pack-elasticsearch@a0cb988442
2017-03-20 14:23:04 -04:00
Boaz Leskes e3d244164c testSendMessage: log message request/response on failures
Original commit: elastic/x-pack-elasticsearch@591284dced
2017-03-20 19:09:16 +01:00
Dimitris Athanasiou 025461c3c8 [ML] Correctly collect datafeed stats for _all (elastic/x-pack-elasticsearch#776)
Datafeed state was always stopped when the stats
endpoint was called with _all. The reason is that
_all was not being expanded into all datafeed IDs.

This commit fixes the issue by expanding _all into
all datafeed IDs in the cluster. Stats are then fetched
for all of them.

relates elastic/x-pack-elasticsearch#693

Original commit: elastic/x-pack-elasticsearch@ccbdb35b6e
2017-03-20 17:51:36 +00:00
David Roberts f96a40c61a [ML] Rename score fields (elastic/x-pack-elasticsearch#769)
* Rename normalized_probability to record_score
* Rename anomaly_score on influencers to influencer_score
* Remove record level anomaly_score
* Remove bucket level max_normalized_probability

Relates elastic/x-pack-elasticsearch#678

Original commit: elastic/x-pack-elasticsearch@c2c05e73d8
2017-03-20 16:19:52 +00:00
Martijn van Groningen 47e28669b3 [TEST] Fix expected error message
Original commit: elastic/x-pack-elasticsearch@5a60b66b4a
2017-03-20 15:07:13 +01:00
Martijn van Groningen 2c8545f38f [TEST] Don't rely on `new Date()` (2)
Relates elastic/x-pack-elasticsearch#746

Original commit: elastic/x-pack-elasticsearch@f1ccdf89f0
2017-03-20 13:19:30 +01:00
Dimitrios Athanasiou 3e8b45f2a2 Revert "[TEST] Muted failing tests, see elastic/x-pack-elasticsearch#771"
This reverts commit elastic/x-pack-elasticsearch@014ae38d5b.

Original commit: elastic/x-pack-elasticsearch@c34a5bcc60
2017-03-20 12:06:14 +00:00
Martijn van Groningen 67e4544267 [TEST] making failing restrictions tighter.
Original commit: elastic/x-pack-elasticsearch@b22352a5ba
2017-03-20 12:27:53 +01:00
Martijn van Groningen 06277c3677 [ML] Upon job close validate if datafeed has been stopped
and fail otherwise.

Original commit: elastic/x-pack-elasticsearch@d13bfe1041
2017-03-20 12:03:52 +01:00
Martijn van Groningen 06ce981fed [TEST] Don't rely on `new Date()`
relates elastic/x-pack-elasticsearch#746

Original commit: elastic/x-pack-elasticsearch@11879958b6
2017-03-20 09:20:05 +01:00
David Roberts aa02fa85da [ML] Reduce caching time for native code dependency to 0 (elastic/x-pack-elasticsearch#772)
Prior to this change the integration tests could fail for up to 24
hours after a change to the C++ output format

Original commit: elastic/x-pack-elasticsearch@b8d9fa0adb
2017-03-19 15:58:19 +00:00
Hendrik Muhs 7832783122 [ML] Rename model plot elastic/x-pack-elasticsearch#284 (elastic/x-pack-elasticsearch#753)
Rename model_debug_output to model_plot

The model debug output is 1 result type of the autodetect (anomaly detection) process reporting bounds/statistics/state - not anomalies - It's main usage as of time of writing is visualization in the UI, naming the 'blue shadow' under the result graph. Rename in order to remove 'debug' in the name

Original commit: elastic/x-pack-elasticsearch@86de428512
2017-03-19 11:55:06 +01:00
Martijn van Groningen a76232f5e4 [TEST] Muted failing tests, see elastic/x-pack-elasticsearch#771
Original commit: elastic/x-pack-elasticsearch@014ae38d5b
2017-03-18 12:26:07 +01:00
Dimitris Athanasiou ca931f4b39 [ML] Default model_snapshots_retention_days to 1 (elastic/x-pack-elasticsearch#759)
relates elastic/x-pack-elasticsearch#722

Original commit: elastic/x-pack-elasticsearch@e68aaf229c
2017-03-17 17:56:46 +00:00
David Kyle 30ea37943c [ML] Change delimiter in the names of state docs (elastic/x-pack-elasticsearch#768)
Original commit: elastic/x-pack-elasticsearch@34acda3ffe
2017-03-17 17:18:52 +00:00
Martijn van Groningen 9d4e3efad4 [ML] Fix deleteCategorizerState() by using delete by query instead of individual blocking delete requests.
Original commit: elastic/x-pack-elasticsearch@f28cc9fc5b
2017-03-17 11:54:31 +01:00
David Kyle a7d42f3732 Delete state in the JobStorageDeletionTask (elastic/x-pack-elasticsearch#702)
Original commit: elastic/x-pack-elasticsearch@272b0894c5
2017-03-17 09:37:28 +00:00
Ali Beyad 407cc3a9d7 Only allow platinum licenses to use custom role providers (elastic/x-pack-elasticsearch#748)
This commit restricts custom role providers to only apply to those
instances of x-pack with a platinum level license.  All other license
types will not be allowed to use custom role providers.  Any custom role
providers implemented via the XPackExtension will not take effect unless
the license is platinum.

relates elastic/x-pack-elasticsearch#720

Original commit: elastic/x-pack-elasticsearch@4fc35494ee
2017-03-16 13:47:35 -04:00
Dimitris Athanasiou 16efd4e474 [ML] Use TimeValue in job and datafeed configs (elastic/x-pack-elasticsearch#732)
relates elastic/x-pack-elasticsearch#679

Original commit: elastic/x-pack-elasticsearch@891edd5bfe
2017-03-16 14:07:47 +00:00
Dimitris Athanasiou 6c9727c2db [ML] Change default data_description to JSON (elastic/x-pack-elasticsearch#707)
relates elastic/x-pack-elasticsearch#674

Original commit: elastic/x-pack-elasticsearch@efc5f4a269
2017-03-16 11:48:57 +00:00
Tanguy Leroux 803654d9bf [Test] Mute MonitoringBulkTests
Related issue: https://github.com/elastic/x-pack-elasticsearch/issues/353

Original commit: elastic/x-pack-elasticsearch@d2600652e7
2017-03-16 09:57:32 +01:00
Ryan Ernst 8c01d6ea69 Tests: Add cluster health check to xpack integ wait conditions (elastic/x-pack-elasticsearch#740)
The wait condition used for integ tests by default calls the cluster
health api with wait_for_nodes nd wait_for_status. However, xpack
overrides the wait condition to add auth, but most of these conditions
still looked at the root ES url, which means the tests are susceptible
to race conditions with the check and node startup. This change modifies
the url for the authenticated wait condtion to check the health api,
with the appropriate wait_for_nodes and wait_for_status.

Original commit: elastic/x-pack-elasticsearch@0b23ef528f
2017-03-15 10:23:26 -07:00
Jay Modi 149e251445 Filtered RestRequests should allow access to the remote address (elastic/x-pack-elasticsearch#741)
When adding support for rest request filtering of sensitive content, the overridden rest request
did not properly delegate the #getRemoteAddress method to the wrapped request. This resulted in a
NPE when a filtered rest request was created and an audit record needed to be generated.

relates elastic/x-pack-elasticsearch#714

Original commit: elastic/x-pack-elasticsearch@710b43355b
2017-03-15 07:13:09 -06:00
Jay Modi bc53dbfb46 Remove unused sign user setting (elastic/x-pack-elasticsearch#736)
User signing was removed with the move to TLS only transport and this change removes a no longer
used setting.

Original commit: elastic/x-pack-elasticsearch@c221f1f4a4
2017-03-14 19:29:16 -07:00
David Roberts 0b7c735aec [ML] Add machine learning privileges/roles (elastic/x-pack-elasticsearch#673)
* Changed ML action names to allow distinguishing of admin and read-only actions
  using wildcards
* Added manage_ml and monitor_ml built-in privileges as subsets of the existing
  manage and monitor privileges
* Added out-of-the-box machine_learning_admin and machine_learning_user roles
* Changed machine learning results endpoints to use a NodeClient rather than an
  InternalClient when searching for results so that index/document level permissions
  applied to ML results are respected

Original commit: elastic/x-pack-elasticsearch@eee800aaa8
2017-03-14 16:13:41 +00:00
Jay Modi 8df7a82435 Remove unnecessary String creation from password char[] (elastic/x-pack-elasticsearch#713)
This commit removes an unnecessary String creation from the char[] of a password and instead uses
a byte[] that is cleared after it is used to prevent the password bytes from sticking around in
memory longer than required.

Original commit: elastic/x-pack-elasticsearch@1154a68965
2017-03-13 18:50:48 -07:00
Dimitris Athanasiou db48e92f54 [ML] Add support for percentiles aggregation (elastic/x-pack-elasticsearch#733)
Original commit: elastic/x-pack-elasticsearch@7c8357ac04
2017-03-13 23:32:33 +00:00
Steffen Siering 07bde45671 Init beats monitoring support (elastic/x-pack-elasticsearch#543)
- introduce builtin user 'beats_system'
- init monitoring-beats ES mapping
- add beats to MonitoredSystem + Resolver

Original commit: elastic/x-pack-elasticsearch@4d7b45d54d
2017-03-13 10:29:42 -07:00
Tanguy Leroux e5cc038616 Remove unused import
Original commit: elastic/x-pack-elasticsearch@742c12e8b8
2017-03-12 13:36:39 -07:00
Tanguy Leroux 73836f43c5 [Test] Mute flaky monitoring tests
Original commit: elastic/x-pack-elasticsearch@3670e827c8
2017-03-12 13:31:05 -07:00
Hendrik Muhs ce1a910542 [ML] DataStreamDiagnostics Rebase (elastic/x-pack-elasticsearch#698)
DataStreamDiagnostics

DataStreamDiagnostics analyzes input data regarding machine learning fit. It checks whether data is sane/plausible as anomaly detection on broken data (or misconfiguration).

Original commit: elastic/x-pack-elasticsearch@2f37d3c960
2017-03-12 09:40:27 -07:00
Jason Tedor 0457de27dd Fix SHAs for Netty dependencies
This commit fixes the SHA hashes for the Netty dependencies after Netty
was upgraded from version 4.1.8 to version 4.1.9.

Original commit: elastic/x-pack-elasticsearch@ca4c376886
2017-03-12 01:19:17 -05:00
Alexander Reelsen 54527b86ea Tests: Remove DynamicIndexNameIntegrationTests (elastic/x-pack-elasticsearch#725)
This integration tests only checks if date math index names are
supported. This is nothing that a watcher test should check for.

Original commit: elastic/x-pack-elasticsearch@8a06988fff
2017-03-11 17:35:29 -08:00
David Kyle b06ce7e7c1 [ML] Remove dead delete job index code
Original commit: elastic/x-pack-elasticsearch@533b2a0207
2017-03-10 16:30:02 -08:00
Dimitris Athanasiou 72f7698647 [ML] Extend aggregation support (elastic/x-pack-elasticsearch#705)
* [ML] Allow summary_count_field_name to be set to custom field

... in aggregated datafeeds.

This allows implementing e.g. distinct_count anomaly detection
with aggregations. (see case 1 in elastic/x-pack-elasticsearch#659)

Relates to elastic/x-pack-elasticsearch#659

* [ML] Handle multiple leaf aggregations

This commit allows multiple numeric leaf aggregations.
That enables use cases where the data are presummarised and
thus the summary count field is a sum aggregation on the
event rate field. (see 2 in elastic/x-pack-elasticsearch#659)

Relates to elastic/x-pack-elasticsearch#659

Original commit: elastic/x-pack-elasticsearch@07a34a18c6
2017-03-09 15:26:44 +00:00
David Roberts 1c52495c5a [ML] The Machine Learning class no longer needs to be a plugin (elastic/x-pack-elasticsearch#712)
Now machine learning is simply a feature of X-Pack, so I have made the
MachineLearning class more like the other feature classes. It no longer extends
Plugin and its createComponents() method takes an InternalClient rather than
a generic Client.

Original commit: elastic/x-pack-elasticsearch@704860147c
2017-03-08 15:54:26 +00:00
Martijn van Groningen e972f0216d [ML] Auto close jobs and auto stop datafeeds if license is invalid.
Original commit: elastic/x-pack-elasticsearch@6e2634b2b6
2017-03-05 11:14:45 +01:00
Martijn van Groningen 6365dec42e [TEST] Changed internal distributed ml tests to extend from ESIntegTest and only run with ml to avoid noise.
Changed DatafeedJobsIT to extend directly from SecurityIntegTest as it uses external cluster.

Original commit: elastic/x-pack-elasticsearch@d646c59491
2017-03-04 08:10:46 +01:00
Martijn van Groningen 49826ae134 [ML] Simplify JobManager#createJobResultIndex(...)
Original commit: elastic/x-pack-elasticsearch@e8b01e183c
2017-03-03 16:33:30 +01:00