Commit Graph

5070 Commits

Author SHA1 Message Date
Hendrik Muhs 7832783122 [ML] Rename model plot elastic/x-pack-elasticsearch#284 (elastic/x-pack-elasticsearch#753)
Rename model_debug_output to model_plot

The model debug output is 1 result type of the autodetect (anomaly detection) process reporting bounds/statistics/state - not anomalies - It's main usage as of time of writing is visualization in the UI, naming the 'blue shadow' under the result graph. Rename in order to remove 'debug' in the name

Original commit: elastic/x-pack-elasticsearch@86de428512
2017-03-19 11:55:06 +01:00
Martijn van Groningen a76232f5e4 [TEST] Muted failing tests, see elastic/x-pack-elasticsearch#771
Original commit: elastic/x-pack-elasticsearch@014ae38d5b
2017-03-18 12:26:07 +01:00
Jason Tedor edf810a422 Adjust line-lengths for license-tools
This commit is a reformatting of the source files in license-tools to
bring them under the 100-column line-length limit.

Original commit: elastic/x-pack-elasticsearch@7d7a6c7a23
2017-03-17 14:17:10 -04:00
Dimitris Athanasiou ca931f4b39 [ML] Default model_snapshots_retention_days to 1 (elastic/x-pack-elasticsearch#759)
relates elastic/x-pack-elasticsearch#722

Original commit: elastic/x-pack-elasticsearch@e68aaf229c
2017-03-17 17:56:46 +00:00
David Kyle 30ea37943c [ML] Change delimiter in the names of state docs (elastic/x-pack-elasticsearch#768)
Original commit: elastic/x-pack-elasticsearch@34acda3ffe
2017-03-17 17:18:52 +00:00
Jason Tedor 3935d82763 Migrate to max line length of 100
This commit moves the checkstyle rule of max line length from 140
characters to 100 characters. We whitelist all existing violations and
will address them in follow-ups.

Relates elastic/x-pack-elasticsearch#762

Original commit: elastic/x-pack-elasticsearch@0971485011
2017-03-17 11:20:26 -04:00
Martijn van Groningen 9d4e3efad4 [ML] Fix deleteCategorizerState() by using delete by query instead of individual blocking delete requests.
Original commit: elastic/x-pack-elasticsearch@f28cc9fc5b
2017-03-17 11:54:31 +01:00
David Kyle a7d42f3732 Delete state in the JobStorageDeletionTask (elastic/x-pack-elasticsearch#702)
Original commit: elastic/x-pack-elasticsearch@272b0894c5
2017-03-17 09:37:28 +00:00
Martijn van Groningen 60c79ccdbb [TEST] Set job close timeout to 20s.
Original commit: elastic/x-pack-elasticsearch@9b92c940cb
2017-03-16 21:57:36 +01:00
Martijn van Groningen 73c5c1576f [TEST] Added logging and removed unneeded jackson-databind test dependency.
Original commit: elastic/x-pack-elasticsearch@58c247d696
2017-03-16 21:17:14 +01:00
Ali Beyad 407cc3a9d7 Only allow platinum licenses to use custom role providers (elastic/x-pack-elasticsearch#748)
This commit restricts custom role providers to only apply to those
instances of x-pack with a platinum level license.  All other license
types will not be allowed to use custom role providers.  Any custom role
providers implemented via the XPackExtension will not take effect unless
the license is platinum.

relates elastic/x-pack-elasticsearch#720

Original commit: elastic/x-pack-elasticsearch@4fc35494ee
2017-03-16 13:47:35 -04:00
Dimitris Athanasiou 16efd4e474 [ML] Use TimeValue in job and datafeed configs (elastic/x-pack-elasticsearch#732)
relates elastic/x-pack-elasticsearch#679

Original commit: elastic/x-pack-elasticsearch@891edd5bfe
2017-03-16 14:07:47 +00:00
Dimitris Athanasiou 6c9727c2db [ML] Change default data_description to JSON (elastic/x-pack-elasticsearch#707)
relates elastic/x-pack-elasticsearch#674

Original commit: elastic/x-pack-elasticsearch@efc5f4a269
2017-03-16 11:48:57 +00:00
Tanguy Leroux 803654d9bf [Test] Mute MonitoringBulkTests
Related issue: https://github.com/elastic/x-pack-elasticsearch/issues/353

Original commit: elastic/x-pack-elasticsearch@d2600652e7
2017-03-16 09:57:32 +01:00
Ryan Ernst baaad36c5e Fix tribe test setup to look for a single node
numNodes isn't available because it is not in the normal context of
cluster configuration.

relates elastic/x-pack-elasticsearch#740

Original commit: elastic/x-pack-elasticsearch@336147f20c
2017-03-15 15:55:10 -07:00
Ryan Ernst 534584d525 Tests: Use cluster health api for wait condition in ml integ test
This was forgotten in elastic/x-pack-elasticsearch#740

Original commit: elastic/x-pack-elasticsearch@47c10dc543
2017-03-15 10:27:38 -07:00
Ryan Ernst 8c01d6ea69 Tests: Add cluster health check to xpack integ wait conditions (elastic/x-pack-elasticsearch#740)
The wait condition used for integ tests by default calls the cluster
health api with wait_for_nodes nd wait_for_status. However, xpack
overrides the wait condition to add auth, but most of these conditions
still looked at the root ES url, which means the tests are susceptible
to race conditions with the check and node startup. This change modifies
the url for the authenticated wait condtion to check the health api,
with the appropriate wait_for_nodes and wait_for_status.

Original commit: elastic/x-pack-elasticsearch@0b23ef528f
2017-03-15 10:23:26 -07:00
Jay Modi 149e251445 Filtered RestRequests should allow access to the remote address (elastic/x-pack-elasticsearch#741)
When adding support for rest request filtering of sensitive content, the overridden rest request
did not properly delegate the #getRemoteAddress method to the wrapped request. This resulted in a
NPE when a filtered rest request was created and an audit record needed to be generated.

relates elastic/x-pack-elasticsearch#714

Original commit: elastic/x-pack-elasticsearch@710b43355b
2017-03-15 07:13:09 -06:00
Jay Modi bc53dbfb46 Remove unused sign user setting (elastic/x-pack-elasticsearch#736)
User signing was removed with the move to TLS only transport and this change removes a no longer
used setting.

Original commit: elastic/x-pack-elasticsearch@c221f1f4a4
2017-03-14 19:29:16 -07:00
David Roberts 0b7c735aec [ML] Add machine learning privileges/roles (elastic/x-pack-elasticsearch#673)
* Changed ML action names to allow distinguishing of admin and read-only actions
  using wildcards
* Added manage_ml and monitor_ml built-in privileges as subsets of the existing
  manage and monitor privileges
* Added out-of-the-box machine_learning_admin and machine_learning_user roles
* Changed machine learning results endpoints to use a NodeClient rather than an
  InternalClient when searching for results so that index/document level permissions
  applied to ML results are respected

Original commit: elastic/x-pack-elasticsearch@eee800aaa8
2017-03-14 16:13:41 +00:00
Jay Modi 8df7a82435 Remove unnecessary String creation from password char[] (elastic/x-pack-elasticsearch#713)
This commit removes an unnecessary String creation from the char[] of a password and instead uses
a byte[] that is cleared after it is used to prevent the password bytes from sticking around in
memory longer than required.

Original commit: elastic/x-pack-elasticsearch@1154a68965
2017-03-13 18:50:48 -07:00
Dimitris Athanasiou db48e92f54 [ML] Add support for percentiles aggregation (elastic/x-pack-elasticsearch#733)
Original commit: elastic/x-pack-elasticsearch@7c8357ac04
2017-03-13 23:32:33 +00:00
Steffen Siering 07bde45671 Init beats monitoring support (elastic/x-pack-elasticsearch#543)
- introduce builtin user 'beats_system'
- init monitoring-beats ES mapping
- add beats to MonitoredSystem + Resolver

Original commit: elastic/x-pack-elasticsearch@4d7b45d54d
2017-03-13 10:29:42 -07:00
Tanguy Leroux e5cc038616 Remove unused import
Original commit: elastic/x-pack-elasticsearch@742c12e8b8
2017-03-12 13:36:39 -07:00
Tanguy Leroux 73836f43c5 [Test] Mute flaky monitoring tests
Original commit: elastic/x-pack-elasticsearch@3670e827c8
2017-03-12 13:31:05 -07:00
Hendrik Muhs ce1a910542 [ML] DataStreamDiagnostics Rebase (elastic/x-pack-elasticsearch#698)
DataStreamDiagnostics

DataStreamDiagnostics analyzes input data regarding machine learning fit. It checks whether data is sane/plausible as anomaly detection on broken data (or misconfiguration).

Original commit: elastic/x-pack-elasticsearch@2f37d3c960
2017-03-12 09:40:27 -07:00
Jason Tedor 0457de27dd Fix SHAs for Netty dependencies
This commit fixes the SHA hashes for the Netty dependencies after Netty
was upgraded from version 4.1.8 to version 4.1.9.

Original commit: elastic/x-pack-elasticsearch@ca4c376886
2017-03-12 01:19:17 -05:00
Alexander Reelsen 54527b86ea Tests: Remove DynamicIndexNameIntegrationTests (elastic/x-pack-elasticsearch#725)
This integration tests only checks if date math index names are
supported. This is nothing that a watcher test should check for.

Original commit: elastic/x-pack-elasticsearch@8a06988fff
2017-03-11 17:35:29 -08:00
David Kyle b06ce7e7c1 [ML] Remove dead delete job index code
Original commit: elastic/x-pack-elasticsearch@533b2a0207
2017-03-10 16:30:02 -08:00
Dimitris Athanasiou 72f7698647 [ML] Extend aggregation support (elastic/x-pack-elasticsearch#705)
* [ML] Allow summary_count_field_name to be set to custom field

... in aggregated datafeeds.

This allows implementing e.g. distinct_count anomaly detection
with aggregations. (see case 1 in elastic/x-pack-elasticsearch#659)

Relates to elastic/x-pack-elasticsearch#659

* [ML] Handle multiple leaf aggregations

This commit allows multiple numeric leaf aggregations.
That enables use cases where the data are presummarised and
thus the summary count field is a sum aggregation on the
event rate field. (see 2 in elastic/x-pack-elasticsearch#659)

Relates to elastic/x-pack-elasticsearch#659

Original commit: elastic/x-pack-elasticsearch@07a34a18c6
2017-03-09 15:26:44 +00:00
David Roberts 1c52495c5a [ML] The Machine Learning class no longer needs to be a plugin (elastic/x-pack-elasticsearch#712)
Now machine learning is simply a feature of X-Pack, so I have made the
MachineLearning class more like the other feature classes. It no longer extends
Plugin and its createComponents() method takes an InternalClient rather than
a generic Client.

Original commit: elastic/x-pack-elasticsearch@704860147c
2017-03-08 15:54:26 +00:00
Martijn van Groningen e972f0216d [ML] Auto close jobs and auto stop datafeeds if license is invalid.
Original commit: elastic/x-pack-elasticsearch@6e2634b2b6
2017-03-05 11:14:45 +01:00
Martijn van Groningen 6365dec42e [TEST] Changed internal distributed ml tests to extend from ESIntegTest and only run with ml to avoid noise.
Changed DatafeedJobsIT to extend directly from SecurityIntegTest as it uses external cluster.

Original commit: elastic/x-pack-elasticsearch@d646c59491
2017-03-04 08:10:46 +01:00
Martijn van Groningen 49826ae134 [ML] Simplify JobManager#createJobResultIndex(...)
Original commit: elastic/x-pack-elasticsearch@e8b01e183c
2017-03-03 16:33:30 +01:00
Dimitrios Athanasiou 9e7d6d7fee [TEST] Add more assertions to DeleteExpiredDataIT
Original commit: elastic/x-pack-elasticsearch@ed34093921
2017-03-03 15:32:33 +00:00
Dimitrios Athanasiou f4c8dfce5a [TEST] Add missing parts in delete_expired_data REST spec
Original commit: elastic/x-pack-elasticsearch@3530ed1550
2017-03-03 13:07:42 +00:00
Dimitris Athanasiou a7a36245c9 [ML] Add end-point for deleting expired results (elastic/x-pack-elasticsearch#670)
This commit adds an end-point to force deletion of expired data:

DELETE /_xpack/ml/_delete_expired_data

A few other things are changed too:

- Delete expired results from now rather than start of day
- Rename MlDaily{Management -> Maintenance}Service
- Refresh job indices when job is closing to ensure latest result
  visibility
- Commit results when quantiles are persisted to ensure they are visible
  for renormalization


Original commit: elastic/x-pack-elasticsearch@8ca5272a94
2017-03-03 11:33:22 +00:00
David Roberts 440d005b1a [ML] Prevent repeated cluster state changes when creating index templates (elastic/x-pack-elasticsearch#687)
Switched the order of the local atomic flag check and cluster state check,
based on the observation that we used to sometimes get a storm of cluster
state updates on initial startup, due to a race between checking for the
presence of an index template and starting to create one.

Original commit: elastic/x-pack-elasticsearch@7ae83648ce
2017-03-03 10:59:24 +00:00
David Kyle 01de84a19f [ML] Wait for templates to be installed before running ML Integ tests (elastic/x-pack-elasticsearch#681)
* [ML] Wait for templates to be installed before running ML Integ tests

* Revert put job template check and preserve template change

* Review comments

Original commit: elastic/x-pack-elasticsearch@21deb34f4a
2017-03-03 10:04:34 +00:00
Alexander Reelsen be2ee21d16 Tests: Ensure date is UTC to prevent failing tests
This test used the local time zone to check for an assertion,
and thus failed in US evenings, because the date in UTC rolled over.

relates elastic/x-pack-elasticsearch#239

Original commit: elastic/x-pack-elasticsearch@6286b30407
2017-03-03 00:20:19 +01:00
Tim Vernum b105118ef0 Add `has_privileges` API (elastic/x-pack-elasticsearch#604)
Security API to determine which (if any) of a specified set of index/cluster privileges are held by the current (runAs) user.
Intended for use by Kibana to distinguish between read/write and read-only users, but should be applicable to other uses cases also.

Closes: elastic/x-pack-elasticsearch#282

Original commit: elastic/x-pack-elasticsearch@8b4cfdb858
2017-03-03 18:41:44 +11:00
Ryan Ernst c92562e9d9 Build: Enable notice generation for x-pack (elastic/x-pack-elasticsearch#695)
The only tricky thing here was what to do about ES jars. We now ignore them for the jar sha checking, and the base elasticsearch license and notice is copied here, and elasticsearch jars are mapped to those license/notice files.

Original commit: elastic/x-pack-elasticsearch@a6373cfe4e
2017-03-02 19:06:15 -08:00
Dimitrios Athanasiou c52689a9a0 [ML] Add job_id filter to job alias
This fixes returning results only for the job asked
when a shared index is used.

The commit also refactors the field count check to
use the field mappings API and solves a bug where the
check blows due to the _default_ type not having properties.

[Zach] Minor test tweak to MlJobIT

Original commit: elastic/x-pack-elasticsearch@729f886879
2017-03-02 21:46:34 -05:00
Martijn van Groningen 34a4778b11 [TEST] Wait a bit longer before the lookback import has completed and it stopped the datafeed automatically
Original commit: elastic/x-pack-elasticsearch@6f9b5f8f28
2017-03-02 22:00:11 +01:00
Martijn van Groningen e9e14cbcea [ML] Improve reason why datafeed is stopped.
Original commit: elastic/x-pack-elasticsearch@2055a64689
2017-03-02 18:03:54 +01:00
Dimitrios Athanasiou 0542a9eb92 [TEST] Disable ml in qa modules where necessary
Original commit: elastic/x-pack-elasticsearch@bb311b44d7
2017-03-02 17:01:35 +00:00
David Kyle d8f9b8452a [ML] Fix unit test
Original commit: elastic/x-pack-elasticsearch@fb6ff7be3f
2017-03-02 16:50:52 +00:00
David Kyle 14a0167781 [ML] Restore missing job stats (elastic/x-pack-elasticsearch#667)
* [ML] Pretty print dates in ModelSizeStats
* [ML] Add last_data_time field to DataCounts
* [ML] Add uptime to job stats
* [ML] Pretty print time fields in ModelSnapshot
* [ML] Rename uptime -> open_time

Original commit: elastic/x-pack-elasticsearch@4ce5258a77
2017-03-02 15:54:20 +00:00
Martijn van Groningen 948a8594bb [TEST] Disable ml in audit qa test
Original commit: elastic/x-pack-elasticsearch@34b7243f69
2017-03-02 16:02:15 +01:00
David Roberts 2c365311af [ML] Document test problem and make it slightly less likely
Relates elastic/x-pack-elasticsearch#689

This does NOT fix the problem, but makes it clearer why it happens
and, by changing ordering, should make it happen less often

Original commit: elastic/x-pack-elasticsearch@9ef60073c2
2017-03-02 14:18:31 +00:00