Commit Graph

1870 Commits

Author SHA1 Message Date
Dimitrios Athanasiou 54b72dc938 [TEST] Add ML categorization integ test
Original commit: elastic/x-pack-elasticsearch@ba8c2a1d64
2017-04-07 18:46:01 +01:00
David Roberts b9941de39d [TEST] Fix some assertions to match prior code change
Original commit: elastic/x-pack-elasticsearch@be4cc9cbef
2017-04-07 18:01:32 +01:00
Zachary Tong 8e2e26fc44 [ML] Forbid 'by' and 'over' as fieldnames for by/over (elastic/x-pack-elasticsearch#1007)
relates elastic/x-pack-elasticsearch#1002

Original commit: elastic/x-pack-elasticsearch@ba9005a58e
2017-04-07 16:09:22 +00:00
Dimitris Athanasiou bf111dde7e [ML] Add notification when no suitable nodes exist to allocate job (elastic/x-pack-elasticsearch#1006)
Relates elastic/x-pack-elasticsearch#878

Original commit: elastic/x-pack-elasticsearch@5a96484662
2017-04-07 16:55:10 +01:00
David Roberts 3986235d93 [ML] Fix large state persistence performance (elastic/x-pack-elasticsearch#1004)
There was a problem with the way CompositeBytesReference was used in the
StateProcessor.  In the case of a large state document we ended up with a
deeply nested CompositeBytesReference that then caused a deep stack and N^2
processing in the bulk action processor.

This change uses an intermediate list of byte arrays that get combined into
a single CompositeBytesReference to avoid the deep nesting.

Additionally, errors in state processing now bubble up to close the state
stream, which will cause the C++ process to stop trying to persist more state.

Finally, the results processor also times out after a similar period (30 minutes)
to that used by the state processor.

Original commit: elastic/x-pack-elasticsearch@ceb31481d1
2017-04-07 15:57:21 +01:00
Hendrik Muhs c7fd1aacff [ML] implement _all for closing jobs (elastic/x-pack-elasticsearch#962)
Add a '_all' functionality for closing ML jobs.

For cluster shutdown due to maintenance and major upgrades we recommend the user to stop all datafeeds and jobs. This change add the ability to close all jobs at once where previously it was required to iterate over all jobs and do a explicit close. This is part one of elastic/x-pack-elasticsearch#795, part two can be found in elastic/x-pack-elasticsearch#995.

relates elastic/x-pack-elasticsearch#795

Original commit: elastic/x-pack-elasticsearch@9b251ed7e1
2017-04-07 14:51:13 +02:00
Dimitrios Athanasiou b41288592c [ML] Some logging improvements
- Downgrade flush logging as it's called too often from datafeeds
- Add log info for job opening

Original commit: elastic/x-pack-elasticsearch@a262069f78
2017-04-07 13:36:04 +01:00
Adrien Grand df3cc61c5b Mute ActivateWatchTests.testDeactivateAndActivate.
Relates elastic/x-pack-elasticsearch#395

Original commit: elastic/x-pack-elasticsearch@bdf5a0dfed
2017-04-07 11:03:31 +02:00
Martijn van Groningen d8170861ea [TEST] Check whether datafeed is stopped differently
Original commit: elastic/x-pack-elasticsearch@9bde2cc2c2
2017-04-07 10:50:48 +02:00
Martijn van Groningen b248b6dbfc [TEST] [ML] Run ensureStableCluster(...) on all nodes and run synched flush before executing the disrupt
Original commit: elastic/x-pack-elasticsearch@d10beacb73
2017-04-07 09:51:13 +02:00
Martijn van Groningen 3986a2a06c [ML] Remove PersistentTask#isCurrentStatus() usages
Original commit: elastic/x-pack-elasticsearch@efe7e1e770
2017-04-06 21:32:54 +02:00
Zachary Tong e2a30331ba Refactor StartDataFeed to more intelligently wait on shard status (elastic/x-pack-elasticsearch#974)
Rather than using an async call, this leverages
the Assignment logic while selecting nodes.

Now with 300% more tests!

Original commit: elastic/x-pack-elasticsearch@300d628f72
2017-04-06 18:14:51 +00:00
David Kyle 9a9ae5edc7 [ML] Restore categoriser state after the anomaly detector (elastic/x-pack-elasticsearch#993)
Original commit: elastic/x-pack-elasticsearch@fc4205f1d6
2017-04-06 18:12:52 +01:00
Martijn van Groningen 5585283216 [ML] Added cause to exception.
Relates to elastic/x-pack-elasticsearch#990

Original commit: elastic/x-pack-elasticsearch@a66bc958b7
2017-04-06 16:34:24 +02:00
David Roberts 97866eb0e8 [ML] Increase timeouts for named pipe connections to native processes (elastic/x-pack-elasticsearch#987)
It has been observed that Amazon EBS volumes created from snapshots can
have very high latency the first time a given block is accessed.  This
can lead to named pipes taking longer than 2 seconds to create.
Since the native processes create their named pipes immediately after
startup, and this only takes a fraction of a second on a local disk, 2
seconds was considered a generous timeout, but it seems that in the case
of a remote NAS with lazy provisioning it's not long enough.  During
debugging a latency of just over 3 seconds was observed.  The timeouts
have been increased to 10 seconds.

relates elastic/x-pack-elasticsearch#922

Original commit: elastic/x-pack-elasticsearch@c90434c948
2017-04-06 14:42:27 +01:00
Jay Modi 92942c9327 Add support for IndexOrDocValuesQuery in FieldExtractor (elastic/x-pack-elasticsearch#976)
This commit adds support for extracting the fields from a IndexOrDocValuesQuery so that we can
still take advantage of the cache when this query is used.

relates elastic/x-pack-elasticsearch#973

Original commit: elastic/x-pack-elasticsearch@0b97202a41
2017-04-06 12:44:25 +01:00
Tim Vernum 573b421446 Support multiple indices in SecurityLifecycleService (elastic/x-pack-elasticsearch#961)
Moves the direct management of the security index from SecurityLifecycleService to IndexLifecycleManager, so that the SecurityLifecycleService can take responsibility for several indices.

Multiple security indices are required as we move away from storing multiple types in a single index.

Original commit: elastic/x-pack-elasticsearch@fde3a42b4d
2017-04-06 21:37:33 +10:00
jaymode 7efc9e1270 Test: Fix IndexAuditTrailMutedTests to set the threadpool
The IndexAuditTrailMutedTests have a threadpool but fail to set it on the test client, which causes
a NPE and tests to fail.

Original commit: elastic/x-pack-elasticsearch@d34a4ce080
2017-04-05 13:50:00 -04:00
Alexander Reelsen 16f5f44d41 Watcher: Dont check index auto creation if disabled/transportclient (elastic/x-pack-elasticsearch#975)
When watcher is disabled, no auto index creation checks should happen.

Relates elastic/elasticsearch#23919

Original commit: elastic/x-pack-elasticsearch@d924838b4d
2017-04-05 18:39:54 +01:00
jaymode a53026498f Test: mute SearchGroupsResolverInMemoryTests
This commit mutes SearchGroupsResolverInMemoryTests as they fail with a thread leak. See elastic/x-pack-elasticsearch#971

Original commit: elastic/x-pack-elasticsearch@be7c988e20
2017-04-05 13:23:53 -04:00
Dimitrios Athanasiou e97035af87 Add missing logging param
Original commit: elastic/x-pack-elasticsearch@30ee528798
2017-04-05 17:00:12 +01:00
Dimitris Athanasiou 0be4082ad7 [ML] Add notification for loading snapshot (elastic/x-pack-elasticsearch#970)
As the snapshot that is loaded is an important operational
aspect of a job, this change adds a notification that displays
the loaded snapshot with its latest_record_timestamp and the
job's latest_record_timestamp. Having both allows us to discover
when a job is recovering after a node failure.

relates elastic/x-pack-elasticsearch#872

Original commit: elastic/x-pack-elasticsearch@c2dee495a2
2017-04-05 16:43:14 +01:00
Tanguy Leroux 7b48bac9f4 [Test] Gives more time to LocalExporterTests to delete indices
The test fails on slow machines because of inflight bulk requests
that hit one node while the others are stopping. This commit adds
more time (10s), equivalent to 2 to 3 collection interval, to delete
the monitoring indices. It also add TRACE logging level for the test.

Original commit: elastic/x-pack-elasticsearch@b433937946
2017-04-05 16:38:12 +02:00
Martijn van Groningen 2edbd0efeb [ML] Removed unused methods
Original commit: elastic/x-pack-elasticsearch@8b3c606147
2017-04-05 14:30:21 +02:00
Martijn van Groningen cbbc3cfe20 [ML] Slightly adjust JobProvider#getAutodetectParams(...)
Original commit: elastic/x-pack-elasticsearch@2aaf618d78
2017-04-05 14:18:00 +02:00
Colin Goodheart-Smithe 00a5759d54 [ML] Adds jobType to Job (elastic/x-pack-elasticsearch#947)
* [ML] Adds jobType to Job

This change adds `jobType` field to teh `Job` class so that when the job is written to the index a `job_type` field is written int he document. This will help separate this type of job from other new job types in the future so migrating the index to allow those new type of jobs will be easer

relates elastic/x-pack-elasticsearch#798

* Addresses review comments

Original commit: elastic/x-pack-elasticsearch@d9fd11edb3
2017-04-05 11:12:26 +01:00
Alexander Reelsen 4b2680dac7 Tests: Ensure watch index is deleted to execute test correctly
The test requires no watch index or alias to exist, so making
sure this is deleted as part of the test.

relates elastic/x-pack-elasticsearch#920

Original commit: elastic/x-pack-elasticsearch@c1cc953ce3
2017-04-05 10:36:43 +01:00
Tim Vernum 7f0fd9e1a3 Failed LDAP SearchResults should be Exceptions (elastic/x-pack-elasticsearch#773)
When the LDAP SDK returns a SearchResult that has a non-success ResultCode, convert it to an exception and call onFailure

A configuration setting controls whether failures in referrals should be fatal (defaults to ignoring errors)

Closes: elastic/x-pack-elasticsearch#717

Original commit: elastic/x-pack-elasticsearch@4159758c2a
2017-04-05 18:40:17 +10:00
David Kyle 7c45cb7ccf Fix check style line length failure
Original commit: elastic/x-pack-elasticsearch@d37595db2b
2017-04-05 09:00:16 +01:00
David Kyle f8bb762a59 [TEST] Revert "Tests: Disable `ml/index_layout/Test unrelated index`."
This reverts commit elastic/x-pack-elasticsearch@097f559d0b.

Original commit: elastic/x-pack-elasticsearch@1f63456866
2017-04-05 08:48:23 +01:00
David Kyle 7b45460951 [ML] Wait for dataCounts to be persisted (elastic/x-pack-elasticsearch#952)
Original commit: elastic/x-pack-elasticsearch@48ca4d7363
2017-04-05 08:45:01 +01:00
Igor Motov d11fbfa70c Persistent Tasks: refactor PersistentTasksService to use ActionListener (elastic/x-pack-elasticsearch#937)
PersistentTasksService methods are not using ActionListener<PersistentTask<?>> instead of PersistentTaskOperationListener.

Original commit: elastic/x-pack-elasticsearch@f95d8bda3d
2017-04-04 13:56:22 -04:00
Zachary Tong bac8f010b4 Remove unnecessary headers in Model yaml tests
Original commit: elastic/x-pack-elasticsearch@74e2a5e29a
2017-04-04 13:51:40 -04:00
Tim Vernum 26fb1a6997 Don't assume FLS arrays are pre-sorted (elastic/x-pack-elasticsearch#939)
The `FieldPermissions` class incorrectly assumed that the `granted` and `denied` arrays were
sorted, so it could do a `binarySearch` to see if `_all` was in the arrays.

Original commit: elastic/x-pack-elasticsearch@49b5875602
2017-04-04 18:14:20 +01:00
Jay Modi 1e42473f77 Remove incorrect usages of Arrays.binarySearch (elastic/x-pack-elasticsearch#954)
This is a follow-on to elastic/x-pack-elasticsearch#939, which removes the use of Arrays.binarySearch in the FieldPermissions
class. This change removes other incorrect uses in the rest of the x-pack code and replaces them
with a stream based implementation.

Original commit: elastic/x-pack-elasticsearch@ccca7e9bad
2017-04-04 17:48:58 +01:00
Dimitris Athanasiou c9834bc826 [ML] Require max time aggregation to ensure correct datafeed restart (elastic/x-pack-elasticsearch#948)
Before this change, aggregation datafeeds used the histogram bucket
key as the record timestamp that is posted to the job. That meant
that the latest_record_timestamp at the end of a datafeed run was
the start of the latest seen histogram bucket. Upon continuing the
datafeed, the search starts from one millisecond after the
latest_record_timestamp. Hence, data may be fetched for a second time.

This change requires a max aggregation on the time_field nested in
the histogram bucket. It then reads the timestamp from that agg.
This ensures datafeed can restart without duplicating data.

relates elastic/x-pack-elasticsearch#874

Original commit: elastic/x-pack-elasticsearch@f820efa866
2017-04-04 17:15:44 +01:00
Zachary Tong 2153c71e8f Wait for datafeed index to hit yellow before proceeding (elastic/x-pack-elasticsearch#940)
relates elastic/x-pack-elasticsearch#783

Original commit: elastic/x-pack-elasticsearch@838482e645
2017-04-04 16:14:02 +00:00
Dimitris Athanasiou 4fdcedb9f9 [ML] Enable datafeed preview with future data (elastic/x-pack-elasticsearch#949)
Original commit: elastic/x-pack-elasticsearch@7f9e450edc
2017-04-04 17:08:01 +01:00
Dimitris Athanasiou 38706faec7 [ML] Improve datafeed notifications (elastic/x-pack-elasticsearch#951)
- include 'real-time' instead of now as the end time for real-time
  datafeeds
- do not notify lookback is completed when datafeed was stopped
- do not notify datafeed switch to real-time when datafeed was stopped

Relates elastic/x-pack-elasticsearch#878

Original commit: elastic/x-pack-elasticsearch@aa22f9b86f
2017-04-04 17:04:41 +01:00
Jason Tedor dee1bd1825 Respond to rename random ASCII helper methods
This commit is response to the renaming of the random ASCII helper
methods in ESTestCase. The name of this method was changed because these
methods only produce random strings generated from [a-zA-Z], not from
all ASCII characters.

Relates elastic/x-pack-elasticsearch#942

Original commit: elastic/x-pack-elasticsearch@a6085964d3
2017-04-04 11:04:37 -04:00
Martijn van Groningen 71149fb6d4 [ML] Fix NPE
Original commit: elastic/x-pack-elasticsearch@421491bce0
2017-04-04 16:15:56 +02:00
Martijn van Groningen 543319f966 [ML] Small cleanup
Original commit: elastic/x-pack-elasticsearch@78a99258dd
2017-04-04 16:10:09 +02:00
David Kyle 778e960d3e [ML] Remove periodic persistence of datacounts (elastic/x-pack-elasticsearch#944)
Original commit: elastic/x-pack-elasticsearch@756f06d316
2017-04-04 14:45:15 +01:00
Igor Motov eb79be392c Persistent Tasks: check the current state in waitForPersistentTaskStatus (elastic/x-pack-elasticsearch#935)
Add a check for the current state waitForPersistentTaskStatus before waiting for the next one. This fixes sporadic failure in testPersistentActionStatusUpdate test.

relates elastic/x-pack-elasticsearch#928

Original commit: elastic/x-pack-elasticsearch@0db4ac92d2
2017-04-04 09:44:56 -04:00
David Kyle 5716b2bf16 [ML] Create job index before adding job to cluster state (elastic/x-pack-elasticsearch#943)
Original commit: elastic/x-pack-elasticsearch@1b586c7763
2017-04-04 14:25:09 +01:00
Martijn van Groningen 8a87a91897 [ML] Renamed DatafeedJobRunner to DatafeedManager
Original commit: elastic/x-pack-elasticsearch@1228488a2e
2017-04-04 14:55:58 +02:00
Tanguy Leroux 4f1115d7f5 [Test] Reenable Monitoring Bulk tests (elastic/x-pack-elasticsearch#908)
This commit reenables the Monitoring Bulk Api REST tests. The XPackRestIT
now enables/disables the local default exporter before executing the monitoring
 tests, and also waits for the monitoring service to be started before executing
 the test.

Original commit: elastic/x-pack-elasticsearch@10b696198c
2017-04-04 14:44:40 +02:00
Martijn van Groningen 0f9bd3a08d [ML] Removed unused vars / params.
Original commit: elastic/x-pack-elasticsearch@5d3280dbec
2017-04-04 14:21:27 +02:00
David Roberts 1465711762 [ML] Wait for state processing to complete before log processing (elastic/x-pack-elasticsearch#946)
State processing can take a lot longer than log processing, even after
the C++ process has closed its end of the pipe.  The pipe has a buffer,
and indexing the state document(s) in that buffer can take more than a
second.

relates elastic/x-pack-elasticsearch#945

Original commit: elastic/x-pack-elasticsearch@65f5075028
2017-04-04 12:04:01 +01:00
Dimitrios Athanasiou edb8c543ed [ML] Downgrade missing snapshot/quantiles log msg to warn
Original commit: elastic/x-pack-elasticsearch@fb763757d1
2017-04-03 22:18:05 +01:00
Martijn van Groningen 77dff92bef [ML] Use PersistentTasksService#waitForPersistentTaskStatus(...) to wait for job and datafeed status and
use PersistentTasksService#removeTask(...) to force close job and force stop datafeed.

Original commit: elastic/x-pack-elasticsearch@4abcf99f93
2017-04-03 20:40:31 +02:00
David Kyle 5b66c7a7ba [ML] Set job create time on server (elastic/x-pack-elasticsearch#910)
* [ML] Set job create time on server

* Job.Builder serialisation tests

* Make setCreateTime package private

Original commit: elastic/x-pack-elasticsearch@d2d75e0d7b
2017-04-03 18:30:47 +01:00
Dimitris Athanasiou 758b689f51 [ML] Improve validations for datafeed with aggregations (elastic/x-pack-elasticsearch#917)
Adds following validations:

- aggregations must contain date_histogram or histogram at the top level
- a date_histogram has to have its time_zone to UTC (or unset which
  defaults to UTC)
- a date_histogram supports calendar intervals only up to 1 week
to avoid the length variability of longer intervals
- aggregation interval must be greater than zero
- aggregation interval must be less than or equal to the bucket_span

Original commit: elastic/x-pack-elasticsearch@404496a886
2017-04-03 17:55:26 +01:00
Alexander Reelsen 84f08ee02c Tests: Ensure test does not accidentally create similar data structures
Original commit: elastic/x-pack-elasticsearch@e683de4039
2017-04-03 16:55:06 +01:00
Martijn van Groningen f0364b1218 [TEST] increased logging
Original commit: elastic/x-pack-elasticsearch@6aeac48889
2017-04-03 15:45:30 +02:00
David Roberts c0c0818d87 [ML] Fix query of all buckets with expand=true (elastic/x-pack-elasticsearch#932)
Previously if you queried all buckets with the expand=true option
you'd get an error as the code would try to do a search with
from=10000&size=10000.  This PR fixes this problem.

Followup for elastic/x-pack-elasticsearch#832

Original commit: elastic/x-pack-elasticsearch@2a5ca0998c
2017-04-03 14:15:35 +01:00
Martijn van Groningen 51662c83eb [TEST] If bulk item failed then log why.
Original commit: elastic/x-pack-elasticsearch@e7cc96460d
2017-04-03 13:49:43 +02:00
David Kyle 622c5ae166 [ML] Remove JobManagers dependency on JobResultsPerister (elastic/x-pack-elasticsearch#915)
* Remove JobManagers dependency on JobResultsPerister

* Remove unneeded call to refresh the state index

Original commit: elastic/x-pack-elasticsearch@0b2351bba7
2017-04-03 12:00:52 +01:00
David Roberts 2d01c3884b [ML] Don't get stats for jobs that are being deleted (elastic/x-pack-elasticsearch#899)
If jobs are being deleted then the operations required to get stats
could fail with unexpected exceptions.  When stats for multiple jobs
were being requested, this would previously cause the whole operation
to fail.

This commit changes the stats endpoint to ignore jobs that are being
deleted.

Fixes elastic/prelert-legacy#837

Original commit: elastic/x-pack-elasticsearch@6ac141a987
2017-04-03 11:04:34 +01:00
David Roberts c5a2fba70a [ML] Ignore ResourceNotFoundException executing C++ process update (elastic/x-pack-elasticsearch#927)
When this happens it means the job has been deleted, which in turn means
the C++ process has been stopped, so there's no need to send it a message
and hence no problem worth logging a stack trace for.

This differs from elastic/x-pack-elasticsearch#896 because elastic/x-pack-elasticsearch#896 was for a similar situation with
closed jobs, whereas this one is for deleted jobs.

Original commit: elastic/x-pack-elasticsearch@9bb4e98fe7
2017-04-03 10:53:32 +01:00
David Kyle 30745b8dd7 [ML] Wait for model snapshot update when closing the results processor (elastic/x-pack-elasticsearch#916)
* [ML] Wait for model snapshot update when closing the results processor

* Review comments

Original commit: elastic/x-pack-elasticsearch@eeb6c3b8d8
2017-04-03 10:01:21 +01:00
Tanguy Leroux 73ddc2323b [Test] Relax LocalExporterTests on node_stats checks
The test is too rigid on checking the right number of node_stats documented that are collected. It happens if a node takes time to start, the node_stats count % numNodes will always be different than 0.

It also adds more logging for LocalBulk failures.

Original commit: elastic/x-pack-elasticsearch@1ebb20b6f6
2017-04-03 10:36:12 +02:00
Simon Willnauer 5e6bfb9a82 Fix cross cluster search with security (elastic/x-pack-elasticsearch#904)
This commit adds an integration test that runs basic cross cluster search
actions across 2 clusters with security installed. This commit also fixes
several issues with respect to internal actions and proxy actions in the context
of cross cluster search.

Relates to elastic/elasticsearch#23830
relates elastic/x-pack-elasticsearch#892

Original commit: elastic/x-pack-elasticsearch@2e5486c259
2017-04-03 09:53:48 +02:00
Igor Motov 4115336f5a Persistent Tasks: require correct allocation id for status updates (elastic/x-pack-elasticsearch#923)
In order to prevent tasks state updates by stale executors, this commit adds a check for correct allocation id during status update operation.

Original commit: elastic/x-pack-elasticsearch@b94eb0e863
2017-04-01 18:17:07 -04:00
Tanguy Leroux b5a285fd83 [Monitoring] Update LocalExporterTests (elastic/x-pack-elasticsearch#835)
This commit changes the LocalExporterTests so that it now test
various randomized cases in a single test. This should speed up
the test as well as minimize the failures due to multiple start
/stop of the exporter. It also uses the MonitoringBulk API
instead of calling the Exporter instances, which makes more sense
since it is the normal way to index monitoring documents.

Related  elastic/x-pack-elasticsearch#416

Original commit: elastic/x-pack-elasticsearch@f8a4af15cd
2017-04-01 10:42:09 +02:00
Chris Earle 16d963b8e3 [Monitoring] Use 'index_patterns' instead of 'template' (elastic/x-pack-elasticsearch#883)
Original commit: elastic/x-pack-elasticsearch@f1a217b597
2017-03-31 17:17:02 -04:00
Igor Motov d7e4390490 Persistent Tasks: Add waitForPersistentTaskStatus method (elastic/x-pack-elasticsearch#901)
This method allows to wait for tasks to change their status to match the supplied predicate.

Original commit: elastic/x-pack-elasticsearch@9f5d4104a0
2017-03-31 16:05:34 -04:00
Jim Ferenczi 232190df97 Fix field_caps handler expectations
Original commit: elastic/x-pack-elasticsearch@628b96bf4b
2017-03-31 18:49:11 +02:00
Jim Ferenczi 2b9ea83eca Add new field_caps API to the list of known actions and handlers
Original commit: elastic/x-pack-elasticsearch@126b369e1e
2017-03-31 16:52:43 +02:00
David Kyle 461bf314fa [ML] Clear bulk requests after execution (elastic/x-pack-elasticsearch#912)
Original commit: elastic/x-pack-elasticsearch@31a4c7e99a
2017-03-31 15:37:01 +01:00
Martijn van Groningen 878c7a4eb9 [ML] Set the correct close timeout
Original commit: elastic/x-pack-elasticsearch@aa0245cc28
2017-03-31 14:01:49 +02:00
David Kyle 9da2853d12 [ML] Hide AnalysisConfig.latency field if not set (elastic/x-pack-elasticsearch#895)
Original commit: elastic/x-pack-elasticsearch@b2b3567820
2017-03-31 12:08:44 +01:00
Alexander Reelsen d7f3a260fd HttpClient: Add support to limit response size (elastic/x-pack-elasticsearch#765)
In order to be stuck with big responses in the HttpClient, this commit
adds a maximum limit for the response. Defaults to 10MB, can be set to
50 MB max.

relates elastic/x-pack-elasticsearch#263

Original commit: elastic/x-pack-elasticsearch@05c449d4c1
2017-03-31 12:39:25 +02:00
David Roberts 95fd6777a0 [ML] Add 'xpack' into action names to match what the rest of X-Pack does (elastic/x-pack-elasticsearch#909)
Original commit: elastic/x-pack-elasticsearch@36be904ad9
2017-03-31 11:22:53 +01:00
David Roberts 23123b9219 [ML] Add extra Java-side detector validation (elastic/x-pack-elasticsearch#894)
Detector configs are validated both by our C++ and by our Java code.
If the C++ is stricter than the Java then error reporting is poor.
This commit adds two extra validation checks to the Java code that
were already present in the C++ validation.

relates elastic/x-pack-elasticsearch#856

Original commit: elastic/x-pack-elasticsearch@bd4ce2377c
2017-03-31 11:06:30 +01:00
David Roberts 21bac70d87 [ML] Don't log a stack trace when updating a non-open process (elastic/x-pack-elasticsearch#896)
It's possible for a C++ process to exit between the time when a
config update message for it is queued and the time that message
is processed.  This commit ensures we don't spam the log with a
stack trace in this situation, as it's not a problem at all.

relates elastic/x-pack-elasticsearch#891

Original commit: elastic/x-pack-elasticsearch@81af8eaf70
2017-03-31 10:56:46 +01:00
Dimitris Athanasiou 0cb2b18265 [ML] Implement cancellation policy for aggregated datafeeds (elastic/x-pack-elasticsearch#862)
Aggregated data extraction is done in 2 phases:

1. search
2. process response

The first phase cannot be currently cancelled. However, it usually
is the fastest of the two.

The second phase processes the histogram buckets in the search
response into flat JSON and then posts the result stream to the job.
This phase can be split into batches where a few buckets are posted
to the job at a time. Cancelling can then work between batches.

This commit changes the AggregationDataExtractor to process the
search response in batches. The definition of a batch is crucial
as it has to be short enough to allow for responsive cancelling,
yet long enough to minimise overhead due to multiple calls to the
post data action. The number of key-value pairs written by the
processor is a good candidate for a batch size measure. By testing,
1000 seems to be an effective number.

relates elastic/x-pack-elasticsearch#802

Original commit: elastic/x-pack-elasticsearch@ce3a172411
2017-03-31 10:15:43 +01:00
Martijn van Groningen cbfa5b5f0e [TEST] adjusted log levels in tests
Original commit: elastic/x-pack-elasticsearch@eb4e0c2450
2017-03-31 10:58:46 +02:00
Martijn van Groningen fb5ccc7d14 [TEST] increased log level
Original commit: elastic/x-pack-elasticsearch@a8ea83dca0
2017-03-31 10:44:06 +02:00
Martijn van Groningen b4bebfb12f [TEST] Increased timeout
Original commit: elastic/x-pack-elasticsearch@a7170e66dd
2017-03-31 10:34:51 +02:00
Martijn van Groningen aba4760b02 [ML] Fix native process threading protection and restructure ml threadpools
The native process can only handle one operation at a time, so in order the protect against multiple operation at a time (e.g. post data and flush or multiple post data operations) there should be protection in place to guarantee that at most only a single thread interacts with the native process. The current protection is broken when a job close is executed, more specifically the wait logic is broken here.

This commit changes the threading logic when interacting with the native process by using a custom `ExecutorService` that that uses a single worker thread from `ml_autodetect_process` thread pool to interact with the native process. Requests from the ml apis are initially being queued and this worker thread executes these requests one by one in the order they were specified.

Removed the general `ml` threadpool and replaced its usages with `ml_autodetect_process` or `management` threadpool.
Added a new threadpool just for (re)normalizer, so that these operations are isolated from other operations.

relates elastic/x-pack-elasticsearch#582

Original commit: elastic/x-pack-elasticsearch@ff0c8dce0b
2017-03-31 10:07:03 +02:00
Martijn van Groningen 7da4724b15 [ML] Start using AllocatedPersistentTask#updatePersistentStatus(...) instead of PersistentTasksService directly
Original commit: elastic/x-pack-elasticsearch@82a7db17e2
2017-03-31 08:28:40 +02:00
Martijn van Groningen 510be083de [TEST] Verify differently that job has closed due to invalid license
Original commit: elastic/x-pack-elasticsearch@70cc389ca7
2017-03-30 22:28:24 +02:00
Martijn van Groningen 93ee4104fb [TEST] increased logging
Original commit: elastic/x-pack-elasticsearch@4d5f35b9a6
2017-03-30 20:09:16 +02:00
Adrien Grand 52a7f956f4 Tests: Disable `ml/index_layout/Test unrelated index`.
Relates elastic/x-pack-elasticsearch#903

Original commit: elastic/x-pack-elasticsearch@097f559d0b
2017-03-30 19:00:50 +02:00
Simon Willnauer 3abe7a2058 Fix compilation issues after elastic/elasticsearch#23788
Original commit: elastic/x-pack-elasticsearch@ea24170b41
2017-03-30 15:00:41 +02:00
Martijn van Groningen f6d8f92d40 [ML] Use volatile boolean to stop blackhole process
Original commit: elastic/x-pack-elasticsearch@3788c7f0cf
2017-03-30 14:54:21 +02:00
Alexander Reelsen 5b2351fad0 Watcher: Improve assertion output in flaky test
The slack tests seem to fail periodically with not output
This commit tries to add some more verbose output by
making the query more broad and take failures into account
to uncover, what happens in this test.

Relates elastic/x-pack-elasticsearch#836

Original commit: elastic/x-pack-elasticsearch@e601b3a0df
2017-03-30 14:50:55 +02:00
Alexander Reelsen dfc21c56be Watcher: Fix BasicSecurityTests.testWatcherMonitorRole
The test tried to create a watches index, that could have been
created by the test infrastructure already (to check if aliases
are supported).

relates elastic/x-pack-elasticsearch#837

Original commit: elastic/x-pack-elasticsearch@29714b2368
2017-03-30 14:03:37 +02:00
David Roberts f49dee89cb [TEST] Adjust expected error message
This was forgotten in elastic/x-pack-elasticsearch@2dbe41caaa

Original commit: elastic/x-pack-elasticsearch@8f7c4c32f4
2017-03-30 11:56:36 +01:00
Dimitris Athanasiou d33bce2ed2 [ML] Add job closing notification (elastic/x-pack-elasticsearch#893)
relates elastic/x-pack-elasticsearch#878

Original commit: elastic/x-pack-elasticsearch@921c25537a
2017-03-30 11:53:00 +01:00
Dimitris Athanasiou 12fd8e04e5 [ML] Allow marking a model snapshot with retain (elastic/x-pack-elasticsearch#880)
This change adds a retain field to model snapshots.
A user can set retain to true/false via the update model snapshot API.
Model snapshots with retain set to true will not be deleted by
the daily maintenance service regardless of whether they expired.

This allows users to keep always keep certain snapshots around for
potentially reverting to in the future.

relates elastic/x-pack-elasticsearch#758

Original commit: elastic/x-pack-elasticsearch@2283989a33
2017-03-30 11:48:36 +01:00
Dimitris Athanasiou 379b800c9f [ML] Handle usage call when MlMetadata is not yet installed (elastic/x-pack-elasticsearch#890)
Original commit: elastic/x-pack-elasticsearch@495286d4ba
2017-03-30 10:55:36 +01:00
David Roberts 72f5e92972 [ML] Adjust error to make sense for FAILED as well as CLOSED jobs (elastic/x-pack-elasticsearch#888)
Previously trying to perform an operation on a FAILED job would tell
you that it hadn't been opened.  This was very confusing if you knew
very well you'd just opened the job.

Relates elastic/x-pack-elasticsearch#879

Original commit: elastic/x-pack-elasticsearch@2dbe41caaa
2017-03-30 10:33:56 +01:00
Martijn van Groningen 9f63dba26c [ML] No need to create an InternalClient instance, because it can be injected.
Original commit: elastic/x-pack-elasticsearch@0cca0cd569
2017-03-30 09:11:46 +02:00
Martijn van Groningen f2654b5872 [ML] Changed job and datafeed lifecycle management
* Removed OPENING and CLOSING job states. Instead when persistent task has been created and
   status hasn't been set then this means we haven't yet started, when the executor changes it to STARTED we have.
   The coordinating node will monitor cs for a period of time until that happens and then returns or times out.
* Refactored job close api to go to node running job task and close job there.
* Changed unexpected job and datafeed exception messages to not mention the state and instead mention that job/datafeed haven't yet started/stopped.

Original commit: elastic/x-pack-elasticsearch@37e778b585
2017-03-29 20:35:10 +02:00
Chris Earle 905fa16f81 [Monitoring] Add .monitoring-alerts-2 Index Template (elastic/x-pack-elasticsearch#819)
Add the `.monitoring-alerts-2` index template via the exporter. This
avoids a very common problem where the user wipes out their monitoring
indices manually, which means that the watches would then create an index
with a dynamic mappings.

This adds a mechanism for posting a template that is not associated with a
Resolver (convenient for the forthcoming work _and_ for a future Logstash
index).

Original commit: elastic/x-pack-elasticsearch@a4cfc48191
2017-03-29 13:24:22 -04:00
Igor Motov 4bb9e00dbf Persistent Tasks: Remove unused stopped and removeOnCompletion flags (elastic/x-pack-elasticsearch#853)
The stopped and removeOnCompletion flags are not currently used, this commit removes them for now to simplify things.

Original commit: elastic/x-pack-elasticsearch@c636c2817e
2017-03-29 12:54:44 -04:00
Zachary Tong 0d202d3d36 Bump ChunkedDataExtractor logging down to DEBUG
Related elastic/x-pack-elasticsearch#866

Original commit: elastic/x-pack-elasticsearch@f510101be2
2017-03-29 12:16:36 -04:00
David Roberts 979d232faa [ML] Set jobs to FAILED state if C++ process dies unexpectedly (elastic/x-pack-elasticsearch#876)
Previously a `kill -9` on the `autodetect` process associated with a
job would leave the job in the OPENED state.

Now if the C++ process dies before a request to close the job is made
then the job state is set to FAILED.

For this purpose C++ process death is defined as end-of-file on the
log stream.  (Technically it would be possible to get end-of-file on
the log stream while the C++ process was still running, but this
would also represent an unexpected and undesirable situation.)

Original commit: elastic/x-pack-elasticsearch@2b74c56a79
2017-03-29 16:23:58 +01:00
Robin Clarke de061597fa Fix grammar in users tool help message
This commit fixes some grammar in the additional help provided by the
users tool.

Relates elastic/x-pack-elasticsearch#826

Original commit: elastic/x-pack-elasticsearch@49c86576d6
2017-03-29 10:42:19 -04:00
David Kyle a85b18e70b Don’t write the usePerPartitionNormalization field unless it is set to true (elastic/x-pack-elasticsearch#873)
Original commit: elastic/x-pack-elasticsearch@db7cd6993d
2017-03-29 15:09:14 +01:00
Daniel Mitterdorfer e5b7148fee Silence ExpirationCallbackTests#testPreExpirationScheduleTime()
Relates elastic/x-pack-elasticsearch#869

Original commit: elastic/x-pack-elasticsearch@fcba6847a0
2017-03-29 09:00:35 +02:00
David Roberts 7eece908b2 [ML] Remove obsolete TODO
It has been decided that filebeat will be used to index ES logs

Original commit: elastic/x-pack-elasticsearch@98f2169429
2017-03-28 16:58:41 +01:00
Martijn van Groningen 2357afd91b [TEST] Use search api to retrieve data counts instead of via get job stats api.
This to avoid to lose data counts when the job gets restarted on another node.
The job stats api returns live data counts, which may not have been persisted to an index,
so getting the data counts via search api will give us a better guarantee that when
the job gets restarted the datacounts are there too. During job restart a get call is being
done to get data counts in the order to initialize the job.

Original commit: elastic/x-pack-elasticsearch@901952da85
2017-03-28 17:10:25 +02:00
David Kyle 42e3eb7ba5 [ML] Use default number of replicas (elastic/x-pack-elasticsearch#861)
Original commit: elastic/x-pack-elasticsearch@7d3d00d196
2017-03-28 15:50:16 +01:00
Tanguy Leroux 642bdb6cd9 [Test] Mute IndicesStatsTests
See https://github.com/elastic/x-pack-elasticsearch/issues/496

Original commit: elastic/x-pack-elasticsearch@80aa60bc27
2017-03-28 16:29:57 +02:00
Colin Goodheart-Smithe b5dadc733b Make Filter actions extend HandledTransportAction (elastic/x-pack-elasticsearch#859)
Previously the GET/PUT/DELETE filters actions were master node actions. This is not necessary since the filters are stored in an index rather than the cluster state. This change makes the actions extend `HandledTransportAction` so they can be run on any node.

The change also makes PutFilterAction.TransportAction use the TransportBulkAction instead of the deprecated TransportIndexAction.

relates elastic/x-pack-elasticsearch#756

Original commit: elastic/x-pack-elasticsearch@c6df04382e
2017-03-28 15:02:39 +01:00
Tanguy Leroux b642ba6351 [Monitoring] Make MonitoringDoc immutable (elastic/x-pack-elasticsearch#785)
This commit makes the MonitoringDoc immutable and removes the type() and id() methods from "resolvers" so that they are not anymore in charge of computing the documents types and ids. Now each MonitoringDoc knows its type and is able to compute its own id if needed.

Original commit: elastic/x-pack-elasticsearch@5161cedcc8
2017-03-28 15:20:08 +02:00
David Roberts 70f621ed7e Remove duplicate actions from transport actions list
Original commit: elastic/x-pack-elasticsearch@da1226ee46
2017-03-28 11:10:44 +01:00
David Kyle fca0feb02d Tidy up (elastic/x-pack-elasticsearch#854)
Original commit: elastic/x-pack-elasticsearch@701407ecf5
2017-03-28 09:56:54 +01:00
Colin Goodheart-Smithe 99db6013ea [ML] Adds a test for ml transport client (elastic/x-pack-elasticsearch#851)
This change also adds synchronous methods to `MachineLearningClient`.

relates elastic/x-pack-elasticsearch#567

Original commit: elastic/x-pack-elasticsearch@b3a4b38a51
2017-03-28 09:05:55 +01:00
Jason Tedor eac00c6a9d Mark x-pack plugin as having native controller
This commit marks the x-pack plugin as having a native controller. This
is now a requirement in core for any plugin that forks a native process
to display a warning to the user when they install the plugin.

Relates elastic/x-pack-elasticsearch#839

Original commit: elastic/x-pack-elasticsearch@3529250023
2017-03-27 15:53:03 -04:00
Igor Motov 9fc6ce83ee Persistent Tasks: Merge NodePersistentTask and RunningPersistentTask (elastic/x-pack-elasticsearch#842)
Refactors NodePersistentTask and RunningPersistentTask into a single AllocatedPersistentTask. Makes it possible to update Persistent Task Status via AllocatedPersistentTask.

Original commit: elastic/x-pack-elasticsearch@8f59d7b819
2017-03-27 14:21:01 -04:00
David Kyle 9b4d399fc3 [ML] Fix bug deleting job with missing alias (elastic/x-pack-elasticsearch#850)
Original commit: elastic/x-pack-elasticsearch@44fd88a834
2017-03-27 17:47:29 +01:00
Alexander Reelsen f3c4ec8a81 Watcher: Allow multiple email addresses for to/cc/bcc (elastic/x-pack-elasticsearch#661)
All of our code supports configuring email addresses in the
email action not only via a JSON array, but also via a comma
separated value (we also have tests for this). However in one bit
we did not support this, where an email template is rendered to
a concrete email.

This commit fixes the last piece, so that users will be able to
specify comma separated email adresses.

The main use case for this is having an array of email addresses,
that can be joined in mustache with a comma in order to send to
several recipients.

Original commit: elastic/x-pack-elasticsearch@19794ba612
2017-03-27 16:56:43 +02:00
Martijn van Groningen fc29fd159a [TEST] Don't lose the actual cause of the exception.
Original commit: elastic/x-pack-elasticsearch@8720288ef5
2017-03-27 15:33:04 +02:00
Dimitrios Athanasiou c0a6b6741e [ML] Improve notification message for shard failures in datafeed
relates elastic/x-pack-elasticsearch#804

Original commit: elastic/x-pack-elasticsearch@2114f94937
2017-03-27 13:52:18 +01:00
Dimitris Athanasiou 27a313c07d [ML] Correctly initialise model size stats on job reopen (elastic/x-pack-elasticsearch#822)
This commit ensures that upon reopening a job, the in-memory
model size stats are correctly initialized from the ones
last persisted in the results index.

This fixes the bug that could be seen upon opening a job
that has processed data and immediately calling its _stats
API only to see the model size stats are zero.

In addition, this PR refactors getting the parameters needed to
open an autodetect job:

- Previously, there was a method chaining together multiple
callbacks to the job provider.
- These methods were retrieving data via GETs which is not
going to work with index rollover.

Note, this PR is not eliminating all GETs. More work is needed
to fully support index rollover.

relates elastic/x-pack-elasticsearch#801

Original commit: elastic/x-pack-elasticsearch@1ef1d44b32
2017-03-27 13:17:44 +01:00
Dimitris Athanasiou 9cd21cca4b [ML] Use default number_of_shards for results and state indices (elastic/x-pack-elasticsearch#849)
Like other x-pack features, these indices are best created with
default number_of_shards. Users can adjust the setting by editing
the respective templates.

relates elastic/x-pack-elasticsearch#723

Original commit: elastic/x-pack-elasticsearch@1d688993be
2017-03-27 13:08:19 +01:00
Alexander Reelsen 374be8b732 Watcher: Support arrays in http response payload (elastic/x-pack-elasticsearch#793)
The xcontent parser was only set to read all data to a map
which did not work, when the returned data was in form of an
array (for example the cat API is doing this, if the response
format is set to JSON).

relates elastic/x-pack-elasticsearch#351

Original commit: elastic/x-pack-elasticsearch@08ad457bf6
2017-03-27 10:22:22 +02:00
Martijn van Groningen 6709b43b97 [ML] Remove redundant comment
Original commit: elastic/x-pack-elasticsearch@7d3d1fdfb5
2017-03-27 09:05:11 +02:00
Martijn van Groningen ea273ab67e [ML] Made fields volatile or final
Original commit: elastic/x-pack-elasticsearch@c9478e2da4
2017-03-24 19:36:12 +01:00
Zachary Tong 4285335dfb [ML] Add a 'force' parameter to CloseJob and StopDataFeed endpoints (elastic/x-pack-elasticsearch#710)
If forced, the internal RemovePersistentTasks API is invoked instead of going through
ML.  This will remove the task, which should trigger the task framework to do
necessary cleanup.

At that point, the Delete* APIs interpret a missing task as CLOSED/STOPPED,
so they can be removed regardless of the original state.

Original commit: elastic/x-pack-elasticsearch@bff23c7840
2017-03-24 13:10:20 -04:00
Martijn van Groningen 1d660f373f [ML] Only push live update if job is opened
Original commit: elastic/x-pack-elasticsearch@5318d65012
2017-03-24 16:32:38 +01:00
Colin Goodheart-Smithe 60a3fbff85 [TEST] Roce ML post data test to use just JSON
This is temporary will we investigate why the test fails when SMILE is used

Original commit: elastic/x-pack-elasticsearch@55c7983a21
2017-03-24 15:23:51 +00:00
David Roberts f816a5708c [ML] Rename action now it's run by _xpack instead of _system (elastic/x-pack-elasticsearch#831)
The _xpack user cannot run internal actions

Original commit: elastic/x-pack-elasticsearch@2dc4a30582
2017-03-24 15:20:20 +00:00
Zachary Tong d33cbaf9be Remove unnecessary norelease
Original commit: elastic/x-pack-elasticsearch@a3bd568a72
2017-03-24 10:28:58 -04:00
Colin Goodheart-Smithe fd54515ecb [ML] Support all XContent types in Data API (elastic/x-pack-elasticsearch#812)
* [ML] Support all XContent types in Data API

This changes the POST Data API so that it accepts all XContent types instead of just JSON.

For now the datafeed is restricted to only sending JSON to the POST data API.

* Rename SimpleJsonRecordReader to XContentRecordReader

Also renames `DataFormat.JSON` to `DataFormat.XCONTENT`

* fixes YAML tests

Original commit: elastic/x-pack-elasticsearch@5fd20690b8
2017-03-24 12:57:02 +00:00
Dimitris Athanasiou 73434dadeb [ML] Remove recursion in bucket expansion (elastic/x-pack-elasticsearch#832)
Relates elastic/x-pack-elasticsearch#752

Original commit: elastic/x-pack-elasticsearch@85392296a5
2017-03-24 12:51:22 +00:00
Dimitrios Athanasiou 28d6b505ed [ML] Minor refactoring to improve code readability
Original commit: elastic/x-pack-elasticsearch@ef2d5fcab6
2017-03-24 10:52:38 +00:00
Martijn van Groningen c85498462c [ML] Replaced norelease with regular todo
Original commit: elastic/x-pack-elasticsearch@c4c8955c3f
2017-03-24 11:46:34 +01:00
Dimitris Athanasiou 1cfd9bd986 [ML] Add ml jobs and datafeeds in usage API (elastic/x-pack-elasticsearch#805)
This adds basic info about jobs and datafeeds, sufficient
for the first release for supporting monitoring and
phone-home.

In particular, usage now displays:

- job count for _all and by state
- detectors statistics (min, max, mean, total) for _all jobs and by job state
- model size statistics (min, max, mean, total) for _all jobs and by job state
- datafeed count for _all and by state

Relates elastic/x-pack-elasticsearch#660

Original commit: elastic/x-pack-elasticsearch@6e0da6c3db
2017-03-24 10:42:12 +00:00
Colin Goodheart-Smithe ea67089cef [ML] Cleans up some NORELEASES (elastic/x-pack-elasticsearch#830)
This change cleans up some NORELEASE comments that are either no longer relevant or actually should be TODO comments

Original commit: elastic/x-pack-elasticsearch@9947f1176e
2017-03-24 10:11:05 +00:00
Dimitrios Athanasiou cc5c34e037 [TEST] Fix open_time assertions
Original commit: elastic/x-pack-elasticsearch@a26425970c
2017-03-24 09:53:29 +00:00
Martijn van Groningen 96f4a72825 [ML] Change how update job api delegates config changes to autodetect process
Submit job updates to a concurrent queue when job update has been processed by ClusterService. Then from a background thread delegate the job updates to the node running the autodetect process. This maintains the same order as how the job config updates have occurred to the cluster state and thus preventing job config updates to the same job to arrive in the wrong order to the job's autodetect process. (the expectation is that in practise this will rarely happen)

The behaviour of the update api changes with this pr, because the api now returns when the update has been made to cluster state, whereas before it would return when the update was made to the autodetect process too. Updating the autodetect process happens in the background. I think that this change in behaviour is acceptable.
Use TP#scheduleWithFixedDelay(...) instead of TP#schedule(...) and
removed the custom rescheduling and cancelling.
Also changed LocalNodeMasterListener#executorName to SAME

Original commit: elastic/x-pack-elasticsearch@c24c0dd7d7
2017-03-24 09:30:55 +01:00
Hendrik Muhs f687f3ea6e [ML] fix test mix up of millisecond with second (elastic/x-pack-elasticsearch#816)
Fix spurious Test failure

fix test resolution bug: mix up of millisecond with second

Original commit: elastic/x-pack-elasticsearch@37b8fed559
2017-03-24 08:11:38 +01:00
Dimitrios Athanasiou 061e3fc1fe [TEST] fix open_time in jobs_get_stats YAML test
Original commit: elastic/x-pack-elasticsearch@2a1c62ab3c
2017-03-23 19:33:26 +00:00
Martijn van Groningen 1bed557911 [ML] Fork on management thread when really opening the job
Original commit: elastic/x-pack-elasticsearch@e528912c23
2017-03-23 19:48:11 +01:00
Dimitris Athanasiou 99e3508267 [ML] Change chunking_config.time_span into a TimeValue (elastic/x-pack-elasticsearch#808)
Original commit: elastic/x-pack-elasticsearch@42d7b06e3f
2017-03-23 18:45:53 +00:00
Zachary Tong e739d86f00 Replace norelease with TODO
Not a blocker, issue opened under elastic/x-pack-elasticsearch#821

Original commit: elastic/x-pack-elasticsearch@d46f6bc556
2017-03-23 13:14:10 -04:00
Zachary Tong c680b79641 Remove norelease
Issue elastic/x-pack-elasticsearch#820 filed to track progress of better error handling, but not a blocker so removing
the norelease

Original commit: elastic/x-pack-elasticsearch@82cf6eebe2
2017-03-23 13:07:50 -04:00
Zachary Tong cf6bca96db Remove norelease and adjust audit logging
Audit logging in wrong location.  Norelease removed because the proper solution
will take some more work, and this edge-case is not likely to be run into frequently.

More details in elastic/x-pack-elasticsearch#818

Original commit: elastic/x-pack-elasticsearch@ab0d731abf
2017-03-23 13:01:37 -04:00
Igor Motov fba5c09e3d Persistent Tasks: remove task restart on failure (elastic/x-pack-elasticsearch#815)
If a persistent task throws an exception, the persistent tasks framework will no longer try to restart the task. This is a temporary measure to prevent threshing the cluster with endless restart attempt. We will revisit this in the future version to make the restart process more robust. Please note, however, that if node executing the task goes down, the task will still be restarted on another node.

Original commit: elastic/x-pack-elasticsearch@30712e0fbf
2017-03-23 12:56:48 -04:00
Hendrik Muhs 6f7f466fa3 [ML] move DataStreamDiagnostics into DataCountsReporter (elastic/x-pack-elasticsearch#775)
repair DataStreamDiagnostics

Moves DataStreamDiagnostics into DataCountsReporter to survive if job is opened/closed/fed in chunks.

relates elastic/x-pack-elasticsearch#764

Original commit: elastic/x-pack-elasticsearch@29c221a451
2017-03-23 16:43:51 +01:00
Martijn van Groningen bb9befcdcb [ML] Include MlMetadata in all xcontent contexts.
Original commit: elastic/x-pack-elasticsearch@e870b5bf6d
2017-03-23 16:10:24 +01:00
Colin Goodheart-Smithe dde0570296 [ML] Removes direct dependency on Jackson Parser (elastic/x-pack-elasticsearch#810)
* [ML] Removes direct dependency on Jackson Parser

The classes that read data from the post data action ready to write to the autodetect process had a direct dependency on Jackson’s `Parser` class. This changes makes those classes depend on XContent instead making them consistent with the way we parse requests and data across Elasticsearch and X-Pack.

* Simplify json record reader

This commit removes the unnecessary `AbstractJsonRecordReader` and `JsonRecordReader` interfaces/classes. These are not required as we do and should only have one implementation of reading json records.

Original commit: elastic/x-pack-elasticsearch@366b8af943
2017-03-23 13:56:32 +00:00
David Roberts 7756067e5d [ML] Check underlying index permissions on datafeed PUT (elastic/x-pack-elasticsearch#809)
Also added a test to ensure underlying index permissions are required
for datafeed preview.

Relates elastic/x-pack-elasticsearch#648

Original commit: elastic/x-pack-elasticsearch@6edadbb401
2017-03-23 13:56:04 +00:00
Martijn van Groningen 93d7b8c14b [ML] State check doesn't need to know if task is stale now that task validation is only done at create time.
Original commit: elastic/x-pack-elasticsearch@d19858240b
2017-03-23 14:40:20 +01:00
Martijn van Groningen df86125a7d [TEST] Adjust assertions.
Original commit: elastic/x-pack-elasticsearch@39b289a4d9
2017-03-23 11:34:39 +01:00
Colin Goodheart-Smithe 795a00ddab Removes unnecessary NORELEASE for ML endpoint
Original commit: elastic/x-pack-elasticsearch@bbb203665b
2017-03-23 08:47:37 +00:00
Tim Vernum 3e0f785f0c Add ldap_dn and ldap_groups to user meta-data (elastic/x-pack-elasticsearch#739)
In the LdapRealm, include the user's DN and the list of groups (the DN for each group) in the User object
This allows this information to be referenced in templated roles.

Closes: elastic/x-pack-elasticsearch#729

Original commit: elastic/x-pack-elasticsearch@065f337109
2017-03-23 15:38:57 +11:00
Zachary Tong cc7a1a821f [ML] Load resource and split into keys manually (elastic/x-pack-elasticsearch#786)
Manually loads the resources, splits the lines and fills the map instead of using the ResourceBundle stuff. 

relates elastic/x-pack-elasticsearch#750 

Original commit: elastic/x-pack-elasticsearch@25ff4d5d19
2017-03-22 15:24:47 -04:00
Igor Motov 5c4933f5ea Make PersistentAction independent from TransportActions (elastic/x-pack-elasticsearch#742)
Removes the transport layer dependency from PersistentActions, makes PersistentActionRegistry immutable and rename actions into tasks in class and variable names.

Original commit: elastic/x-pack-elasticsearch@e3e5b79c28
2017-03-22 14:02:30 -04:00
Martijn van Groningen d779bf66a5 [ML] Also serialize the job id in a update job request.
Relates to elastic/x-pack-elasticsearch#787 and elastic/x-pack-elasticsearch#799

Original commit: elastic/x-pack-elasticsearch@fc64d25bcb
2017-03-22 18:09:10 +01:00
Martijn van Groningen 6ddc626c17 [ML] Never restart job locally upon cancel
Relates to elastic/x-pack-elasticsearch#749

Original commit: elastic/x-pack-elasticsearch@6943d75178
2017-03-22 17:09:19 +01:00
Martijn van Groningen f20efa7152 [ML] Renamed action names from */anomaly_detectors/* to */job/*
The action name is currently inconsistent with the class name.
Also as more types of jobs will be added, they should reuse the existing actions,
so it make more sense to gave a generic job name for the action name. Note that
it is ok to have additional rest handlers for new type of jobs.

Original commit: elastic/x-pack-elasticsearch@a70d46fb8a
2017-03-22 16:47:27 +01:00
Martijn van Groningen a59badf842 [ML] Restore model state as part of opening a job.
Before the model state was restored in the background when the job was already opened.

 relates elastic/x-pack-elasticsearch#751

Original commit: elastic/x-pack-elasticsearch@44a3d98365
2017-03-22 16:33:03 +01:00
Jay Modi 85aedb6776 Cancel LDAP runnables if they are pending execution too long (elastic/x-pack-elasticsearch#727)
The LdapRealm submits runnables to the generic thread pool when a bind is necessary as a bind must
be a synchronous operation and we do not want to block network threads on this. However, the
generic threadpool could be full and this runnable could get queued. When this happens requests
appear to hang; to prevent this we submit a delayed executable that will stop the runnable from
attempting to connect to Ldap and provide an exceptional response to the listener.

relates elastic/x-pack-elasticsearch#716

Original commit: elastic/x-pack-elasticsearch@3e43b17f1d
2017-03-22 05:27:46 -07:00
Jay Modi 8ba6e8b3eb Fix merging of field level security rules (elastic/x-pack-elasticsearch#796)
This commit fixes the merging of field level security rules from multiple roles. Prior to 5.2, the
merging was treated as the merging of languages whereas after 5.2, this incorrectly became a merge
of all rules meaning a single wildcard could cause denials to be ignored.

Original commit: elastic/x-pack-elasticsearch@42f9e6d8b0
2017-03-22 05:10:30 -07:00
Martijn van Groningen 3f5d9850ae [ML] Removed public constructors for testing.
Original commit: elastic/x-pack-elasticsearch@a5d45f2b66
2017-03-22 10:52:18 +01:00
Martijn van Groningen c12d487c01 [ML] Make fields final
Original commit: elastic/x-pack-elasticsearch@49c7d79835
2017-03-22 10:46:59 +01:00
Ryan Ernst 8a8ed1cfbb Plugins: Update jarhell calls with new set based api (elastic/x-pack-elasticsearch#743)
This is the xpack side of elastic/elasticsearch#23596

Original commit: elastic/x-pack-elasticsearch@d1654b1ccd
2017-03-21 12:13:37 -07:00
Dimitris Athanasiou f7c4c754c2 [ML] Accept 'now' in start/end params of start datafeed API (elastic/x-pack-elasticsearch#784)
The params start/end of the start datafeed API now accept
'now' as a value.

Also adds a validation that start must be earlier than end.

relates elastic/x-pack-elasticsearch#781

Original commit: elastic/x-pack-elasticsearch@5396dcb5e8
2017-03-21 17:05:36 +00:00
Jay Modi 72248adcbb Asynchronous export of monitoring data (elastic/x-pack-elasticsearch#718)
This commit removes the blocking invocation of bulk requests in monitoring as in some cases this
can lead to exhaustion of the generic threadpool, which effectively prevents the node from
operating normally.

One behavior change that is made by this commit is the MonitoringService will no longer wait
indefinitely when closing as this can lead to a node blocking forever and never shutting down.
Instead a wait of 10 seconds is added, which aligns with the security index audit trail's behavior
on shutdown.

relates elastic/x-pack-elasticsearch#715

Original commit: elastic/x-pack-elasticsearch@5ba7f49aab
2017-03-21 12:15:13 -04:00
Dimitris Athanasiou 58827dd433 [ML] Do not advance time when datafeed encounters extraction problems (elastic/x-pack-elasticsearch#782)
Original commit: elastic/x-pack-elasticsearch@fcc9af6869
2017-03-21 14:50:10 +00:00
Martijn van Groningen cb5d44c9c4 [ML] Don't assign cancel holders to node tasks, but use the services that stop datafeed and jobs directly.
Relates to elastic/x-pack-elasticsearch#749

Original commit: elastic/x-pack-elasticsearch@ff074b0442
2017-03-21 15:30:13 +01:00
Martijn van Groningen 6fae1867ec [ML] Uses AbstractRunnable for running datafeed, so that we handle rejected execution exceptions.
Original commit: elastic/x-pack-elasticsearch@d04d3fa401
2017-03-21 14:18:09 +01:00
Martijn van Groningen dd91ac0cd3 [ML] Don't use default locale.
Original commit: elastic/x-pack-elasticsearch@f1e5016008
2017-03-21 13:28:11 +01:00
Martijn van Groningen 4f5b3dfea3 [ML] Set ml' tp queue sizes to a reasonable level.
Original commit: elastic/x-pack-elasticsearch@7f7b2defcd
2017-03-21 13:09:46 +01:00
Dimitris Athanasiou 62721c78ea [ML] Make ModelSnapshot immutable (elastic/x-pack-elasticsearch#770)
In preparation of extending the model snapshot update API,
it makes sense to make it immutable to avoid any confusion
that could be caused by having setters on it.

I also like immutable stuff :-)

Original commit: elastic/x-pack-elasticsearch@6b2ee527a8
2017-03-21 12:00:54 +00:00
Martijn van Groningen 7cf959cb66 [ML] renamed getter as it overwrote super method that was used for a different purpose.
Original commit: elastic/x-pack-elasticsearch@a8b70e769b
2017-03-21 11:24:41 +01:00
Alexander Reelsen ce0391f3c7 Watcher: Update dependencies (elastic/x-pack-elasticsearch#613)
Updated dependencies for javax.activation and mail to latest versions.

Original commit: elastic/x-pack-elasticsearch@d73529f98f
2017-03-21 10:28:20 +01:00
Alexander Reelsen 0c7c2f521c Watcher: Remove scheduler based trigger engine (elastic/x-pack-elasticsearch#724)
The scheduler based trigger engine is not enabled by default
as the ticker based trigger engine is used. As we dont use it
in production, this commit removes this specific implementation.

It also removes some uneeded abstractions like AbstractTriggerEngine, TriggerEngine.Listener and TriggerEngine.Job

Original commit: elastic/x-pack-elasticsearch@b17a2e9d62
2017-03-21 10:27:41 +01:00
Jay Modi 1a7e842c15 Make XPackFeatureSet#usage calls asynchronous (elastic/x-pack-elasticsearch#738)
This commit makes the XPackFeatureSet#usage calls asynchronous. Previously these were synchronous
calls that would execute a multi-search request from the currently elected master node in a blocking
fashion. The multi-search request is now executed asynchronously.

relates elastic/x-pack-elasticsearch#213

Original commit: elastic/x-pack-elasticsearch@a0cb988442
2017-03-20 14:23:04 -04:00
Boaz Leskes e3d244164c testSendMessage: log message request/response on failures
Original commit: elastic/x-pack-elasticsearch@591284dced
2017-03-20 19:09:16 +01:00
Dimitris Athanasiou 025461c3c8 [ML] Correctly collect datafeed stats for _all (elastic/x-pack-elasticsearch#776)
Datafeed state was always stopped when the stats
endpoint was called with _all. The reason is that
_all was not being expanded into all datafeed IDs.

This commit fixes the issue by expanding _all into
all datafeed IDs in the cluster. Stats are then fetched
for all of them.

relates elastic/x-pack-elasticsearch#693

Original commit: elastic/x-pack-elasticsearch@ccbdb35b6e
2017-03-20 17:51:36 +00:00
David Roberts f96a40c61a [ML] Rename score fields (elastic/x-pack-elasticsearch#769)
* Rename normalized_probability to record_score
* Rename anomaly_score on influencers to influencer_score
* Remove record level anomaly_score
* Remove bucket level max_normalized_probability

Relates elastic/x-pack-elasticsearch#678

Original commit: elastic/x-pack-elasticsearch@c2c05e73d8
2017-03-20 16:19:52 +00:00
Martijn van Groningen 47e28669b3 [TEST] Fix expected error message
Original commit: elastic/x-pack-elasticsearch@5a60b66b4a
2017-03-20 15:07:13 +01:00
Martijn van Groningen 2c8545f38f [TEST] Don't rely on `new Date()` (2)
Relates elastic/x-pack-elasticsearch#746

Original commit: elastic/x-pack-elasticsearch@f1ccdf89f0
2017-03-20 13:19:30 +01:00
Dimitrios Athanasiou 3e8b45f2a2 Revert "[TEST] Muted failing tests, see elastic/x-pack-elasticsearch#771"
This reverts commit elastic/x-pack-elasticsearch@014ae38d5b.

Original commit: elastic/x-pack-elasticsearch@c34a5bcc60
2017-03-20 12:06:14 +00:00
Martijn van Groningen 67e4544267 [TEST] making failing restrictions tighter.
Original commit: elastic/x-pack-elasticsearch@b22352a5ba
2017-03-20 12:27:53 +01:00
Martijn van Groningen 06277c3677 [ML] Upon job close validate if datafeed has been stopped
and fail otherwise.

Original commit: elastic/x-pack-elasticsearch@d13bfe1041
2017-03-20 12:03:52 +01:00
Martijn van Groningen 06ce981fed [TEST] Don't rely on `new Date()`
relates elastic/x-pack-elasticsearch#746

Original commit: elastic/x-pack-elasticsearch@11879958b6
2017-03-20 09:20:05 +01:00
David Roberts aa02fa85da [ML] Reduce caching time for native code dependency to 0 (elastic/x-pack-elasticsearch#772)
Prior to this change the integration tests could fail for up to 24
hours after a change to the C++ output format

Original commit: elastic/x-pack-elasticsearch@b8d9fa0adb
2017-03-19 15:58:19 +00:00
Hendrik Muhs 7832783122 [ML] Rename model plot elastic/x-pack-elasticsearch#284 (elastic/x-pack-elasticsearch#753)
Rename model_debug_output to model_plot

The model debug output is 1 result type of the autodetect (anomaly detection) process reporting bounds/statistics/state - not anomalies - It's main usage as of time of writing is visualization in the UI, naming the 'blue shadow' under the result graph. Rename in order to remove 'debug' in the name

Original commit: elastic/x-pack-elasticsearch@86de428512
2017-03-19 11:55:06 +01:00
Martijn van Groningen a76232f5e4 [TEST] Muted failing tests, see elastic/x-pack-elasticsearch#771
Original commit: elastic/x-pack-elasticsearch@014ae38d5b
2017-03-18 12:26:07 +01:00
Dimitris Athanasiou ca931f4b39 [ML] Default model_snapshots_retention_days to 1 (elastic/x-pack-elasticsearch#759)
relates elastic/x-pack-elasticsearch#722

Original commit: elastic/x-pack-elasticsearch@e68aaf229c
2017-03-17 17:56:46 +00:00
David Kyle 30ea37943c [ML] Change delimiter in the names of state docs (elastic/x-pack-elasticsearch#768)
Original commit: elastic/x-pack-elasticsearch@34acda3ffe
2017-03-17 17:18:52 +00:00
Martijn van Groningen 9d4e3efad4 [ML] Fix deleteCategorizerState() by using delete by query instead of individual blocking delete requests.
Original commit: elastic/x-pack-elasticsearch@f28cc9fc5b
2017-03-17 11:54:31 +01:00
David Kyle a7d42f3732 Delete state in the JobStorageDeletionTask (elastic/x-pack-elasticsearch#702)
Original commit: elastic/x-pack-elasticsearch@272b0894c5
2017-03-17 09:37:28 +00:00
Ali Beyad 407cc3a9d7 Only allow platinum licenses to use custom role providers (elastic/x-pack-elasticsearch#748)
This commit restricts custom role providers to only apply to those
instances of x-pack with a platinum level license.  All other license
types will not be allowed to use custom role providers.  Any custom role
providers implemented via the XPackExtension will not take effect unless
the license is platinum.

relates elastic/x-pack-elasticsearch#720

Original commit: elastic/x-pack-elasticsearch@4fc35494ee
2017-03-16 13:47:35 -04:00
Dimitris Athanasiou 16efd4e474 [ML] Use TimeValue in job and datafeed configs (elastic/x-pack-elasticsearch#732)
relates elastic/x-pack-elasticsearch#679

Original commit: elastic/x-pack-elasticsearch@891edd5bfe
2017-03-16 14:07:47 +00:00
Dimitris Athanasiou 6c9727c2db [ML] Change default data_description to JSON (elastic/x-pack-elasticsearch#707)
relates elastic/x-pack-elasticsearch#674

Original commit: elastic/x-pack-elasticsearch@efc5f4a269
2017-03-16 11:48:57 +00:00
Tanguy Leroux 803654d9bf [Test] Mute MonitoringBulkTests
Related issue: https://github.com/elastic/x-pack-elasticsearch/issues/353

Original commit: elastic/x-pack-elasticsearch@d2600652e7
2017-03-16 09:57:32 +01:00
Ryan Ernst 8c01d6ea69 Tests: Add cluster health check to xpack integ wait conditions (elastic/x-pack-elasticsearch#740)
The wait condition used for integ tests by default calls the cluster
health api with wait_for_nodes nd wait_for_status. However, xpack
overrides the wait condition to add auth, but most of these conditions
still looked at the root ES url, which means the tests are susceptible
to race conditions with the check and node startup. This change modifies
the url for the authenticated wait condtion to check the health api,
with the appropriate wait_for_nodes and wait_for_status.

Original commit: elastic/x-pack-elasticsearch@0b23ef528f
2017-03-15 10:23:26 -07:00
Jay Modi 149e251445 Filtered RestRequests should allow access to the remote address (elastic/x-pack-elasticsearch#741)
When adding support for rest request filtering of sensitive content, the overridden rest request
did not properly delegate the #getRemoteAddress method to the wrapped request. This resulted in a
NPE when a filtered rest request was created and an audit record needed to be generated.

relates elastic/x-pack-elasticsearch#714

Original commit: elastic/x-pack-elasticsearch@710b43355b
2017-03-15 07:13:09 -06:00
Jay Modi bc53dbfb46 Remove unused sign user setting (elastic/x-pack-elasticsearch#736)
User signing was removed with the move to TLS only transport and this change removes a no longer
used setting.

Original commit: elastic/x-pack-elasticsearch@c221f1f4a4
2017-03-14 19:29:16 -07:00
David Roberts 0b7c735aec [ML] Add machine learning privileges/roles (elastic/x-pack-elasticsearch#673)
* Changed ML action names to allow distinguishing of admin and read-only actions
  using wildcards
* Added manage_ml and monitor_ml built-in privileges as subsets of the existing
  manage and monitor privileges
* Added out-of-the-box machine_learning_admin and machine_learning_user roles
* Changed machine learning results endpoints to use a NodeClient rather than an
  InternalClient when searching for results so that index/document level permissions
  applied to ML results are respected

Original commit: elastic/x-pack-elasticsearch@eee800aaa8
2017-03-14 16:13:41 +00:00
Jay Modi 8df7a82435 Remove unnecessary String creation from password char[] (elastic/x-pack-elasticsearch#713)
This commit removes an unnecessary String creation from the char[] of a password and instead uses
a byte[] that is cleared after it is used to prevent the password bytes from sticking around in
memory longer than required.

Original commit: elastic/x-pack-elasticsearch@1154a68965
2017-03-13 18:50:48 -07:00
Dimitris Athanasiou db48e92f54 [ML] Add support for percentiles aggregation (elastic/x-pack-elasticsearch#733)
Original commit: elastic/x-pack-elasticsearch@7c8357ac04
2017-03-13 23:32:33 +00:00
Steffen Siering 07bde45671 Init beats monitoring support (elastic/x-pack-elasticsearch#543)
- introduce builtin user 'beats_system'
- init monitoring-beats ES mapping
- add beats to MonitoredSystem + Resolver

Original commit: elastic/x-pack-elasticsearch@4d7b45d54d
2017-03-13 10:29:42 -07:00
Tanguy Leroux e5cc038616 Remove unused import
Original commit: elastic/x-pack-elasticsearch@742c12e8b8
2017-03-12 13:36:39 -07:00
Tanguy Leroux 73836f43c5 [Test] Mute flaky monitoring tests
Original commit: elastic/x-pack-elasticsearch@3670e827c8
2017-03-12 13:31:05 -07:00
Hendrik Muhs ce1a910542 [ML] DataStreamDiagnostics Rebase (elastic/x-pack-elasticsearch#698)
DataStreamDiagnostics

DataStreamDiagnostics analyzes input data regarding machine learning fit. It checks whether data is sane/plausible as anomaly detection on broken data (or misconfiguration).

Original commit: elastic/x-pack-elasticsearch@2f37d3c960
2017-03-12 09:40:27 -07:00
Jason Tedor 0457de27dd Fix SHAs for Netty dependencies
This commit fixes the SHA hashes for the Netty dependencies after Netty
was upgraded from version 4.1.8 to version 4.1.9.

Original commit: elastic/x-pack-elasticsearch@ca4c376886
2017-03-12 01:19:17 -05:00
Alexander Reelsen 54527b86ea Tests: Remove DynamicIndexNameIntegrationTests (elastic/x-pack-elasticsearch#725)
This integration tests only checks if date math index names are
supported. This is nothing that a watcher test should check for.

Original commit: elastic/x-pack-elasticsearch@8a06988fff
2017-03-11 17:35:29 -08:00
David Kyle b06ce7e7c1 [ML] Remove dead delete job index code
Original commit: elastic/x-pack-elasticsearch@533b2a0207
2017-03-10 16:30:02 -08:00
Dimitris Athanasiou 72f7698647 [ML] Extend aggregation support (elastic/x-pack-elasticsearch#705)
* [ML] Allow summary_count_field_name to be set to custom field

... in aggregated datafeeds.

This allows implementing e.g. distinct_count anomaly detection
with aggregations. (see case 1 in elastic/x-pack-elasticsearch#659)

Relates to elastic/x-pack-elasticsearch#659

* [ML] Handle multiple leaf aggregations

This commit allows multiple numeric leaf aggregations.
That enables use cases where the data are presummarised and
thus the summary count field is a sum aggregation on the
event rate field. (see 2 in elastic/x-pack-elasticsearch#659)

Relates to elastic/x-pack-elasticsearch#659

Original commit: elastic/x-pack-elasticsearch@07a34a18c6
2017-03-09 15:26:44 +00:00
David Roberts 1c52495c5a [ML] The Machine Learning class no longer needs to be a plugin (elastic/x-pack-elasticsearch#712)
Now machine learning is simply a feature of X-Pack, so I have made the
MachineLearning class more like the other feature classes. It no longer extends
Plugin and its createComponents() method takes an InternalClient rather than
a generic Client.

Original commit: elastic/x-pack-elasticsearch@704860147c
2017-03-08 15:54:26 +00:00
Martijn van Groningen e972f0216d [ML] Auto close jobs and auto stop datafeeds if license is invalid.
Original commit: elastic/x-pack-elasticsearch@6e2634b2b6
2017-03-05 11:14:45 +01:00
Martijn van Groningen 6365dec42e [TEST] Changed internal distributed ml tests to extend from ESIntegTest and only run with ml to avoid noise.
Changed DatafeedJobsIT to extend directly from SecurityIntegTest as it uses external cluster.

Original commit: elastic/x-pack-elasticsearch@d646c59491
2017-03-04 08:10:46 +01:00
Martijn van Groningen 49826ae134 [ML] Simplify JobManager#createJobResultIndex(...)
Original commit: elastic/x-pack-elasticsearch@e8b01e183c
2017-03-03 16:33:30 +01:00
Dimitrios Athanasiou 9e7d6d7fee [TEST] Add more assertions to DeleteExpiredDataIT
Original commit: elastic/x-pack-elasticsearch@ed34093921
2017-03-03 15:32:33 +00:00
Dimitrios Athanasiou f4c8dfce5a [TEST] Add missing parts in delete_expired_data REST spec
Original commit: elastic/x-pack-elasticsearch@3530ed1550
2017-03-03 13:07:42 +00:00
Dimitris Athanasiou a7a36245c9 [ML] Add end-point for deleting expired results (elastic/x-pack-elasticsearch#670)
This commit adds an end-point to force deletion of expired data:

DELETE /_xpack/ml/_delete_expired_data

A few other things are changed too:

- Delete expired results from now rather than start of day
- Rename MlDaily{Management -> Maintenance}Service
- Refresh job indices when job is closing to ensure latest result
  visibility
- Commit results when quantiles are persisted to ensure they are visible
  for renormalization


Original commit: elastic/x-pack-elasticsearch@8ca5272a94
2017-03-03 11:33:22 +00:00
David Roberts 440d005b1a [ML] Prevent repeated cluster state changes when creating index templates (elastic/x-pack-elasticsearch#687)
Switched the order of the local atomic flag check and cluster state check,
based on the observation that we used to sometimes get a storm of cluster
state updates on initial startup, due to a race between checking for the
presence of an index template and starting to create one.

Original commit: elastic/x-pack-elasticsearch@7ae83648ce
2017-03-03 10:59:24 +00:00
David Kyle 01de84a19f [ML] Wait for templates to be installed before running ML Integ tests (elastic/x-pack-elasticsearch#681)
* [ML] Wait for templates to be installed before running ML Integ tests

* Revert put job template check and preserve template change

* Review comments

Original commit: elastic/x-pack-elasticsearch@21deb34f4a
2017-03-03 10:04:34 +00:00
Alexander Reelsen be2ee21d16 Tests: Ensure date is UTC to prevent failing tests
This test used the local time zone to check for an assertion,
and thus failed in US evenings, because the date in UTC rolled over.

relates elastic/x-pack-elasticsearch#239

Original commit: elastic/x-pack-elasticsearch@6286b30407
2017-03-03 00:20:19 +01:00
Tim Vernum b105118ef0 Add `has_privileges` API (elastic/x-pack-elasticsearch#604)
Security API to determine which (if any) of a specified set of index/cluster privileges are held by the current (runAs) user.
Intended for use by Kibana to distinguish between read/write and read-only users, but should be applicable to other uses cases also.

Closes: elastic/x-pack-elasticsearch#282

Original commit: elastic/x-pack-elasticsearch@8b4cfdb858
2017-03-03 18:41:44 +11:00
Ryan Ernst c92562e9d9 Build: Enable notice generation for x-pack (elastic/x-pack-elasticsearch#695)
The only tricky thing here was what to do about ES jars. We now ignore them for the jar sha checking, and the base elasticsearch license and notice is copied here, and elasticsearch jars are mapped to those license/notice files.

Original commit: elastic/x-pack-elasticsearch@a6373cfe4e
2017-03-02 19:06:15 -08:00
Dimitrios Athanasiou c52689a9a0 [ML] Add job_id filter to job alias
This fixes returning results only for the job asked
when a shared index is used.

The commit also refactors the field count check to
use the field mappings API and solves a bug where the
check blows due to the _default_ type not having properties.

[Zach] Minor test tweak to MlJobIT

Original commit: elastic/x-pack-elasticsearch@729f886879
2017-03-02 21:46:34 -05:00
Martijn van Groningen 34a4778b11 [TEST] Wait a bit longer before the lookback import has completed and it stopped the datafeed automatically
Original commit: elastic/x-pack-elasticsearch@6f9b5f8f28
2017-03-02 22:00:11 +01:00
Martijn van Groningen e9e14cbcea [ML] Improve reason why datafeed is stopped.
Original commit: elastic/x-pack-elasticsearch@2055a64689
2017-03-02 18:03:54 +01:00
David Kyle d8f9b8452a [ML] Fix unit test
Original commit: elastic/x-pack-elasticsearch@fb6ff7be3f
2017-03-02 16:50:52 +00:00
David Kyle 14a0167781 [ML] Restore missing job stats (elastic/x-pack-elasticsearch#667)
* [ML] Pretty print dates in ModelSizeStats
* [ML] Add last_data_time field to DataCounts
* [ML] Add uptime to job stats
* [ML] Pretty print time fields in ModelSnapshot
* [ML] Rename uptime -> open_time

Original commit: elastic/x-pack-elasticsearch@4ce5258a77
2017-03-02 15:54:20 +00:00
David Roberts 2c365311af [ML] Document test problem and make it slightly less likely
Relates elastic/x-pack-elasticsearch#689

This does NOT fix the problem, but makes it clearer why it happens
and, by changing ordering, should make it happen less often

Original commit: elastic/x-pack-elasticsearch@9ef60073c2
2017-03-02 14:18:31 +00:00
Dimitris Athanasiou 746eb2ead4 [ML] Enable machine learning in x-pack (elastic/x-pack-elasticsearch#683)
Original commit: elastic/x-pack-elasticsearch@d718b6f7e8
2017-03-02 13:51:50 +00:00
Luca Cavanna 70a4a69e39 Fix indentation in some yaml tests (elastic/x-pack-elasticsearch#686)
The yaml test runner now throws error when skip or do sections are malformed, such as they don't start with the proper token (START_OBJECT). That signals bad indentation, which was previously ignored. Thanks (or due to) our pull parsing code, we were still able to properly parse the sections, yet other runners weren't able to.

Original commit: elastic/x-pack-elasticsearch@920201207c
2017-03-02 12:43:43 +01:00
Martijn van Groningen 0f5bfca69a [TEST] wait for yellow, to give ml indices time to get ready, open job api fails otherwise
Original commit: elastic/x-pack-elasticsearch@b1002cd289
2017-03-02 09:50:45 +01:00
Martijn van Groningen b5db0d9f44 [TEST] Change `DatafeedJobsIT` test to use the external cluster as it doesn't start/stop nodes.
Only distributed ml tests, only enable ml as security, watcher, monitoring etc. only add noise to the tests.
Disable mock big arrays for distributed ml tests, as nodes starting/stopping in tests somehow leaves allocations around,
which only is a test problem.

Original commit: elastic/x-pack-elasticsearch@5ff1e69036
2017-03-01 23:09:57 +01:00
Zachary Tong eed0e41a29 [ML] Share job result indices by default (elastic/x-pack-elasticsearch#638)
This moves the index structure to using a single, shared index
(.ml-anomalies-shared).  Custom indices can still be used by manually
setting `results_index`.

An alias is always created which points from `.ml-anomalies-<jobid>`
to `.ml-anomalies-shared`.

User defined indices are prepended with "custom-"

Index helper functions have been renamed to make this clear.  Furthermore,
accessing an index should always be done either by fetching the
currently configured index/alias from the state, or using the preconfigured
alias.  Because the user can specify a custom physical index, it is
impossible to determine the physical index "by convention" now.
The helpers have been configured to reflect that.

Original commit: elastic/x-pack-elasticsearch@a5368eb230
2017-03-01 13:28:12 -05:00
Ali Beyad b20578b9f6 Enables X-Pack extensions to implement custom roles providers (elastic/x-pack-elasticsearch#603)
This commit adds the ability for x-pack extensions to optionally
provide custom roles providers, which are used to resolve any roles
into role descriptors that are not found in the reserved or native
realms.  This feature enables the ability to define and provide roles
from other sources, without having to pre-define such roles in the security
config files.

relates elastic/x-pack-elasticsearch#77

Original commit: elastic/x-pack-elasticsearch@bbbe7a49bf
2017-03-01 12:20:34 -05:00
Jay Modi 02579c7acc IteratingActionListener should store context before calling consumers (elastic/x-pack-elasticsearch#675)
As part of authentication, we use a iterating action listener to perform asynchronous authentication against the realm
chain. When this listener is called with a response or a failure, it could be called from a thread that is not owned by
the Elasticsearch threadpool such as a LDAPConnectionReader thread. When this happens, we need to ensure that the
ThreadContext is not left with items in it otherwise we leave behind things like Authentication and hit obscure errors.

This commit stores the context when the listener calls the consumer or onResponse/onFailure is invoked, which prevents
us from polluting a external thread's ThreadContext.

Original commit: elastic/x-pack-elasticsearch@0f50fb6c10
2017-03-01 10:40:42 -05:00
Martijn van Groningen 61ca6d435f [ML] Enforce the stop datafeed aoi to wait until any operation that the datafeed is doing has completed.
Original commit: elastic/x-pack-elasticsearch@e8974191a2
2017-03-01 15:14:41 +01:00
Alexander Reelsen e492b17c10 Watcher: Ensure all json builders use try-with-resources (elastic/x-pack-elasticsearch#650)
Some json builders in the codebase were not closed. even
though this is not needed for the BytesStreamOutput being used,
there is more closing logic in the jackson classes, which we
should not rely on, that those never change or are ok to not
close.

Original commit: elastic/x-pack-elasticsearch@05a43d80ff
2017-03-01 13:45:10 +01:00
David Kyle 4edfbe664f [ML] Delete unused AuditActivity class (elastic/x-pack-elasticsearch#672)
Original commit: elastic/x-pack-elasticsearch@185ceab4b4
2017-03-01 10:42:27 +00:00
David Roberts 094be3c28b [ML] Put C++ process/PID and file@line in square brackets (elastic/x-pack-elasticsearch#671)
This is the Elastic convention, and also makes it clearer where the
actual log message from the C++ starts

Original commit: elastic/x-pack-elasticsearch@dc9aeefb08
2017-03-01 09:33:30 +00:00
Martijn van Groningen ac93195407 [TEST] Don't install xpack plugin as the test doesn't rely on it initializes the ml classes it tests on its own.
Original commit: elastic/x-pack-elasticsearch@e505efcddb
2017-03-01 09:32:34 +01:00
Igor Motov 377c1ec2b4 Simplify names of PersistentTasks-related classes
PersistentTask -> NodePersistentTask
PersistentTasksInProgress -> PersistentTasks
PersistentTaskInProgress -> PersistentTask

Original commit: elastic/x-pack-elasticsearch@0947dbca56
2017-02-28 16:37:09 -07:00
Martijn van Groningen 9d6c344520 [ML] Disable ml in case a node is a tribe node and
in case of tribe node client do not even bother to check for native process version, because it isn't running.

Original commit: elastic/x-pack-elasticsearch@e6fdd5e82a
2017-02-28 21:31:09 +01:00
Martijn van Groningen 6783f823a8 [ML] Change stop datafeed api delegate to node hosting datafeed task and execute cancel locally,
instead of only removing the persistent task from cluster state.

Original commit: elastic/x-pack-elasticsearch@3974b20827
2017-02-28 16:36:13 +01:00
David Kyle 59b50bb18c [ML] Index template registry with simple versioning (elastic/x-pack-elasticsearch#655)
[ML] * Add MachineLearningTemplateRegistry class

[ML] * Add blocking method to put templates required by tests

[ML]  * Add version check for templates

[ML] * Review comments

Original commit: elastic/x-pack-elasticsearch@07d315e56d
2017-02-28 15:34:05 +00:00
Martijn van Groningen 6864111acc [ML] Fixed typo
Original commit: elastic/x-pack-elasticsearch@e66e7febe2
2017-02-28 14:56:43 +01:00
Martijn van Groningen b7b24ed877 [ML] Removed norelease: Don't specify an id for certain documents, but let ES generate an id.
This norelease can be removed, because we no longer create indices automatically when missing, but use index templates.

Original commit: elastic/x-pack-elasticsearch@b6880ce7b7
2017-02-28 14:38:57 +01:00
Martijn van Groningen 759f0b1281 [ML] Close job api should use provided client instead of transport actions directly
Original commit: elastic/x-pack-elasticsearch@cb92f24ee1
2017-02-28 13:44:21 +01:00
David Kyle fc35b6022f Fix checkstyle long lines
Original commit: elastic/x-pack-elasticsearch@7fa810292f
2017-02-28 09:42:36 +00:00
Martijn van Groningen 93a2a567cb [ML] Cut open job and start datafeed apis to use persistent task's assignment explanation.
Also included assignment explanation to both job and datafeed stats apis and
included executor node to datafeed stats api

Original commit: elastic/x-pack-elasticsearch@783bc77ef6
2017-02-28 10:38:51 +01:00
David Kyle 69880373fc Remove the ml_messages.properties file (elastic/x-pack-elasticsearch#657)
String constants are no in Messages

Original commit: elastic/x-pack-elasticsearch@ab8af83e10
2017-02-28 09:33:47 +00:00
Dimitris Athanasiou 2d27e8ad81 [ML] Warn when lookback only datafeed retrieved no data (elastic/x-pack-elasticsearch#651)
When the datafeed indexes are concrete indexes, the search
will throw IndexNotFoundException, thus a notification is being
generated. However, when the indexes are patterns, the search
does not fail.

This commit adds a warning when a lookback-only datafeed
retrieves no data as it is certain that the user misconfigured
something and it's useful to make him aware of it.

Original commit: elastic/x-pack-elasticsearch@e3490ea978
2017-02-27 14:18:07 +00:00
javanna adc9808eff Convert watcher search source to json
If any format other than json is sent, it should get converted to json, as watcher works under the assumption that the script is a template, hence json.

Original commit: elastic/x-pack-elasticsearch@554bcd306b
2017-02-27 12:28:04 +01:00
javanna 39ccc67a7b disable content-type randomization for some ml tests
post_data and get_model seem to support json only

Original commit: elastic/x-pack-elasticsearch@683f998228
2017-02-27 12:28:04 +01:00
javanna 33ccc3bd6c adapt to ObjectPath changes to support binary formats
Original commit: elastic/x-pack-elasticsearch@11782418e9
2017-02-27 12:28:04 +01:00
Martijn van Groningen 212098fc7c [TEST] Attempt to stabilize distributed ml tests.
Relates to elastic/x-pack-elasticsearch#653

Original commit: elastic/x-pack-elasticsearch@e4919ab2b4
2017-02-27 11:11:19 +01:00
David Roberts 2ca1ddf017 [ML] Tell native controller process to stop during graceful shutdown (elastic/x-pack-elasticsearch#644)
Fixes elastic/prelert-legacy#803

On Windows the JVM hangs during shutdown if the native controller process
is still running. A thread dump suggests this is because of a finalizer being
blocked. We can avoid the problem by explicitly telling the controller to exit
when the node is stopped. Although the hang is Windows-specific, it doesn't
hurt to do this on *nix too.

Original commit: elastic/x-pack-elasticsearch@6fa1c2b966
2017-02-27 08:52:03 +00:00
Colin Goodheart-Smithe 4a81c85f71 Reapplying: Adds cast hint for eclipse
Eclipse errors because the direct supperclass of the two listeners is package private and it doesn’t have the smarts to look higher in the heirarchy. So we need to add a cast hint to force Eclipse to consider the elements as Listeners

This fix was mistakenly removed in another commit

Original commit: elastic/x-pack-elasticsearch@6305456d5a
2017-02-27 08:43:27 +00:00
Dimitris Athanasiou 380f6d9bff [ML] Add datafeed preview end-point (elastic/x-pack-elasticsearch#645)
Adds a preview end-point: `/_xpack/ml/datafeeds/{datafeed_id}/_preview`

The endpoint returns an array with the JSON records that will reach
writer to the c++ process. Thus, the preview can be used to verify
that a datafeed is configured correctly with regard to itself and
its corresponding job.

Original commit: elastic/x-pack-elasticsearch@1e0e9c906d
2017-02-25 10:17:06 +00:00
Dimitris Athanasiou d49930040e [TEST] Disable autodetect_process in tests not needing it (elastic/x-pack-elasticsearch#649)
Original commit: elastic/x-pack-elasticsearch@38897c0040
2017-02-24 18:57:33 +00:00
David Kyle 08b7cb6bca [ML] Refactor Auditor (elastic/x-pack-elasticsearch#631)
* Refactor Auditor
* Inject Auditor


Original commit: elastic/x-pack-elasticsearch@d9efe5fcef
2017-02-24 15:56:38 +00:00
Martijn van Groningen a189e1b759 [TEST] Added more failure scenarios
Original commit: elastic/x-pack-elasticsearch@a27134323e
2017-02-24 16:35:28 +01:00
David Kyle 8a3706f395 [ML] Influencers come after analysis fields (elastic/x-pack-elasticsearch#646)
Original commit: elastic/x-pack-elasticsearch@f37835b140
2017-02-24 15:12:35 +00:00
Martijn van Groningen 8ecdf3b288 [TEST] Attempt to reduce `MockPageCacheRecycler` assertion failures by waiting until there are no active recoveries.
Original commit: elastic/x-pack-elasticsearch@1b142f29bc
2017-02-24 15:23:02 +01:00
Martijn van Groningen 2c5e32f3d3 [ML] Removed workaround that task status named writable needed to have the same name as action name.
Original commit: elastic/x-pack-elasticsearch@99d9d80073
2017-02-24 14:32:53 +01:00
Martijn van Groningen cd4e543479 [ML] Tweaked starting datafeed logic, so that it will not start when job hasn't started.
[TEST] Unmuted ml full cluster restart test.

Original commit: elastic/x-pack-elasticsearch@c8cd9870c9
2017-02-24 13:47:44 +01:00
David Kyle f67d8c3746 [ML] Fix intermittent test failure (occasionally deleted an index we don’t care about)
Original commit: elastic/x-pack-elasticsearch@56c7bdb5b9
2017-02-24 12:20:47 +00:00
David Kyle a09e8a260a [ML] Use Index Templates (elastic/x-pack-elasticsearch#622)
* [ML] Index template for notifications, meta and state indices

* Add ignore_unavailable option to searches so missing indices don’t throw errors

* Test for job existence before returning results

* Template mapping for results indices

* Fix tests

* Reinstate merge conflict dropped code

* Fix and tidy up JobProviderTests

Tests had badly named & duplicated tests

* Don’t check if the job indices already exist when allocating

Indices are created on demand from templates

* No need to create indices in initialisation service

Using templates now

* Remove unused code and add comment

Original commit: elastic/x-pack-elasticsearch@1ba115d4fc
2017-02-24 10:20:07 +00:00
Colin Goodheart-Smithe 7f6edbdd4b Adds cast hint for eclipse
Eclipse errors because the direct supperclass of the two listeners is package private and it doesn’t have the smarts to look higher in the heirarchy. So we need to add a cast hint to force Eclipse to consider the elements as Listeners

Original commit: elastic/x-pack-elasticsearch@a7aff45571
2017-02-24 08:38:33 +00:00
Igor Motov 615dd0f240 Request and Status in Persistent Tasks should be serialized using their writable names
Refactors xcontent serialization of Request and Status to use their writable names instead of action name. That simplifies the parsing logic, allows reuse of the same status object for multiple actions and is consistent with how named objects in xcontent are used.

Original commit: elastic/x-pack-elasticsearch@3d059ea952
2017-02-23 18:55:38 -05:00
Martijn van Groningen 9f48a1f677 [ML] Added the node the job is running on to the response of the job stats api.
Original commit: elastic/x-pack-elasticsearch@d92149ca59
2017-02-23 20:48:40 +01:00
Jay Modi 405230c308 Do not add null listeners to the DnRoleMapper (elastic/x-pack-elasticsearch#636)
The DnRoleMapper was changed when moving to asynchronous authentication, which introduced a regression that leads to a
NPE being thrown on file refresh. The cause is the addition of a null listener from the LdapRealm upon construction.

This change removes the listener constructor argument as it was always null or a empty runnable and adds a check in the
addListener method to ensure the listener is not null. Finally, the DnRoleMapperTests had a bug where the temporary
filenames did not have the correct suffix and tests were not actually testing what was intended.

relates elastic/x-pack-elasticsearch#608

Original commit: elastic/x-pack-elasticsearch@f47f258590
2017-02-23 11:16:14 -07:00
Ali Beyad 4a001706e7 Simplifies security index state changes and template/mapping updates (elastic/x-pack-elasticsearch#551)
Currently, both the NativeUsersStore and NativeRolesStore can undergo
multiple state transitions. This is done primarily to check if the
security index is usable before it proceeds. However, such checks are
only needed for the tests, because if the security index is unavailable
when it is needed, the downstream actions invoked by the
NativeUsersStore and NativeRolesStore will throw the appropriate
exceptions notifying of that condition. In addition, both the
NativeUsersStore and NativeRolesStore had much duplicate code that
listened for cluster state changes and made the exact same state
transitions.

This commit removes the complicated state transitions in both classes
and enables both classes to use the SecurityTemplateService to monitor
all of the security index lifecycle changes they need to be aware of.

This commit also moves the logic for determining if the security index
needs template and/or mapping updates to the SecurityLifecycleService,
and makes the NativeRealmMigrator solely responsible for applying the
updates.

Original commit: elastic/x-pack-elasticsearch@b31d144597
2017-02-23 13:05:39 -05:00
jaymode e61f87d5ff Test: do not use the same name twice
In rare cases, the CertificateToolTests#testPromptingForInstanceInformation could try to use the same name multiple
times. This causes the test to fail with a message for an unknown prompt. This commit ensures no duplicates are added.

relates elastic/x-pack-elasticsearch#597

Original commit: elastic/x-pack-elasticsearch@ab8bb7ad50
2017-02-23 12:34:06 -05:00
Igor Motov db88cc458d Add persistent task assignment explanations.
This commit allows persistent actions to indicate why a task was or wasn't assigned to a certain node.

Original commit: elastic/x-pack-elasticsearch@cdacf9b10f
2017-02-23 11:22:08 -05:00
Martijn van Groningen d87926ab86 [ML] Never stop the datafeeder when no data can be found
Changed `ProblemTracker#updateEmptyDataCount(boolean)` into `ProblemTracker#reportEmptyDataCount()`.
and added `ProblemTracker#reportNoneEmptyCount()` to reset the empty count counter.

Original commit: elastic/x-pack-elasticsearch@c1a44d6fd3
2017-02-23 16:32:08 +01:00
Martijn van Groningen e7d56e92f8 [Ml] Rename `useNativeProcess` setting to `xpack.ml.autodetect_process` and un-deprecate it.
Original commit: elastic/x-pack-elasticsearch@eb2143e6f0
2017-02-23 15:29:44 +01:00
Dimitris Athanasiou 412d5e35b1 [ML] model_debug_config: add enabled and remove bounds_percentile (elastic/x-pack-elasticsearch#627)
This allows it to be disabled/enabled via job updates.
It also simplifies it by removing bounds_percentile as it currently
rarely makes sense to set the bounds to another value than 95.0.

Original commit: elastic/x-pack-elasticsearch@c27fce2d86
2017-02-23 14:11:23 +00:00
David Roberts 33b10970ee [ML] Temporary workaround to make some non-ml integration tests work (elastic/x-pack-elasticsearch#630)
This will be replaced with a better solution when elastic/x-pack-elasticsearch#566 is implemented

Original commit: elastic/x-pack-elasticsearch@bf0d9e0176
2017-02-23 11:13:58 +00:00
Dimitris Athanasiou d3d73ca095 [ML] Remove data that expired their respective retention period (elastic/x-pack-elasticsearch#610)
This PR introduces a service component that is scheduled to run
once per day, around 30 minutes past midnight, local time.
Listeners can be subscribed to the service in order to get notified
when the next service trigger occurs so that they can execute
management tasks.

Currently, there are 2 such listeners: one is performing removal
of results and the other of model snapshots that have expired
their retention period.

Finally, note that the service is only installed on the master
node to ensure there is only one running on the cluster.

Original commit: elastic/x-pack-elasticsearch@c8edb6016a
2017-02-23 11:06:14 +00:00
Martijn van Groningen a03e9d36c2 [TEST] mute test
Original commit: elastic/x-pack-elasticsearch@3989e5a074
2017-02-23 11:38:44 +01:00
Martijn van Groningen b40d5dde9b [ML] Before selecting an executor node for running job task, make sure that all required indices exist and are accessible.
Original commit: elastic/x-pack-elasticsearch@5459a0d5f2
2017-02-23 10:55:16 +01:00
Alexander Reelsen 43a09a2fa7 Watcher: Add license files (elastic/x-pack-elasticsearch#619)
Add license, notice and SHA sums for watcher compile dependencies
(excluding test deps).

Relates elastic/x-pack-elasticsearch#607

Original commit: elastic/x-pack-elasticsearch@06ce510436
2017-02-23 10:24:16 +01:00
Martijn van Groningen 2eba41cfac [TEST] Fix mocking bug
Original commit: elastic/x-pack-elasticsearch@61268c1180
2017-02-23 09:59:45 +01:00
Martijn van Groningen 6195000b8a [TEST] Change ml cleanup workaround.
Original commit: elastic/x-pack-elasticsearch@793d6cfb78
2017-02-23 09:08:10 +01:00
Tim Vernum 9f6285f362 [TEST] Waiting longer for native realm migration
Let LicensingTests.testNativeRealmMigratorWorksUnderBasicLicense wait for 3s (instead of 1s) as it is occasionally failing.
Also added logging so that if/when it does fail we can see how long we were waiting and what was taking place.

Original commit: elastic/x-pack-elasticsearch@22882ec412
2017-02-23 14:15:17 +11:00
Tim Vernum 672819f038 LICENSE files for security (elastic/x-pack-elasticsearch#614)
* License information for unboundid-ldapsdk
   From: https://docs.ldap.com/ldap-sdk/docs/LICENSE-UnboundID-LDAPSDK.txt

* License information for Bouncy Castle
   From:  java -cp ~/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.55/bcprov-jdk15on-1.55.jar org.bouncycastle.LICENSE


Original commit: elastic/x-pack-elasticsearch@b32d5aade3
2017-02-23 10:43:34 +11:00
Ryan Ernst bca7d87ac4 Rename threadpool time method
See elastic/elasticsearch#23175

Original commit: elastic/x-pack-elasticsearch@0e185c0a44
2017-02-22 15:11:42 -08:00
Dimitris Athanasiou eda2f15b44 [ML] Remove deprecated use of TransportDeleteAction (elastic/x-pack-elasticsearch#628)
Original commit: elastic/x-pack-elasticsearch@b5fdb32a0c
2017-02-22 22:14:32 +00:00
Martijn van Groningen 783a2cf01b [TEST] Added more test logging
Original commit: elastic/x-pack-elasticsearch@d68a1e3510
2017-02-22 21:47:01 +01:00
Ryan Ernst 2cd3979654 Merge pull request elastic/x-pack-elasticsearch#617 from rjernst/integ_test_order
Build: Convert integ test dsl to new split cluster/runner dsl

Original commit: elastic/x-pack-elasticsearch@454edd48d1
2017-02-22 12:43:25 -08:00
Martijn van Groningen 124fe0ad04 [ML] Fix line length
Original commit: elastic/x-pack-elasticsearch@2fba2434b8
2017-02-22 21:37:25 +01:00
polyfractal 4041545cfa [ML] Put back check for inconsistent job id
Lost during the refactor, this checks to make sure an ID in both the URL and the
body match.

Original commit: elastic/x-pack-elasticsearch@ef96275c99
2017-02-22 15:21:03 -05:00
Martijn van Groningen 9da18a14c5 [ML] Set createTime if not specified in validate api too
Original commit: elastic/x-pack-elasticsearch@477fb0fc52
2017-02-22 21:03:55 +01:00
polyfractal f856a79ef3 Fix method signature in test
Original commit: elastic/x-pack-elasticsearch@4340952f77
2017-02-22 14:08:19 -05:00
Zachary Tong 41811efcf1 [ML] Job's CreateTime must be non-null (elastic/x-pack-elasticsearch#616)
Also simplifies Job.Builder to only have one build() method.

Original commit: elastic/x-pack-elasticsearch@50e4b4c210
2017-02-22 13:45:13 -05:00
Martijn van Groningen 9a5b410110 [TEST] Add test that verifies that machine learning works correctly after full cluster restart.
Added missing named xcontent entries, tweaked start datafeed validation logic and fixed status serialization issue that were uncovered by this test.

Original commit: elastic/x-pack-elasticsearch@8b3fb012df
2017-02-22 17:25:08 +01:00
Dimitrios Athanasiou 4bb57137a9 fix unit test
Original commit: elastic/x-pack-elasticsearch@af9276f02f
2017-02-22 14:42:39 +00:00
Dimitrios Athanasiou a6c62f0717 [ML] Search model state docs using correct id
Original commit: elastic/x-pack-elasticsearch@70400101ad
2017-02-22 13:54:18 +00:00
David Kyle f06b4fbbaf [ML] remove duplicate code
Original commit: elastic/x-pack-elasticsearch@340480477c
2017-02-22 13:52:38 +00:00
Dimitrios Athanasiou b233c4729d [ML] Correctly fetch persistent tasks in JobUpdateAction
Original commit: elastic/x-pack-elasticsearch@440bfdd06e
2017-02-22 11:26:31 +00:00
David Roberts 614aeeeb53 [ML] Add back SuperCSV license/notice (elastic/x-pack-elasticsearch#618)
These were lost during the migration from prelert-legacy to x-pack

Original commit: elastic/x-pack-elasticsearch@da338d4a44
2017-02-22 10:49:26 +00:00
David Kyle f82634b6b1 [ML] Use ExceptionsHelper.convertToElastic to wrap exceptions in TransportJobTaskAction (elastic/x-pack-elasticsearch#577)
* Wrap exceptions if they are not ElasticsearchExceptions in TransportJobTaskAction

* Review comments

Original commit: elastic/x-pack-elasticsearch@07e900ffaa
2017-02-22 10:03:22 +00:00
Ryan Ernst 8527bc2415 Build: Convert integ test dsl to new split cluster/runner dsl
This is the xpack side of elastic/elasticsearch#23304

Original commit: elastic/x-pack-elasticsearch@8eddd7fb0d
2017-02-22 00:56:52 -08:00
polyfractal 45ecebac20 [Watcher] Fix tests due to upstream method signature change
Caused by elastic/elasticsearch#23288

Original commit: elastic/x-pack-elasticsearch@6f041708f9
2017-02-21 13:48:53 -05:00
Martijn van Groningen 9100fa6333 [ML] Rename `xpack.ml.allocation_enabled` boolean setting to `node.ml`.
Remove `node.attr.xpack.ml.allocation_enabled` node attribute and instead use `node.attr.max_running_jobs` attribute to indicate a node is a ml node.
(in addition to indicating how many jobs can hosted on a node)

Original commit: elastic/x-pack-elasticsearch@7979bc55b4
2017-02-21 14:10:50 +01:00
Dimitris Athanasiou 165c0d0e4f [ML] Move metadata package classes at the top (elastic/x-pack-elasticsearch#605)
Original commit: elastic/x-pack-elasticsearch@579f86e7b9
2017-02-21 13:08:16 +00:00
Dimitrios Athanasiou b9f83da5ac [ML] Include all fields in Job copy constructor
Fields were missing from the job's copy constructor.
This was the cause of intermittent test failures.

Original commit: elastic/x-pack-elasticsearch@3cd705ab3c
2017-02-21 12:49:08 +00:00
Martijn van Groningen 0e44683de6 [TEST] Change default close timeout for close job requests in tests.
Original commit: elastic/x-pack-elasticsearch@240bb0561c
2017-02-21 13:07:25 +01:00
Dimitrios Athanasiou c49a17e7ca [ML] Fix model snapshot YAML tests
Original commit: elastic/x-pack-elasticsearch@e29663da35
2017-02-21 11:42:24 +00:00
Dimitris Athanasiou 6c97ac7691 [ML] Remove model snapshot restore_priority and... (elastic/x-pack-elasticsearch#598)
...correctly update the job's model_snapshot_id whenever
a new model snapshot is persisted. Load the snapshot that
corresponds to the job's model_snapshot_id when a job is
opened.

This PR makes a series of changes that completes the change
of having job's active model snapshot persisted on the job.

Original commit: elastic/x-pack-elasticsearch@c2e23fa1ee
2017-02-20 17:45:28 +00:00
Alexander Reelsen b58b92dc79 Merge pull request elastic/x-pack-elasticsearch#591 from spinscale/1702-rest-hijack-delete-by-query
Delete/Update by query was not disabled for the watcher index
on HTTP.

Test suite was renamed from hijack to `block_direct_index_access` in the watcher tests

relates elastic/x-pack-elasticsearch#586

Original commit: elastic/x-pack-elasticsearch@5984ee3624
2017-02-20 13:24:00 +01:00
David Roberts 782f97a60f Relax regex for getting C++ build so it works for developers with uncommitted changes
Original commit: elastic/x-pack-elasticsearch@a6faea0cc6
2017-02-20 11:25:19 +00:00
Alexander Reelsen 7e8b43bfb1 Merge branch 'master' into 1702-rest-hijack-delete-by-query
Original commit: elastic/x-pack-elasticsearch@c206f84300
2017-02-20 12:06:55 +01:00
David Roberts 5233ab9831 Don't try to get the native code version in the transport client (elastic/x-pack-elasticsearch#600)
The native controller process won't be running

Original commit: elastic/x-pack-elasticsearch@d848e86cd5
2017-02-20 10:56:20 +00:00
Alexander Reelsen d2c705206c renamed test suite
Original commit: elastic/x-pack-elasticsearch@784d5ee68a
2017-02-20 11:43:03 +01:00
Martijn van Groningen 4b1ed5b453 [ML] Add a setting (`xpack.ml.node_concurrent_job_allocations`) to control the maximum concurrent job allocations.
A job allocation is either a job task in OPENING state or a job task that has been assigned to an executor node, but the executor node hasn't had the opportunity to set the job task status to OPENING.

In order to keep track of restarted tasks, `allocationIdOnLastStatusUpdate` field was added to `PersistentTaskInProgress` class.
This will allow persistent task implementors to detect whether the executor node has changed or has been unset since the last status update has occured.

Original commit: elastic/x-pack-elasticsearch@b7b85a8274
2017-02-20 11:22:19 +01:00
David Roberts 0d1181eabb [ML] Show C++ process info in X-Pack's info response (elastic/x-pack-elasticsearch#581)
The /_xpack endpoint now has a "native_code_info" value in the feature
info for ml

Original commit: elastic/x-pack-elasticsearch@6b4b408f4a
2017-02-18 20:14:52 +00:00
Martijn van Groningen 8a6cea0350 [ml] For the data, flush and update apis check the job state on the node running the job task instead of the coordinating node.
[test] Also re-enabled multi node qa tests.

Original commit: elastic/x-pack-elasticsearch@efb24131f0
2017-02-18 19:18:29 +01:00
Martijn van Groningen a74d1329ba [test] changed default close timeout
Original commit: elastic/x-pack-elasticsearch@f07cca869f
2017-02-18 15:40:13 +01:00
Martijn van Groningen 31c88ae1e5 [ml] Set job state to failed when fetching required information for opening a job failed.
Relates to elastic/x-pack-elasticsearch#596

Original commit: elastic/x-pack-elasticsearch@6ad699df86
2017-02-18 13:13:57 +01:00
Jay Modi 68324f5a50 Remove usage of deprecated methods that do not accept a XContentType (elastic/x-pack-elasticsearch#549)
This commit removes the usages and definition of deprecated methods that do not accept the XContentType for the source.

Additionally, usages of *Entity classes from the http client library have been changed to always specify the content
type.

Original commit: elastic/x-pack-elasticsearch@29d336a008
2017-02-17 14:45:00 -05:00
Martijn van Groningen 22022ea25f [ml] Fix start datafeed validation
relates elastic/x-pack-elasticsearch#596

Original commit: elastic/x-pack-elasticsearch@103fd7f2b4
2017-02-17 20:14:08 +01:00
David Kyle b9a4a2c621 Revert "Don’t wait on job close (elastic/x-pack-elasticsearch#574)"
This reverts commit elastic/x-pack-elasticsearch@99ed0e0dba and fixes the failing tests

Original commit: elastic/x-pack-elasticsearch@403e38316d
2017-02-17 17:05:41 +00:00
Zachary Tong 3fd1c90707 [ML] Move all Job validation to builder and make Job private (elastic/x-pack-elasticsearch#585)
All validation should be done in the builder and the Job object should just be a dumb, immutable blob.

Original commit: elastic/x-pack-elasticsearch@a78e6edbe9
2017-02-17 11:39:41 -05:00
Dimitrios Athanasiou fa9f4e3bca [ML] Remove leftoever debug statement
Original commit: elastic/x-pack-elasticsearch@a3f529b656
2017-02-17 16:16:48 +00:00
David Kyle 45d228dd41 [ML] Synchronise updates to running C++ process (elastic/x-pack-elasticsearch#557)
* Update running process from cluster state update

* Put a big lock on the UpdateProcessAction

* Protect concurrent update of the process with a semaphore

* Review comments

Original commit: elastic/x-pack-elasticsearch@d08b53247a
2017-02-17 16:00:46 +00:00
Martijn van Groningen 75470b4036 [ML] No need to remove the job task manually, when job task is cancelled the persistent task will be removed too.
Original commit: elastic/x-pack-elasticsearch@e0e3947a53
2017-02-17 14:58:00 +01:00
Alexander Reelsen 7d7da4262f Watcher: Hijack REST delete/udpate by query operations
Delete/Update by query was not disabled for the watcher index
on HTTP.

relates elastic/x-pack-elasticsearch#586

Original commit: elastic/x-pack-elasticsearch@b41b375379
2017-02-17 14:32:31 +01:00
Dimitris Athanasiou d970869e00 [ML] Add builder to JobUpdate (elastic/x-pack-elasticsearch#589)
Original commit: elastic/x-pack-elasticsearch@07a69eb407
2017-02-17 11:45:16 +00:00
David Kyle 48ee5e021d Don’t wait on job close (elastic/x-pack-elasticsearch#574)
Original commit: elastic/x-pack-elasticsearch@99ed0e0dba
2017-02-17 09:25:13 +00:00
Dimitris Athanasiou 4f6146cecc [ML] Improve logging on job closing (elastic/x-pack-elasticsearch#587)
Original commit: elastic/x-pack-elasticsearch@58d814fd71
2017-02-17 09:20:23 +00:00
Igor Motov d733aa684d Make persistent task persist full cluster restart
This commit moves persistent tasks from ClusterState.Custom to MetaData.Custom and adds ability for the task to remain in the metadata after completion.

Original commit: elastic/x-pack-elasticsearch@751b40de2a
2017-02-16 19:35:06 -05:00
Zach f4e2830ab1 [ML] Fix MlJobIT#testCreateJobInSharedIndex: "index_name" is now "results_index_name"
relates elastic/x-pack-elasticsearch#584

Original commit: elastic/x-pack-elasticsearch@fbf9fdd9b0
2017-02-16 18:21:13 -05:00
Igor Motov 11a46dd938 [ML] Suppress MlJobIT.testCreateJobInSharedIndexUpdatesMapping
Original commit: elastic/x-pack-elasticsearch@214ff88ac3
2017-02-16 14:24:32 -05:00
Dimitris Athanasiou 1ae8ed495e [ML] Add keyword mapping to audit message to enable aggs (elastic/x-pack-elasticsearch#583)
The messages stored in the .ml-notifications index include a
message field that is mapped to text. It makes sense to keep
that mapping as it can improve search experience in messages.
However, the ML kibana app needs to aggregate over messages,
thus we also need to provide a keyword mapping.

Original commit: elastic/x-pack-elasticsearch@5a5df53d85
2017-02-16 17:53:26 +00:00
Dimitris Athanasiou 6edc051e8d [ML] Close job after a lookback-only datafeed stops (elastic/x-pack-elasticsearch#580)
Original commit: elastic/x-pack-elasticsearch@febf026ba9
2017-02-16 17:48:57 +00:00
David Kyle 15987f49c0 [ML] Update index mapping for jobs in shared indices (elastic/x-pack-elasticsearch#572)
* Update input field mappings if job results index exists

* Add field names to the index mapping in shared indices

* Throw error if creating job will violate the index.mapping.total_fields.limit

* Review comments

Original commit: elastic/x-pack-elasticsearch@d7712d9263
2017-02-16 17:34:47 +00:00
jaymode 6ec6db8544 Test: handle case where multiple state_uuids are found in monitoring shards docs
Original commit: elastic/x-pack-elasticsearch@8d95776081
2017-02-16 11:34:17 -05:00
David Kyle c5506b7440 Consistent defaults for _open job (elastic/x-pack-elasticsearch#578)
Matches OpenJobAction

Original commit: elastic/x-pack-elasticsearch@24c07913a1
2017-02-16 15:41:20 +00:00
David Kyle 545921698c Update in use messages (elastic/x-pack-elasticsearch#576)
[ML] Update in use messages

Original commit: elastic/x-pack-elasticsearch@629961f1ed
2017-02-16 14:44:52 +00:00
jaymode ff96939c5f Authentication should set the proper version on the stream
This commit ensures that the authentication sets the correct version on the stream when it is serialized over the wire
so that there is not a version mismatch between the authentication and the connection it came from.

Original commit: elastic/x-pack-elasticsearch@267d7068f4
2017-02-16 08:44:29 -05:00
Martijn van Groningen 3572cff0a8 [TEST] slightly changed workaround cleanup for ml integ tests.
Original commit: elastic/x-pack-elasticsearch@3eba1e76ab
2017-02-16 14:28:06 +01:00
Martijn van Groningen 55b2111bdf [TEST] wait number of nodes instead of green
Original commit: elastic/x-pack-elasticsearch@d5e47d4e28
2017-02-16 13:54:30 +01:00
Tim Vernum de10d26bfc [Security] Treat LDAP exceptions as auth failures (elastic/x-pack-elasticsearch#565)
Any failure to communicate with an LDAP server should be treated as if authentication failed rather than a fatal "stop the world" error.

Because the AD realm validates the user's password by attempting to bind to the directory with that username/password combination, connection failures are to be expected and must be handled gracefully because the AD realm may not be the last realm in the order-chain, and any exception would prevent later realms from being tested.

Closes: elastic/x-pack-elasticsearch#564

Original commit: elastic/x-pack-elasticsearch@5282290b00
2017-02-16 22:57:12 +11:00
David Kyle e77a71827d Fix integ test after field rename
Original commit: elastic/x-pack-elasticsearch@4ebb93fb44
2017-02-16 11:19:15 +00:00
David Kyle 26df31a3d0 Rename Job.index_name -> Job.results_index_name (elastic/x-pack-elasticsearch#570)
Original commit: elastic/x-pack-elasticsearch@50fc6515e7
2017-02-16 10:26:38 +00:00
Martijn van Groningen d03c6befd5 [ML] Don't set node attributes when ml is disabled.
Original commit: elastic/x-pack-elasticsearch@d450750f41
2017-02-16 10:52:30 +01:00
Colin Goodheart-Smithe 19fc532961 Restricts certain ml endpoints if license forbids ml (elastic/x-pack-elasticsearch#568)
This change adds checks to the following machine learning actions to check that machine learning is permitted by the license before executing the request and throw and exception if the license forbids it:

* Create job
* Create data feed
* Open job
* Start data feed

This is not a final list of the restrictions we want to have in place if the license forbids ML access but it serves as a starting point and can be easily updated following further discussions.

The change also moves the `transportClientMode` check to `createComponents` so we don’t try to start the server parts of machine learning (job manager, connection to named pipes etc.) if we are running in a transport client.

Original commit: elastic/x-pack-elasticsearch@6c19ebd3bc
2017-02-16 09:40:53 +00:00
Martijn van Groningen 54d57f6398 [ml] When trying to find a node to open a job take `max_running_jobs` into account instead of just trying and then failing.
Also before even creating a persistent task for the job, check if there ml nodes available to run the job, otherwise fail quickly.

relates elastic/x-pack-elasticsearch#545

Original commit: elastic/x-pack-elasticsearch@63fd72e56a
2017-02-16 09:58:44 +01:00
Zach e441e0dc4e [ML] Fix order that DBQ is constructed
Also use Job.ID constant instead of string literal.

Kudos to @dimitris-athanasiou for this one :)

Original commit: elastic/x-pack-elasticsearch@906b080110
2017-02-15 15:53:45 -05:00
Martijn van Groningen 6d79210f79 [ml] Registered named writable entry for DatafeedState.
Also added a test to multi node qa module that tests the datafeeder, which should have caught this.

Original commit: elastic/x-pack-elasticsearch@89e4875f6c
2017-02-15 21:38:19 +01:00
jaymode c8b5be186d remove signing of authentication information
Now that TLS is required for node to node communication, we no longer need to sign the
authentication information to prevent tampering.

Original commit: elastic/x-pack-elasticsearch@1f86cf2395
2017-02-15 12:55:24 -05:00
Zachary Tong 65cd119733 [ML] Fix injection of DomainSplit, and only inject for Painless
Was accidentally injecting the script object, not the string version of the code.  Also
added a check so we only inject for Painless scripts (and not groovy, etc).

Minor style tweaks too.

Original commit: elastic/x-pack-elasticsearch@58c7275bd8
2017-02-15 09:31:16 -05:00
Martijn van Groningen 76ea6444cf [test] if enabled always specify setting other randomly specify setting to false.
Original commit: elastic/x-pack-elasticsearch@eacdb979e2
2017-02-14 22:28:43 +01:00
Zachary Tong cb29cbd8a9 [ML] Allow result indices to be shared. (elastic/x-pack-elasticsearch#555)
Essentially an update to https://github.com/elastic/prelert-legacy/pull/736 .  Still does not
default to using shared indices, but adds the capability for two jobs to share the same one
without conflict

Still does not default to using shared indices, just adds the capability for two jobs to share the same one without conflict.

Original commit: elastic/x-pack-elasticsearch@60d93a06ea
2017-02-14 14:59:28 -05:00
Zachary Tong 72ef1acdc0 Move domainSplit injection to ScrollDataExtractor (elastic/x-pack-elasticsearch#554)
By injecting immediately before use, we don't have to store it in the config and return
huge blobs to the user when they request the config

Original commit: elastic/x-pack-elasticsearch@2e53c930ea
2017-02-14 12:26:21 -05:00
Adrien Grand ac8f3ffec4 Fix security source filtering. (elastic/x-pack-elasticsearch#561)
Source filtering is currently written as if the fact that an object is accepted
means that all sub fields and objects are implicitly accepted, which is not
true.

Original commit: elastic/x-pack-elasticsearch@afc4440e7a
2017-02-14 17:44:20 +01:00
Martijn van Groningen 6e411c6118 [ml] Added allocation enabled node attribute setting
This node attribute setting defaults to true when ml is enabled and jobs are only opened on nodes where the allocation enabled node attribute is set to true.

Original commit: elastic/x-pack-elasticsearch@fd3f1b3058
2017-02-14 16:58:08 +01:00
Colin Goodheart-Smithe 6183015639 Adds licensing to machine learning feature (elastic/x-pack-elasticsearch#559)
This change adds licensing to the maching learning feature, and only allows access to machine learning if a trial or platinum license is installed.

Further, this change also renames `MlPlugin` to `MachineLearning` in line with the other feature plugin names and move the enabled setting to `XPackSettings`

Original commit: elastic/x-pack-elasticsearch@48ea9d781b
2017-02-14 15:47:37 +00:00
Martijn van Groningen 67fe584e0f [ml] Also allow closing failed jobs
Relates to elastic/x-pack-elasticsearch#545

Original commit: elastic/x-pack-elasticsearch@1c814afbf0
2017-02-14 15:38:46 +01:00
Dimitris Athanasiou 4976feb53b [ML] Extract MlInitializationService steps into respective methods (elastic/x-pack-elasticsearch#560)
Original commit: elastic/x-pack-elasticsearch@c43d02e752
2017-02-14 14:05:39 +00:00
David Kyle 27a90f4f0d Remove the job ignore_downtime option (elastic/x-pack-elasticsearch#550)
Original commit: elastic/x-pack-elasticsearch@5a7c4fdfea
2017-02-14 09:43:25 +00:00
Jason Tedor c4be485514 Add support for dispatching bad requests
This commit is a response to an upstream change in Elasticsearch which
added a bad dispatch request handler.

Relates elastic/x-pack-elasticsearch#553

Original commit: elastic/x-pack-elasticsearch@c28baee799
2017-02-13 17:39:44 -05:00
jaymode 82a80dbf42 Update seriailization after backport of content type changes
Original commit: elastic/x-pack-elasticsearch@0ef4dda41d
2017-02-13 13:18:56 -05:00
Martijn van Groningen 5eb8414215 [ml] Specify the minimum supported version,
so that we don't send ml metadata to older nodes.

Original commit: elastic/x-pack-elasticsearch@f196b5c461
2017-02-13 18:15:48 +01:00
Tanguy Leroux a6e67fe51e Monitoring: remove collectors setting elastic/elasticsearch#4957
This commit removes the undocumented, test used only, setting xpack.monitoring.collection.collectors. It should not be used by users at all, and has been initially created for collector tests.

From  elastic/elasticsearch#4957

Original commit: elastic/x-pack-elasticsearch@0fa39a94b5
2017-02-13 16:36:59 +01:00
Martijn van Groningen fbcac2710a [ml] changed start datafeed validation
Original commit: elastic/x-pack-elasticsearch@266d360cae
2017-02-13 14:04:32 +01:00
Dimitris Athanasiou 3e43b591df [ML] Correctly return job stats for _all (elastic/x-pack-elasticsearch#540)
Original commit: elastic/x-pack-elasticsearch@a286ea36e9
2017-02-13 09:10:35 +00:00
Jason Tedor 1b4fec642c Enforce Java version
The Elastic Secrets vault is served via HTTPS with a Let's Encrypt
certificate. The root certificate that cross-signed the Let's Encrypt
certificates were not trusted by the JDK until 8u101. This commit adds a
version check at the start of the build to make it clear the cause of
the underlying issue, and what the fix is.

Relates elastic/x-pack-elasticsearch#541

Original commit: elastic/x-pack-elasticsearch@6bf8076cb6
2017-02-11 12:12:26 -05:00
Ryan Ernst 2571921605 Rename x-pack project names to new names with split repo
Original commit: elastic/x-pack-elasticsearch@5a908f5dcc
2017-02-10 11:02:42 -08:00
Ryan Ernst 1fb742a0ad Rename core plugin and transport dirs
Original commit: elastic/x-pack-elasticsearch@1844685f77
2017-02-10 11:02:42 -08:00