Commit Graph

5490 Commits

Author SHA1 Message Date
Dimitrios Athanasiou 16d8dc6815 [ML] Log when no node found to open job/start datafeed
Relates elastic/x-pack-elasticsearch#878

Original commit: elastic/x-pack-elasticsearch@a64b7eaa80
2017-04-08 15:18:48 +01:00
Dimitris Athanasiou 5c9364d5fe [ML] Increase stop datafeed default timeout to 5 minutes (elastic/x-pack-elasticsearch#992)
Increase the timeout to give enough time for a datafeed to
stop smoothly.

This is the second step to avoid hitting the default timeout.
The first was ensuring aggregated datafeed is cancellable in
a responsive manner. The third and final step will be to
apply chunking in aggregated datafeeds in order to shorten
the duration of the search, which will make cancellation even
more responsive.

Relates elastic/x-pack-elasticsearch#803

Original commit: elastic/x-pack-elasticsearch@db642330ec
2017-04-08 14:44:08 +01:00
Dimitrios Athanasiou 734243a4df [TEST] Remove unused logger from CategorizationIT
Original commit: elastic/x-pack-elasticsearch@7b812eeede
2017-04-08 14:42:47 +01:00
Jay Modi cdfcfc5540 Do not contact AWS in offline mode (elastic/x-pack-elasticsearch#1001)
This commit restores the ability to build x-pack-elasticsearch without issues when running without
access to the internet. When the `--offline` flag is used, we will not try to contact vault and the
aws apis to retrieve the ml-cpp binaries but instead gradle will use a cached version even though
it may be expired.

relates elastic/x-pack-elasticsearch#726

Original commit: elastic/x-pack-elasticsearch@b0915d8fa9
2017-04-08 07:08:28 -06:00
Dimitrios Athanasiou 10df00beb1 [ML] Fix typo in assignment_explanation field name
Closes  elastic/x-pack-elasticsearch#1017

Original commit: elastic/x-pack-elasticsearch@b18e2a38fc
2017-04-08 13:17:14 +01:00
Ryan Ernst e8399cfccf Build: Build 5.x version of xpack for bwc tests to use (elastic/x-pack-elasticsearch#885)
This is analagous of the bwc-zip for elasticsearch. The one caveat is
due to the structure of how ES+xpack must be checked out, we end up with
a third clone of elasticsearch (the second being in :distribution:bwc-zip).
But the rolling upgrade integ test passes with this change.

relates elastic/x-pack-elasticsearch#870

Original commit: elastic/x-pack-elasticsearch@34bdce6e99
2017-04-07 22:19:30 -07:00
Jay Modi 984055392e Rename ml delete by query to xpack delete by query (elastic/x-pack-elasticsearch#1005)
This commit renames and moves the forked delete by query classes from being ml specific to being a
xpack common class since an upcoming security feature plans to make use of this. Additionally, this
commit fixes a issue where the dbq action was being executed by the calling user instead of the
xpack user for certain requests. This was found when adding a authorization change that restricts
this action's execution to the xpack user only.

Original commit: elastic/x-pack-elasticsearch@d5967e7255
2017-04-07 19:06:01 -06:00
debadair 40fc8058e5 [DOCS] Fixed includes to correctly reference other repos.
Original commit: elastic/x-pack-elasticsearch@f6a0d09575
2017-04-07 17:32:03 -07:00
Tanguy Leroux 9246261bc5 [Test] Generate less fields in LocalExporterTests
The test creates random documents, it looks like it generates too
   many fields.

Original commit: elastic/x-pack-elasticsearch@f55a3d1fc7
2017-04-07 20:47:40 +02:00
Dimitrios Athanasiou 54b72dc938 [TEST] Add ML categorization integ test
Original commit: elastic/x-pack-elasticsearch@ba8c2a1d64
2017-04-07 18:46:01 +01:00
David Roberts b9941de39d [TEST] Fix some assertions to match prior code change
Original commit: elastic/x-pack-elasticsearch@be4cc9cbef
2017-04-07 18:01:32 +01:00
Zachary Tong 8e2e26fc44 [ML] Forbid 'by' and 'over' as fieldnames for by/over (elastic/x-pack-elasticsearch#1007)
relates elastic/x-pack-elasticsearch#1002

Original commit: elastic/x-pack-elasticsearch@ba9005a58e
2017-04-07 16:09:22 +00:00
Dimitris Athanasiou bf111dde7e [ML] Add notification when no suitable nodes exist to allocate job (elastic/x-pack-elasticsearch#1006)
Relates elastic/x-pack-elasticsearch#878

Original commit: elastic/x-pack-elasticsearch@5a96484662
2017-04-07 16:55:10 +01:00
David Roberts 3986235d93 [ML] Fix large state persistence performance (elastic/x-pack-elasticsearch#1004)
There was a problem with the way CompositeBytesReference was used in the
StateProcessor.  In the case of a large state document we ended up with a
deeply nested CompositeBytesReference that then caused a deep stack and N^2
processing in the bulk action processor.

This change uses an intermediate list of byte arrays that get combined into
a single CompositeBytesReference to avoid the deep nesting.

Additionally, errors in state processing now bubble up to close the state
stream, which will cause the C++ process to stop trying to persist more state.

Finally, the results processor also times out after a similar period (30 minutes)
to that used by the state processor.

Original commit: elastic/x-pack-elasticsearch@ceb31481d1
2017-04-07 15:57:21 +01:00
Brandon Kobel fbefaf5b6d Adding xpack.reporting.index to the reporting-settings (elastic/x-pack-elasticsearch#1000)
* Adding xpack.reporting.index to the reporting-settings

* Fixing text size

Original commit: elastic/x-pack-elasticsearch@334274e49d
2017-04-07 10:01:02 -04:00
Hendrik Muhs c7fd1aacff [ML] implement _all for closing jobs (elastic/x-pack-elasticsearch#962)
Add a '_all' functionality for closing ML jobs.

For cluster shutdown due to maintenance and major upgrades we recommend the user to stop all datafeeds and jobs. This change add the ability to close all jobs at once where previously it was required to iterate over all jobs and do a explicit close. This is part one of elastic/x-pack-elasticsearch#795, part two can be found in elastic/x-pack-elasticsearch#995.

relates elastic/x-pack-elasticsearch#795

Original commit: elastic/x-pack-elasticsearch@9b251ed7e1
2017-04-07 14:51:13 +02:00
Dimitrios Athanasiou b41288592c [ML] Some logging improvements
- Downgrade flush logging as it's called too often from datafeeds
- Add log info for job opening

Original commit: elastic/x-pack-elasticsearch@a262069f78
2017-04-07 13:36:04 +01:00
Adrien Grand df3cc61c5b Mute ActivateWatchTests.testDeactivateAndActivate.
Relates elastic/x-pack-elasticsearch#395

Original commit: elastic/x-pack-elasticsearch@bdf5a0dfed
2017-04-07 11:03:31 +02:00
Martijn van Groningen d8170861ea [TEST] Check whether datafeed is stopped differently
Original commit: elastic/x-pack-elasticsearch@9bde2cc2c2
2017-04-07 10:50:48 +02:00
Martijn van Groningen b248b6dbfc [TEST] [ML] Run ensureStableCluster(...) on all nodes and run synched flush before executing the disrupt
Original commit: elastic/x-pack-elasticsearch@d10beacb73
2017-04-07 09:51:13 +02:00
debadair ee5ab5b1d2 [DOCS] Migrated security topics from x-pack repo to x-pack-elasticsearch.
Original commit: elastic/x-pack-elasticsearch@e54aa1fd0a
2017-04-06 18:35:32 -07:00
debadair e804d78be5 [DOCS] Migrated rest-api topics from x-pack repo to x-pack-elasticsearch.
Original commit: elastic/x-pack-elasticsearch@46c9bf780a
2017-04-06 18:04:39 -07:00
debadair 5c588bcd8d [DOCS] Merged an added notification setting in x-pack master to x-pack-elasticsearch.
Original commit: elastic/x-pack-elasticsearch@8a30126721
2017-04-06 17:57:33 -07:00
debadair 9f505d16d4 [DOCS] Migrated settings topics from x-pack repo to x-pack-elasticsearch.
Original commit: elastic/x-pack-elasticsearch@e56dcf6066
2017-04-06 17:39:12 -07:00
Martijn van Groningen 3986a2a06c [ML] Remove PersistentTask#isCurrentStatus() usages
Original commit: elastic/x-pack-elasticsearch@efe7e1e770
2017-04-06 21:32:54 +02:00
Zachary Tong e2a30331ba Refactor StartDataFeed to more intelligently wait on shard status (elastic/x-pack-elasticsearch#974)
Rather than using an async call, this leverages
the Assignment logic while selecting nodes.

Now with 300% more tests!

Original commit: elastic/x-pack-elasticsearch@300d628f72
2017-04-06 18:14:51 +00:00
David Kyle 9a9ae5edc7 [ML] Restore categoriser state after the anomaly detector (elastic/x-pack-elasticsearch#993)
Original commit: elastic/x-pack-elasticsearch@fc4205f1d6
2017-04-06 18:12:52 +01:00
Lisa Cawley e339cf82df [DOCS] Added examples for all ML job APIs (elastic/x-pack-elasticsearch#980)
Original commit: elastic/x-pack-elasticsearch@7911b53af9
2017-04-06 08:07:49 -07:00
Martijn van Groningen 5585283216 [ML] Added cause to exception.
Relates to elastic/x-pack-elasticsearch#990

Original commit: elastic/x-pack-elasticsearch@a66bc958b7
2017-04-06 16:34:24 +02:00
David Roberts 97866eb0e8 [ML] Increase timeouts for named pipe connections to native processes (elastic/x-pack-elasticsearch#987)
It has been observed that Amazon EBS volumes created from snapshots can
have very high latency the first time a given block is accessed.  This
can lead to named pipes taking longer than 2 seconds to create.
Since the native processes create their named pipes immediately after
startup, and this only takes a fraction of a second on a local disk, 2
seconds was considered a generous timeout, but it seems that in the case
of a remote NAS with lazy provisioning it's not long enough.  During
debugging a latency of just over 3 seconds was observed.  The timeouts
have been increased to 10 seconds.

relates elastic/x-pack-elasticsearch#922

Original commit: elastic/x-pack-elasticsearch@c90434c948
2017-04-06 14:42:27 +01:00
Jay Modi 92942c9327 Add support for IndexOrDocValuesQuery in FieldExtractor (elastic/x-pack-elasticsearch#976)
This commit adds support for extracting the fields from a IndexOrDocValuesQuery so that we can
still take advantage of the cache when this query is used.

relates elastic/x-pack-elasticsearch#973

Original commit: elastic/x-pack-elasticsearch@0b97202a41
2017-04-06 12:44:25 +01:00
Tim Vernum 573b421446 Support multiple indices in SecurityLifecycleService (elastic/x-pack-elasticsearch#961)
Moves the direct management of the security index from SecurityLifecycleService to IndexLifecycleManager, so that the SecurityLifecycleService can take responsibility for several indices.

Multiple security indices are required as we move away from storing multiple types in a single index.

Original commit: elastic/x-pack-elasticsearch@fde3a42b4d
2017-04-06 21:37:33 +10:00
jaymode 7efc9e1270 Test: Fix IndexAuditTrailMutedTests to set the threadpool
The IndexAuditTrailMutedTests have a threadpool but fail to set it on the test client, which causes
a NPE and tests to fail.

Original commit: elastic/x-pack-elasticsearch@d34a4ce080
2017-04-05 13:50:00 -04:00
Alexander Reelsen 16f5f44d41 Watcher: Dont check index auto creation if disabled/transportclient (elastic/x-pack-elasticsearch#975)
When watcher is disabled, no auto index creation checks should happen.

Relates elastic/elasticsearch#23919

Original commit: elastic/x-pack-elasticsearch@d924838b4d
2017-04-05 18:39:54 +01:00
jaymode a53026498f Test: mute SearchGroupsResolverInMemoryTests
This commit mutes SearchGroupsResolverInMemoryTests as they fail with a thread leak. See elastic/x-pack-elasticsearch#971

Original commit: elastic/x-pack-elasticsearch@be7c988e20
2017-04-05 13:23:53 -04:00
Dimitrios Athanasiou e97035af87 Add missing logging param
Original commit: elastic/x-pack-elasticsearch@30ee528798
2017-04-05 17:00:12 +01:00
Dimitris Athanasiou 0be4082ad7 [ML] Add notification for loading snapshot (elastic/x-pack-elasticsearch#970)
As the snapshot that is loaded is an important operational
aspect of a job, this change adds a notification that displays
the loaded snapshot with its latest_record_timestamp and the
job's latest_record_timestamp. Having both allows us to discover
when a job is recovering after a node failure.

relates elastic/x-pack-elasticsearch#872

Original commit: elastic/x-pack-elasticsearch@c2dee495a2
2017-04-05 16:43:14 +01:00
Tanguy Leroux 7b48bac9f4 [Test] Gives more time to LocalExporterTests to delete indices
The test fails on slow machines because of inflight bulk requests
that hit one node while the others are stopping. This commit adds
more time (10s), equivalent to 2 to 3 collection interval, to delete
the monitoring indices. It also add TRACE logging level for the test.

Original commit: elastic/x-pack-elasticsearch@b433937946
2017-04-05 16:38:12 +02:00
Martijn van Groningen 2edbd0efeb [ML] Removed unused methods
Original commit: elastic/x-pack-elasticsearch@8b3c606147
2017-04-05 14:30:21 +02:00
Martijn van Groningen cbbc3cfe20 [ML] Slightly adjust JobProvider#getAutodetectParams(...)
Original commit: elastic/x-pack-elasticsearch@2aaf618d78
2017-04-05 14:18:00 +02:00
Colin Goodheart-Smithe 00a5759d54 [ML] Adds jobType to Job (elastic/x-pack-elasticsearch#947)
* [ML] Adds jobType to Job

This change adds `jobType` field to teh `Job` class so that when the job is written to the index a `job_type` field is written int he document. This will help separate this type of job from other new job types in the future so migrating the index to allow those new type of jobs will be easer

relates elastic/x-pack-elasticsearch#798

* Addresses review comments

Original commit: elastic/x-pack-elasticsearch@d9fd11edb3
2017-04-05 11:12:26 +01:00
Martijn van Groningen 2b5c0faebd [TEST] wait for cs version to be the same on all nodes before stopping datafeed
Original commit: elastic/x-pack-elasticsearch@6fd6224047
2017-04-05 11:37:36 +02:00
Alexander Reelsen 4b2680dac7 Tests: Ensure watch index is deleted to execute test correctly
The test requires no watch index or alias to exist, so making
sure this is deleted as part of the test.

relates elastic/x-pack-elasticsearch#920

Original commit: elastic/x-pack-elasticsearch@c1cc953ce3
2017-04-05 10:36:43 +01:00
Tim Vernum 7f0fd9e1a3 Failed LDAP SearchResults should be Exceptions (elastic/x-pack-elasticsearch#773)
When the LDAP SDK returns a SearchResult that has a non-success ResultCode, convert it to an exception and call onFailure

A configuration setting controls whether failures in referrals should be fatal (defaults to ignoring errors)

Closes: elastic/x-pack-elasticsearch#717

Original commit: elastic/x-pack-elasticsearch@4159758c2a
2017-04-05 18:40:17 +10:00
David Kyle 7c45cb7ccf Fix check style line length failure
Original commit: elastic/x-pack-elasticsearch@d37595db2b
2017-04-05 09:00:16 +01:00
David Kyle f8bb762a59 [TEST] Revert "Tests: Disable `ml/index_layout/Test unrelated index`."
This reverts commit elastic/x-pack-elasticsearch@097f559d0b.

Original commit: elastic/x-pack-elasticsearch@1f63456866
2017-04-05 08:48:23 +01:00
Martijn van Groningen 4a57c92e6f [TEST] unmuted test
Original commit: elastic/x-pack-elasticsearch@5dd80c32c7
2017-04-05 09:46:02 +02:00
David Kyle 7b45460951 [ML] Wait for dataCounts to be persisted (elastic/x-pack-elasticsearch#952)
Original commit: elastic/x-pack-elasticsearch@48ca4d7363
2017-04-05 08:45:01 +01:00
Lisa Cawley 843a0d8b3f [DOCS] Add ML documentation to master (elastic/x-pack-elasticsearch#959)
Original commit: elastic/x-pack-elasticsearch@666a10bd23
2017-04-04 15:26:39 -07:00
Igor Motov d11fbfa70c Persistent Tasks: refactor PersistentTasksService to use ActionListener (elastic/x-pack-elasticsearch#937)
PersistentTasksService methods are not using ActionListener<PersistentTask<?>> instead of PersistentTaskOperationListener.

Original commit: elastic/x-pack-elasticsearch@f95d8bda3d
2017-04-04 13:56:22 -04:00