Commit Graph

5304 Commits

Author SHA1 Message Date
David Kyle 00bc35cf9f [ML] Consistently throw missing resource exception on all endpoints if the job id is not known. (elastic/x-pack-elasticsearch#1024)
Original commit: elastic/x-pack-elasticsearch@6676a03599
2017-04-10 15:23:08 +01:00
Tanguy Leroux 310b85083d [Test] Change how LocalExporterTests stops
This commit changes how LocalExporterTests stops the monitoring
components: it first stops the monitoring service (but keeps the
local exporter enabled), deletes and checks if monitoring indices
are recreated, and then disables the local exporter.

Original commit: elastic/x-pack-elasticsearch@4c4809a660
2017-04-10 15:14:49 +02:00
Martijn van Groningen 454d6b3390 [ML] Reintroduced the closing job state
Closing a job may take a while. In the meantime it is possible to start a datafeed, because before this change the job state remained OPENED.
With this change when the executor node receives the close job request, it will first set the status to CLOSING and after that closes the job (closing autodetect process, etc.).

relates elastic/x-pack-elasticsearch#990

Original commit: elastic/x-pack-elasticsearch@d8d89c0756
2017-04-10 15:08:50 +02:00
Tanguy Leroux 6396edc6a7 [Test] Add Monitoring Bulk API REST tests with privileges (elastic/x-pack-elasticsearch#968)
This commit removes the smoke-test-monitoring-with-security project
and replaces it with a REST test.

Original commit: elastic/x-pack-elasticsearch@f1665815c2
2017-04-10 15:08:19 +02:00
javanna 07a99c4792 [TEST] exclude num_reduce_phases from the SearchTransformTests comparisons
Original commit: elastic/x-pack-elasticsearch@d3a5c7e86b
2017-04-10 14:21:38 +02:00
David Roberts e8337344a6 [ML] Include C++ 3rd party notices in X-Pack combined NOTICES file (elastic/x-pack-elasticsearch#953)
relates elastic/x-pack-elasticsearch#620

Original commit: elastic/x-pack-elasticsearch@5d73bd340c
2017-04-10 12:40:07 +01:00
Martijn van Groningen 63c04ef19d [ML] Change close job api to no longer extend from TransportJobTaskAction.
The execution has diverged too much from post data, flush and update process apis, since the close all jobs have been added.
The logic is now easier to understand as it exist in a single source file instead of in both CloseJobAction and TransportJobTaskAction.

Original commit: elastic/x-pack-elasticsearch@daf5fabad5
2017-04-10 13:33:12 +02:00
Luca Cavanna cb07e93fde Adapt to getProperty move out of MultiBucketsAggregation.Bucket (elastic/x-pack-elasticsearch#1015)
Adapt to getProperty move out of MultiBucketsAggregation.Bucket

Original commit: elastic/x-pack-elasticsearch@3e8ca753cd
2017-04-10 13:28:37 +02:00
Tim Vernum 18835fe4df Expand debug logging for authentication / realms (elastic/x-pack-elasticsearch#641)
Users currently have difficulty diagnosing authentication failures.
Some logging messages mislead them, and in other cases there are unexpected behaviours that are not logged at all.
These additional DEBUG log messages and change some existing messages in an attempt to alleviate that problem.

Original commit: elastic/x-pack-elasticsearch@c6ea98b038
2017-04-10 21:21:40 +10:00
Luca Cavanna ef4de13ca4 Adapt ChunkedDataExtractor to removal of Aggregations#getProperty (elastic/x-pack-elasticsearch#1008)
Adapt ChunkedDataExtractor to removal of Aggregations#getProperty

Original commit: elastic/x-pack-elasticsearch@01ee2fee58
2017-04-10 12:32:11 +02:00
Martijn van Groningen 0ae0f93994 [ML] Stop immediately when job isn't opened
Original commit: elastic/x-pack-elasticsearch@cdb4cf47f5
2017-04-10 10:54:40 +02:00
Martijn van Groningen 0c2ef5608e [ML] Use same CS for validating job
Original commit: elastic/x-pack-elasticsearch@00b593e8c2
2017-04-10 10:32:42 +02:00
Dimitris Athanasiou a1cb22836c [ML] Add more categorization validations (elastic/x-pack-elasticsearch#1019)
- validates that when mlcategory is used, categorization_field_name is
set
- validates that when categorization_field_name is set, mlcategory is
used

relates elastic/x-pack-elasticsearch#986

Original commit: elastic/x-pack-elasticsearch@e861a3ed58
2017-04-09 18:18:17 +01:00
Dimitrios Athanasiou 0df726f6dd [ML] Fix logger usage
Original commit: elastic/x-pack-elasticsearch@f85148e0ff
2017-04-08 15:52:44 +01:00
Dimitrios Athanasiou 16d8dc6815 [ML] Log when no node found to open job/start datafeed
Relates elastic/x-pack-elasticsearch#878

Original commit: elastic/x-pack-elasticsearch@a64b7eaa80
2017-04-08 15:18:48 +01:00
Dimitris Athanasiou 5c9364d5fe [ML] Increase stop datafeed default timeout to 5 minutes (elastic/x-pack-elasticsearch#992)
Increase the timeout to give enough time for a datafeed to
stop smoothly.

This is the second step to avoid hitting the default timeout.
The first was ensuring aggregated datafeed is cancellable in
a responsive manner. The third and final step will be to
apply chunking in aggregated datafeeds in order to shorten
the duration of the search, which will make cancellation even
more responsive.

Relates elastic/x-pack-elasticsearch#803

Original commit: elastic/x-pack-elasticsearch@db642330ec
2017-04-08 14:44:08 +01:00
Dimitrios Athanasiou 734243a4df [TEST] Remove unused logger from CategorizationIT
Original commit: elastic/x-pack-elasticsearch@7b812eeede
2017-04-08 14:42:47 +01:00
Jay Modi cdfcfc5540 Do not contact AWS in offline mode (elastic/x-pack-elasticsearch#1001)
This commit restores the ability to build x-pack-elasticsearch without issues when running without
access to the internet. When the `--offline` flag is used, we will not try to contact vault and the
aws apis to retrieve the ml-cpp binaries but instead gradle will use a cached version even though
it may be expired.

relates elastic/x-pack-elasticsearch#726

Original commit: elastic/x-pack-elasticsearch@b0915d8fa9
2017-04-08 07:08:28 -06:00
Dimitrios Athanasiou 10df00beb1 [ML] Fix typo in assignment_explanation field name
Closes  elastic/x-pack-elasticsearch#1017

Original commit: elastic/x-pack-elasticsearch@b18e2a38fc
2017-04-08 13:17:14 +01:00
Ryan Ernst e8399cfccf Build: Build 5.x version of xpack for bwc tests to use (elastic/x-pack-elasticsearch#885)
This is analagous of the bwc-zip for elasticsearch. The one caveat is
due to the structure of how ES+xpack must be checked out, we end up with
a third clone of elasticsearch (the second being in :distribution:bwc-zip).
But the rolling upgrade integ test passes with this change.

relates elastic/x-pack-elasticsearch#870

Original commit: elastic/x-pack-elasticsearch@34bdce6e99
2017-04-07 22:19:30 -07:00
Jay Modi 984055392e Rename ml delete by query to xpack delete by query (elastic/x-pack-elasticsearch#1005)
This commit renames and moves the forked delete by query classes from being ml specific to being a
xpack common class since an upcoming security feature plans to make use of this. Additionally, this
commit fixes a issue where the dbq action was being executed by the calling user instead of the
xpack user for certain requests. This was found when adding a authorization change that restricts
this action's execution to the xpack user only.

Original commit: elastic/x-pack-elasticsearch@d5967e7255
2017-04-07 19:06:01 -06:00
debadair 40fc8058e5 [DOCS] Fixed includes to correctly reference other repos.
Original commit: elastic/x-pack-elasticsearch@f6a0d09575
2017-04-07 17:32:03 -07:00
Tanguy Leroux 9246261bc5 [Test] Generate less fields in LocalExporterTests
The test creates random documents, it looks like it generates too
   many fields.

Original commit: elastic/x-pack-elasticsearch@f55a3d1fc7
2017-04-07 20:47:40 +02:00
Dimitrios Athanasiou 54b72dc938 [TEST] Add ML categorization integ test
Original commit: elastic/x-pack-elasticsearch@ba8c2a1d64
2017-04-07 18:46:01 +01:00
David Roberts b9941de39d [TEST] Fix some assertions to match prior code change
Original commit: elastic/x-pack-elasticsearch@be4cc9cbef
2017-04-07 18:01:32 +01:00
Zachary Tong 8e2e26fc44 [ML] Forbid 'by' and 'over' as fieldnames for by/over (elastic/x-pack-elasticsearch#1007)
relates elastic/x-pack-elasticsearch#1002

Original commit: elastic/x-pack-elasticsearch@ba9005a58e
2017-04-07 16:09:22 +00:00
Dimitris Athanasiou bf111dde7e [ML] Add notification when no suitable nodes exist to allocate job (elastic/x-pack-elasticsearch#1006)
Relates elastic/x-pack-elasticsearch#878

Original commit: elastic/x-pack-elasticsearch@5a96484662
2017-04-07 16:55:10 +01:00
David Roberts 3986235d93 [ML] Fix large state persistence performance (elastic/x-pack-elasticsearch#1004)
There was a problem with the way CompositeBytesReference was used in the
StateProcessor.  In the case of a large state document we ended up with a
deeply nested CompositeBytesReference that then caused a deep stack and N^2
processing in the bulk action processor.

This change uses an intermediate list of byte arrays that get combined into
a single CompositeBytesReference to avoid the deep nesting.

Additionally, errors in state processing now bubble up to close the state
stream, which will cause the C++ process to stop trying to persist more state.

Finally, the results processor also times out after a similar period (30 minutes)
to that used by the state processor.

Original commit: elastic/x-pack-elasticsearch@ceb31481d1
2017-04-07 15:57:21 +01:00
Brandon Kobel fbefaf5b6d Adding xpack.reporting.index to the reporting-settings (elastic/x-pack-elasticsearch#1000)
* Adding xpack.reporting.index to the reporting-settings

* Fixing text size

Original commit: elastic/x-pack-elasticsearch@334274e49d
2017-04-07 10:01:02 -04:00
Hendrik Muhs c7fd1aacff [ML] implement _all for closing jobs (elastic/x-pack-elasticsearch#962)
Add a '_all' functionality for closing ML jobs.

For cluster shutdown due to maintenance and major upgrades we recommend the user to stop all datafeeds and jobs. This change add the ability to close all jobs at once where previously it was required to iterate over all jobs and do a explicit close. This is part one of elastic/x-pack-elasticsearch#795, part two can be found in elastic/x-pack-elasticsearch#995.

relates elastic/x-pack-elasticsearch#795

Original commit: elastic/x-pack-elasticsearch@9b251ed7e1
2017-04-07 14:51:13 +02:00
Dimitrios Athanasiou b41288592c [ML] Some logging improvements
- Downgrade flush logging as it's called too often from datafeeds
- Add log info for job opening

Original commit: elastic/x-pack-elasticsearch@a262069f78
2017-04-07 13:36:04 +01:00
Adrien Grand df3cc61c5b Mute ActivateWatchTests.testDeactivateAndActivate.
Relates elastic/x-pack-elasticsearch#395

Original commit: elastic/x-pack-elasticsearch@bdf5a0dfed
2017-04-07 11:03:31 +02:00
Martijn van Groningen d8170861ea [TEST] Check whether datafeed is stopped differently
Original commit: elastic/x-pack-elasticsearch@9bde2cc2c2
2017-04-07 10:50:48 +02:00
Martijn van Groningen b248b6dbfc [TEST] [ML] Run ensureStableCluster(...) on all nodes and run synched flush before executing the disrupt
Original commit: elastic/x-pack-elasticsearch@d10beacb73
2017-04-07 09:51:13 +02:00
debadair ee5ab5b1d2 [DOCS] Migrated security topics from x-pack repo to x-pack-elasticsearch.
Original commit: elastic/x-pack-elasticsearch@e54aa1fd0a
2017-04-06 18:35:32 -07:00
debadair e804d78be5 [DOCS] Migrated rest-api topics from x-pack repo to x-pack-elasticsearch.
Original commit: elastic/x-pack-elasticsearch@46c9bf780a
2017-04-06 18:04:39 -07:00
debadair 5c588bcd8d [DOCS] Merged an added notification setting in x-pack master to x-pack-elasticsearch.
Original commit: elastic/x-pack-elasticsearch@8a30126721
2017-04-06 17:57:33 -07:00
debadair 9f505d16d4 [DOCS] Migrated settings topics from x-pack repo to x-pack-elasticsearch.
Original commit: elastic/x-pack-elasticsearch@e56dcf6066
2017-04-06 17:39:12 -07:00
Martijn van Groningen 3986a2a06c [ML] Remove PersistentTask#isCurrentStatus() usages
Original commit: elastic/x-pack-elasticsearch@efe7e1e770
2017-04-06 21:32:54 +02:00
Zachary Tong e2a30331ba Refactor StartDataFeed to more intelligently wait on shard status (elastic/x-pack-elasticsearch#974)
Rather than using an async call, this leverages
the Assignment logic while selecting nodes.

Now with 300% more tests!

Original commit: elastic/x-pack-elasticsearch@300d628f72
2017-04-06 18:14:51 +00:00
David Kyle 9a9ae5edc7 [ML] Restore categoriser state after the anomaly detector (elastic/x-pack-elasticsearch#993)
Original commit: elastic/x-pack-elasticsearch@fc4205f1d6
2017-04-06 18:12:52 +01:00
Lisa Cawley e339cf82df [DOCS] Added examples for all ML job APIs (elastic/x-pack-elasticsearch#980)
Original commit: elastic/x-pack-elasticsearch@7911b53af9
2017-04-06 08:07:49 -07:00
Martijn van Groningen 5585283216 [ML] Added cause to exception.
Relates to elastic/x-pack-elasticsearch#990

Original commit: elastic/x-pack-elasticsearch@a66bc958b7
2017-04-06 16:34:24 +02:00
David Roberts 97866eb0e8 [ML] Increase timeouts for named pipe connections to native processes (elastic/x-pack-elasticsearch#987)
It has been observed that Amazon EBS volumes created from snapshots can
have very high latency the first time a given block is accessed.  This
can lead to named pipes taking longer than 2 seconds to create.
Since the native processes create their named pipes immediately after
startup, and this only takes a fraction of a second on a local disk, 2
seconds was considered a generous timeout, but it seems that in the case
of a remote NAS with lazy provisioning it's not long enough.  During
debugging a latency of just over 3 seconds was observed.  The timeouts
have been increased to 10 seconds.

relates elastic/x-pack-elasticsearch#922

Original commit: elastic/x-pack-elasticsearch@c90434c948
2017-04-06 14:42:27 +01:00
Jay Modi 92942c9327 Add support for IndexOrDocValuesQuery in FieldExtractor (elastic/x-pack-elasticsearch#976)
This commit adds support for extracting the fields from a IndexOrDocValuesQuery so that we can
still take advantage of the cache when this query is used.

relates elastic/x-pack-elasticsearch#973

Original commit: elastic/x-pack-elasticsearch@0b97202a41
2017-04-06 12:44:25 +01:00
Tim Vernum 573b421446 Support multiple indices in SecurityLifecycleService (elastic/x-pack-elasticsearch#961)
Moves the direct management of the security index from SecurityLifecycleService to IndexLifecycleManager, so that the SecurityLifecycleService can take responsibility for several indices.

Multiple security indices are required as we move away from storing multiple types in a single index.

Original commit: elastic/x-pack-elasticsearch@fde3a42b4d
2017-04-06 21:37:33 +10:00
jaymode 7efc9e1270 Test: Fix IndexAuditTrailMutedTests to set the threadpool
The IndexAuditTrailMutedTests have a threadpool but fail to set it on the test client, which causes
a NPE and tests to fail.

Original commit: elastic/x-pack-elasticsearch@d34a4ce080
2017-04-05 13:50:00 -04:00
Alexander Reelsen 16f5f44d41 Watcher: Dont check index auto creation if disabled/transportclient (elastic/x-pack-elasticsearch#975)
When watcher is disabled, no auto index creation checks should happen.

Relates elastic/elasticsearch#23919

Original commit: elastic/x-pack-elasticsearch@d924838b4d
2017-04-05 18:39:54 +01:00
jaymode a53026498f Test: mute SearchGroupsResolverInMemoryTests
This commit mutes SearchGroupsResolverInMemoryTests as they fail with a thread leak. See elastic/x-pack-elasticsearch#971

Original commit: elastic/x-pack-elasticsearch@be7c988e20
2017-04-05 13:23:53 -04:00
Dimitrios Athanasiou e97035af87 Add missing logging param
Original commit: elastic/x-pack-elasticsearch@30ee528798
2017-04-05 17:00:12 +01:00