Commit Graph

5276 Commits

Author SHA1 Message Date
Brandon Kobel fbefaf5b6d Adding xpack.reporting.index to the reporting-settings (elastic/x-pack-elasticsearch#1000)
* Adding xpack.reporting.index to the reporting-settings

* Fixing text size

Original commit: elastic/x-pack-elasticsearch@334274e49d
2017-04-07 10:01:02 -04:00
Hendrik Muhs c7fd1aacff [ML] implement _all for closing jobs (elastic/x-pack-elasticsearch#962)
Add a '_all' functionality for closing ML jobs.

For cluster shutdown due to maintenance and major upgrades we recommend the user to stop all datafeeds and jobs. This change add the ability to close all jobs at once where previously it was required to iterate over all jobs and do a explicit close. This is part one of elastic/x-pack-elasticsearch#795, part two can be found in elastic/x-pack-elasticsearch#995.

relates elastic/x-pack-elasticsearch#795

Original commit: elastic/x-pack-elasticsearch@9b251ed7e1
2017-04-07 14:51:13 +02:00
Dimitrios Athanasiou b41288592c [ML] Some logging improvements
- Downgrade flush logging as it's called too often from datafeeds
- Add log info for job opening

Original commit: elastic/x-pack-elasticsearch@a262069f78
2017-04-07 13:36:04 +01:00
Adrien Grand df3cc61c5b Mute ActivateWatchTests.testDeactivateAndActivate.
Relates elastic/x-pack-elasticsearch#395

Original commit: elastic/x-pack-elasticsearch@bdf5a0dfed
2017-04-07 11:03:31 +02:00
Martijn van Groningen d8170861ea [TEST] Check whether datafeed is stopped differently
Original commit: elastic/x-pack-elasticsearch@9bde2cc2c2
2017-04-07 10:50:48 +02:00
Martijn van Groningen b248b6dbfc [TEST] [ML] Run ensureStableCluster(...) on all nodes and run synched flush before executing the disrupt
Original commit: elastic/x-pack-elasticsearch@d10beacb73
2017-04-07 09:51:13 +02:00
debadair ee5ab5b1d2 [DOCS] Migrated security topics from x-pack repo to x-pack-elasticsearch.
Original commit: elastic/x-pack-elasticsearch@e54aa1fd0a
2017-04-06 18:35:32 -07:00
debadair e804d78be5 [DOCS] Migrated rest-api topics from x-pack repo to x-pack-elasticsearch.
Original commit: elastic/x-pack-elasticsearch@46c9bf780a
2017-04-06 18:04:39 -07:00
debadair 5c588bcd8d [DOCS] Merged an added notification setting in x-pack master to x-pack-elasticsearch.
Original commit: elastic/x-pack-elasticsearch@8a30126721
2017-04-06 17:57:33 -07:00
debadair 9f505d16d4 [DOCS] Migrated settings topics from x-pack repo to x-pack-elasticsearch.
Original commit: elastic/x-pack-elasticsearch@e56dcf6066
2017-04-06 17:39:12 -07:00
Martijn van Groningen 3986a2a06c [ML] Remove PersistentTask#isCurrentStatus() usages
Original commit: elastic/x-pack-elasticsearch@efe7e1e770
2017-04-06 21:32:54 +02:00
Zachary Tong e2a30331ba Refactor StartDataFeed to more intelligently wait on shard status (elastic/x-pack-elasticsearch#974)
Rather than using an async call, this leverages
the Assignment logic while selecting nodes.

Now with 300% more tests!

Original commit: elastic/x-pack-elasticsearch@300d628f72
2017-04-06 18:14:51 +00:00
David Kyle 9a9ae5edc7 [ML] Restore categoriser state after the anomaly detector (elastic/x-pack-elasticsearch#993)
Original commit: elastic/x-pack-elasticsearch@fc4205f1d6
2017-04-06 18:12:52 +01:00
Lisa Cawley e339cf82df [DOCS] Added examples for all ML job APIs (elastic/x-pack-elasticsearch#980)
Original commit: elastic/x-pack-elasticsearch@7911b53af9
2017-04-06 08:07:49 -07:00
Martijn van Groningen 5585283216 [ML] Added cause to exception.
Relates to elastic/x-pack-elasticsearch#990

Original commit: elastic/x-pack-elasticsearch@a66bc958b7
2017-04-06 16:34:24 +02:00
David Roberts 97866eb0e8 [ML] Increase timeouts for named pipe connections to native processes (elastic/x-pack-elasticsearch#987)
It has been observed that Amazon EBS volumes created from snapshots can
have very high latency the first time a given block is accessed.  This
can lead to named pipes taking longer than 2 seconds to create.
Since the native processes create their named pipes immediately after
startup, and this only takes a fraction of a second on a local disk, 2
seconds was considered a generous timeout, but it seems that in the case
of a remote NAS with lazy provisioning it's not long enough.  During
debugging a latency of just over 3 seconds was observed.  The timeouts
have been increased to 10 seconds.

relates elastic/x-pack-elasticsearch#922

Original commit: elastic/x-pack-elasticsearch@c90434c948
2017-04-06 14:42:27 +01:00
Jay Modi 92942c9327 Add support for IndexOrDocValuesQuery in FieldExtractor (elastic/x-pack-elasticsearch#976)
This commit adds support for extracting the fields from a IndexOrDocValuesQuery so that we can
still take advantage of the cache when this query is used.

relates elastic/x-pack-elasticsearch#973

Original commit: elastic/x-pack-elasticsearch@0b97202a41
2017-04-06 12:44:25 +01:00
Tim Vernum 573b421446 Support multiple indices in SecurityLifecycleService (elastic/x-pack-elasticsearch#961)
Moves the direct management of the security index from SecurityLifecycleService to IndexLifecycleManager, so that the SecurityLifecycleService can take responsibility for several indices.

Multiple security indices are required as we move away from storing multiple types in a single index.

Original commit: elastic/x-pack-elasticsearch@fde3a42b4d
2017-04-06 21:37:33 +10:00
jaymode 7efc9e1270 Test: Fix IndexAuditTrailMutedTests to set the threadpool
The IndexAuditTrailMutedTests have a threadpool but fail to set it on the test client, which causes
a NPE and tests to fail.

Original commit: elastic/x-pack-elasticsearch@d34a4ce080
2017-04-05 13:50:00 -04:00
Alexander Reelsen 16f5f44d41 Watcher: Dont check index auto creation if disabled/transportclient (elastic/x-pack-elasticsearch#975)
When watcher is disabled, no auto index creation checks should happen.

Relates elastic/elasticsearch#23919

Original commit: elastic/x-pack-elasticsearch@d924838b4d
2017-04-05 18:39:54 +01:00
jaymode a53026498f Test: mute SearchGroupsResolverInMemoryTests
This commit mutes SearchGroupsResolverInMemoryTests as they fail with a thread leak. See elastic/x-pack-elasticsearch#971

Original commit: elastic/x-pack-elasticsearch@be7c988e20
2017-04-05 13:23:53 -04:00
Dimitrios Athanasiou e97035af87 Add missing logging param
Original commit: elastic/x-pack-elasticsearch@30ee528798
2017-04-05 17:00:12 +01:00
Dimitris Athanasiou 0be4082ad7 [ML] Add notification for loading snapshot (elastic/x-pack-elasticsearch#970)
As the snapshot that is loaded is an important operational
aspect of a job, this change adds a notification that displays
the loaded snapshot with its latest_record_timestamp and the
job's latest_record_timestamp. Having both allows us to discover
when a job is recovering after a node failure.

relates elastic/x-pack-elasticsearch#872

Original commit: elastic/x-pack-elasticsearch@c2dee495a2
2017-04-05 16:43:14 +01:00
Tanguy Leroux 7b48bac9f4 [Test] Gives more time to LocalExporterTests to delete indices
The test fails on slow machines because of inflight bulk requests
that hit one node while the others are stopping. This commit adds
more time (10s), equivalent to 2 to 3 collection interval, to delete
the monitoring indices. It also add TRACE logging level for the test.

Original commit: elastic/x-pack-elasticsearch@b433937946
2017-04-05 16:38:12 +02:00
Martijn van Groningen 2edbd0efeb [ML] Removed unused methods
Original commit: elastic/x-pack-elasticsearch@8b3c606147
2017-04-05 14:30:21 +02:00
Martijn van Groningen cbbc3cfe20 [ML] Slightly adjust JobProvider#getAutodetectParams(...)
Original commit: elastic/x-pack-elasticsearch@2aaf618d78
2017-04-05 14:18:00 +02:00
Colin Goodheart-Smithe 00a5759d54 [ML] Adds jobType to Job (elastic/x-pack-elasticsearch#947)
* [ML] Adds jobType to Job

This change adds `jobType` field to teh `Job` class so that when the job is written to the index a `job_type` field is written int he document. This will help separate this type of job from other new job types in the future so migrating the index to allow those new type of jobs will be easer

relates elastic/x-pack-elasticsearch#798

* Addresses review comments

Original commit: elastic/x-pack-elasticsearch@d9fd11edb3
2017-04-05 11:12:26 +01:00
Martijn van Groningen 2b5c0faebd [TEST] wait for cs version to be the same on all nodes before stopping datafeed
Original commit: elastic/x-pack-elasticsearch@6fd6224047
2017-04-05 11:37:36 +02:00
Alexander Reelsen 4b2680dac7 Tests: Ensure watch index is deleted to execute test correctly
The test requires no watch index or alias to exist, so making
sure this is deleted as part of the test.

relates elastic/x-pack-elasticsearch#920

Original commit: elastic/x-pack-elasticsearch@c1cc953ce3
2017-04-05 10:36:43 +01:00
Tim Vernum 7f0fd9e1a3 Failed LDAP SearchResults should be Exceptions (elastic/x-pack-elasticsearch#773)
When the LDAP SDK returns a SearchResult that has a non-success ResultCode, convert it to an exception and call onFailure

A configuration setting controls whether failures in referrals should be fatal (defaults to ignoring errors)

Closes: elastic/x-pack-elasticsearch#717

Original commit: elastic/x-pack-elasticsearch@4159758c2a
2017-04-05 18:40:17 +10:00
David Kyle 7c45cb7ccf Fix check style line length failure
Original commit: elastic/x-pack-elasticsearch@d37595db2b
2017-04-05 09:00:16 +01:00
David Kyle f8bb762a59 [TEST] Revert "Tests: Disable `ml/index_layout/Test unrelated index`."
This reverts commit elastic/x-pack-elasticsearch@097f559d0b.

Original commit: elastic/x-pack-elasticsearch@1f63456866
2017-04-05 08:48:23 +01:00
Martijn van Groningen 4a57c92e6f [TEST] unmuted test
Original commit: elastic/x-pack-elasticsearch@5dd80c32c7
2017-04-05 09:46:02 +02:00
David Kyle 7b45460951 [ML] Wait for dataCounts to be persisted (elastic/x-pack-elasticsearch#952)
Original commit: elastic/x-pack-elasticsearch@48ca4d7363
2017-04-05 08:45:01 +01:00
Lisa Cawley 843a0d8b3f [DOCS] Add ML documentation to master (elastic/x-pack-elasticsearch#959)
Original commit: elastic/x-pack-elasticsearch@666a10bd23
2017-04-04 15:26:39 -07:00
Igor Motov d11fbfa70c Persistent Tasks: refactor PersistentTasksService to use ActionListener (elastic/x-pack-elasticsearch#937)
PersistentTasksService methods are not using ActionListener<PersistentTask<?>> instead of PersistentTaskOperationListener.

Original commit: elastic/x-pack-elasticsearch@f95d8bda3d
2017-04-04 13:56:22 -04:00
Zachary Tong bac8f010b4 Remove unnecessary headers in Model yaml tests
Original commit: elastic/x-pack-elasticsearch@74e2a5e29a
2017-04-04 13:51:40 -04:00
Tim Vernum 26fb1a6997 Don't assume FLS arrays are pre-sorted (elastic/x-pack-elasticsearch#939)
The `FieldPermissions` class incorrectly assumed that the `granted` and `denied` arrays were
sorted, so it could do a `binarySearch` to see if `_all` was in the arrays.

Original commit: elastic/x-pack-elasticsearch@49b5875602
2017-04-04 18:14:20 +01:00
Jay Modi 1e42473f77 Remove incorrect usages of Arrays.binarySearch (elastic/x-pack-elasticsearch#954)
This is a follow-on to elastic/x-pack-elasticsearch#939, which removes the use of Arrays.binarySearch in the FieldPermissions
class. This change removes other incorrect uses in the rest of the x-pack code and replaces them
with a stream based implementation.

Original commit: elastic/x-pack-elasticsearch@ccca7e9bad
2017-04-04 17:48:58 +01:00
Dimitris Athanasiou c9834bc826 [ML] Require max time aggregation to ensure correct datafeed restart (elastic/x-pack-elasticsearch#948)
Before this change, aggregation datafeeds used the histogram bucket
key as the record timestamp that is posted to the job. That meant
that the latest_record_timestamp at the end of a datafeed run was
the start of the latest seen histogram bucket. Upon continuing the
datafeed, the search starts from one millisecond after the
latest_record_timestamp. Hence, data may be fetched for a second time.

This change requires a max aggregation on the time_field nested in
the histogram bucket. It then reads the timestamp from that agg.
This ensures datafeed can restart without duplicating data.

relates elastic/x-pack-elasticsearch#874

Original commit: elastic/x-pack-elasticsearch@f820efa866
2017-04-04 17:15:44 +01:00
Zachary Tong 2153c71e8f Wait for datafeed index to hit yellow before proceeding (elastic/x-pack-elasticsearch#940)
relates elastic/x-pack-elasticsearch#783

Original commit: elastic/x-pack-elasticsearch@838482e645
2017-04-04 16:14:02 +00:00
Dimitris Athanasiou 4fdcedb9f9 [ML] Enable datafeed preview with future data (elastic/x-pack-elasticsearch#949)
Original commit: elastic/x-pack-elasticsearch@7f9e450edc
2017-04-04 17:08:01 +01:00
Dimitris Athanasiou 38706faec7 [ML] Improve datafeed notifications (elastic/x-pack-elasticsearch#951)
- include 'real-time' instead of now as the end time for real-time
  datafeeds
- do not notify lookback is completed when datafeed was stopped
- do not notify datafeed switch to real-time when datafeed was stopped

Relates elastic/x-pack-elasticsearch#878

Original commit: elastic/x-pack-elasticsearch@aa22f9b86f
2017-04-04 17:04:41 +01:00
Jason Tedor dee1bd1825 Respond to rename random ASCII helper methods
This commit is response to the renaming of the random ASCII helper
methods in ESTestCase. The name of this method was changed because these
methods only produce random strings generated from [a-zA-Z], not from
all ASCII characters.

Relates elastic/x-pack-elasticsearch#942

Original commit: elastic/x-pack-elasticsearch@a6085964d3
2017-04-04 11:04:37 -04:00
Martijn van Groningen 71149fb6d4 [ML] Fix NPE
Original commit: elastic/x-pack-elasticsearch@421491bce0
2017-04-04 16:15:56 +02:00
Martijn van Groningen 543319f966 [ML] Small cleanup
Original commit: elastic/x-pack-elasticsearch@78a99258dd
2017-04-04 16:10:09 +02:00
David Kyle 778e960d3e [ML] Remove periodic persistence of datacounts (elastic/x-pack-elasticsearch#944)
Original commit: elastic/x-pack-elasticsearch@756f06d316
2017-04-04 14:45:15 +01:00
Igor Motov eb79be392c Persistent Tasks: check the current state in waitForPersistentTaskStatus (elastic/x-pack-elasticsearch#935)
Add a check for the current state waitForPersistentTaskStatus before waiting for the next one. This fixes sporadic failure in testPersistentActionStatusUpdate test.

relates elastic/x-pack-elasticsearch#928

Original commit: elastic/x-pack-elasticsearch@0db4ac92d2
2017-04-04 09:44:56 -04:00
David Kyle 5716b2bf16 [ML] Create job index before adding job to cluster state (elastic/x-pack-elasticsearch#943)
Original commit: elastic/x-pack-elasticsearch@1b586c7763
2017-04-04 14:25:09 +01:00
Martijn van Groningen 8a87a91897 [ML] Renamed DatafeedJobRunner to DatafeedManager
Original commit: elastic/x-pack-elasticsearch@1228488a2e
2017-04-04 14:55:58 +02:00