Commit Graph

5615 Commits

Author SHA1 Message Date
Zachary Tong e2a30331ba Refactor StartDataFeed to more intelligently wait on shard status (elastic/x-pack-elasticsearch#974)
Rather than using an async call, this leverages
the Assignment logic while selecting nodes.

Now with 300% more tests!

Original commit: elastic/x-pack-elasticsearch@300d628f72
2017-04-06 18:14:51 +00:00
David Kyle 9a9ae5edc7 [ML] Restore categoriser state after the anomaly detector (elastic/x-pack-elasticsearch#993)
Original commit: elastic/x-pack-elasticsearch@fc4205f1d6
2017-04-06 18:12:52 +01:00
Lisa Cawley e339cf82df [DOCS] Added examples for all ML job APIs (elastic/x-pack-elasticsearch#980)
Original commit: elastic/x-pack-elasticsearch@7911b53af9
2017-04-06 08:07:49 -07:00
Martijn van Groningen 5585283216 [ML] Added cause to exception.
Relates to elastic/x-pack-elasticsearch#990

Original commit: elastic/x-pack-elasticsearch@a66bc958b7
2017-04-06 16:34:24 +02:00
David Roberts 97866eb0e8 [ML] Increase timeouts for named pipe connections to native processes (elastic/x-pack-elasticsearch#987)
It has been observed that Amazon EBS volumes created from snapshots can
have very high latency the first time a given block is accessed.  This
can lead to named pipes taking longer than 2 seconds to create.
Since the native processes create their named pipes immediately after
startup, and this only takes a fraction of a second on a local disk, 2
seconds was considered a generous timeout, but it seems that in the case
of a remote NAS with lazy provisioning it's not long enough.  During
debugging a latency of just over 3 seconds was observed.  The timeouts
have been increased to 10 seconds.

relates elastic/x-pack-elasticsearch#922

Original commit: elastic/x-pack-elasticsearch@c90434c948
2017-04-06 14:42:27 +01:00
Jay Modi 92942c9327 Add support for IndexOrDocValuesQuery in FieldExtractor (elastic/x-pack-elasticsearch#976)
This commit adds support for extracting the fields from a IndexOrDocValuesQuery so that we can
still take advantage of the cache when this query is used.

relates elastic/x-pack-elasticsearch#973

Original commit: elastic/x-pack-elasticsearch@0b97202a41
2017-04-06 12:44:25 +01:00
Tim Vernum 573b421446 Support multiple indices in SecurityLifecycleService (elastic/x-pack-elasticsearch#961)
Moves the direct management of the security index from SecurityLifecycleService to IndexLifecycleManager, so that the SecurityLifecycleService can take responsibility for several indices.

Multiple security indices are required as we move away from storing multiple types in a single index.

Original commit: elastic/x-pack-elasticsearch@fde3a42b4d
2017-04-06 21:37:33 +10:00
jaymode 7efc9e1270 Test: Fix IndexAuditTrailMutedTests to set the threadpool
The IndexAuditTrailMutedTests have a threadpool but fail to set it on the test client, which causes
a NPE and tests to fail.

Original commit: elastic/x-pack-elasticsearch@d34a4ce080
2017-04-05 13:50:00 -04:00
Alexander Reelsen 16f5f44d41 Watcher: Dont check index auto creation if disabled/transportclient (elastic/x-pack-elasticsearch#975)
When watcher is disabled, no auto index creation checks should happen.

Relates elastic/elasticsearch#23919

Original commit: elastic/x-pack-elasticsearch@d924838b4d
2017-04-05 18:39:54 +01:00
jaymode a53026498f Test: mute SearchGroupsResolverInMemoryTests
This commit mutes SearchGroupsResolverInMemoryTests as they fail with a thread leak. See elastic/x-pack-elasticsearch#971

Original commit: elastic/x-pack-elasticsearch@be7c988e20
2017-04-05 13:23:53 -04:00
Dimitrios Athanasiou e97035af87 Add missing logging param
Original commit: elastic/x-pack-elasticsearch@30ee528798
2017-04-05 17:00:12 +01:00
Dimitris Athanasiou 0be4082ad7 [ML] Add notification for loading snapshot (elastic/x-pack-elasticsearch#970)
As the snapshot that is loaded is an important operational
aspect of a job, this change adds a notification that displays
the loaded snapshot with its latest_record_timestamp and the
job's latest_record_timestamp. Having both allows us to discover
when a job is recovering after a node failure.

relates elastic/x-pack-elasticsearch#872

Original commit: elastic/x-pack-elasticsearch@c2dee495a2
2017-04-05 16:43:14 +01:00
Tanguy Leroux 7b48bac9f4 [Test] Gives more time to LocalExporterTests to delete indices
The test fails on slow machines because of inflight bulk requests
that hit one node while the others are stopping. This commit adds
more time (10s), equivalent to 2 to 3 collection interval, to delete
the monitoring indices. It also add TRACE logging level for the test.

Original commit: elastic/x-pack-elasticsearch@b433937946
2017-04-05 16:38:12 +02:00
Martijn van Groningen 2edbd0efeb [ML] Removed unused methods
Original commit: elastic/x-pack-elasticsearch@8b3c606147
2017-04-05 14:30:21 +02:00
Martijn van Groningen cbbc3cfe20 [ML] Slightly adjust JobProvider#getAutodetectParams(...)
Original commit: elastic/x-pack-elasticsearch@2aaf618d78
2017-04-05 14:18:00 +02:00
Colin Goodheart-Smithe 00a5759d54 [ML] Adds jobType to Job (elastic/x-pack-elasticsearch#947)
* [ML] Adds jobType to Job

This change adds `jobType` field to teh `Job` class so that when the job is written to the index a `job_type` field is written int he document. This will help separate this type of job from other new job types in the future so migrating the index to allow those new type of jobs will be easer

relates elastic/x-pack-elasticsearch#798

* Addresses review comments

Original commit: elastic/x-pack-elasticsearch@d9fd11edb3
2017-04-05 11:12:26 +01:00
Martijn van Groningen 2b5c0faebd [TEST] wait for cs version to be the same on all nodes before stopping datafeed
Original commit: elastic/x-pack-elasticsearch@6fd6224047
2017-04-05 11:37:36 +02:00
Alexander Reelsen 4b2680dac7 Tests: Ensure watch index is deleted to execute test correctly
The test requires no watch index or alias to exist, so making
sure this is deleted as part of the test.

relates elastic/x-pack-elasticsearch#920

Original commit: elastic/x-pack-elasticsearch@c1cc953ce3
2017-04-05 10:36:43 +01:00
Tim Vernum 7f0fd9e1a3 Failed LDAP SearchResults should be Exceptions (elastic/x-pack-elasticsearch#773)
When the LDAP SDK returns a SearchResult that has a non-success ResultCode, convert it to an exception and call onFailure

A configuration setting controls whether failures in referrals should be fatal (defaults to ignoring errors)

Closes: elastic/x-pack-elasticsearch#717

Original commit: elastic/x-pack-elasticsearch@4159758c2a
2017-04-05 18:40:17 +10:00
David Kyle 7c45cb7ccf Fix check style line length failure
Original commit: elastic/x-pack-elasticsearch@d37595db2b
2017-04-05 09:00:16 +01:00
David Kyle f8bb762a59 [TEST] Revert "Tests: Disable `ml/index_layout/Test unrelated index`."
This reverts commit elastic/x-pack-elasticsearch@097f559d0b.

Original commit: elastic/x-pack-elasticsearch@1f63456866
2017-04-05 08:48:23 +01:00
Martijn van Groningen 4a57c92e6f [TEST] unmuted test
Original commit: elastic/x-pack-elasticsearch@5dd80c32c7
2017-04-05 09:46:02 +02:00
David Kyle 7b45460951 [ML] Wait for dataCounts to be persisted (elastic/x-pack-elasticsearch#952)
Original commit: elastic/x-pack-elasticsearch@48ca4d7363
2017-04-05 08:45:01 +01:00
Lisa Cawley 843a0d8b3f [DOCS] Add ML documentation to master (elastic/x-pack-elasticsearch#959)
Original commit: elastic/x-pack-elasticsearch@666a10bd23
2017-04-04 15:26:39 -07:00
Igor Motov d11fbfa70c Persistent Tasks: refactor PersistentTasksService to use ActionListener (elastic/x-pack-elasticsearch#937)
PersistentTasksService methods are not using ActionListener<PersistentTask<?>> instead of PersistentTaskOperationListener.

Original commit: elastic/x-pack-elasticsearch@f95d8bda3d
2017-04-04 13:56:22 -04:00
Zachary Tong bac8f010b4 Remove unnecessary headers in Model yaml tests
Original commit: elastic/x-pack-elasticsearch@74e2a5e29a
2017-04-04 13:51:40 -04:00
Tim Vernum 26fb1a6997 Don't assume FLS arrays are pre-sorted (elastic/x-pack-elasticsearch#939)
The `FieldPermissions` class incorrectly assumed that the `granted` and `denied` arrays were
sorted, so it could do a `binarySearch` to see if `_all` was in the arrays.

Original commit: elastic/x-pack-elasticsearch@49b5875602
2017-04-04 18:14:20 +01:00
Jay Modi 1e42473f77 Remove incorrect usages of Arrays.binarySearch (elastic/x-pack-elasticsearch#954)
This is a follow-on to elastic/x-pack-elasticsearch#939, which removes the use of Arrays.binarySearch in the FieldPermissions
class. This change removes other incorrect uses in the rest of the x-pack code and replaces them
with a stream based implementation.

Original commit: elastic/x-pack-elasticsearch@ccca7e9bad
2017-04-04 17:48:58 +01:00
Dimitris Athanasiou c9834bc826 [ML] Require max time aggregation to ensure correct datafeed restart (elastic/x-pack-elasticsearch#948)
Before this change, aggregation datafeeds used the histogram bucket
key as the record timestamp that is posted to the job. That meant
that the latest_record_timestamp at the end of a datafeed run was
the start of the latest seen histogram bucket. Upon continuing the
datafeed, the search starts from one millisecond after the
latest_record_timestamp. Hence, data may be fetched for a second time.

This change requires a max aggregation on the time_field nested in
the histogram bucket. It then reads the timestamp from that agg.
This ensures datafeed can restart without duplicating data.

relates elastic/x-pack-elasticsearch#874

Original commit: elastic/x-pack-elasticsearch@f820efa866
2017-04-04 17:15:44 +01:00
Zachary Tong 2153c71e8f Wait for datafeed index to hit yellow before proceeding (elastic/x-pack-elasticsearch#940)
relates elastic/x-pack-elasticsearch#783

Original commit: elastic/x-pack-elasticsearch@838482e645
2017-04-04 16:14:02 +00:00
Dimitris Athanasiou 4fdcedb9f9 [ML] Enable datafeed preview with future data (elastic/x-pack-elasticsearch#949)
Original commit: elastic/x-pack-elasticsearch@7f9e450edc
2017-04-04 17:08:01 +01:00
Dimitris Athanasiou 38706faec7 [ML] Improve datafeed notifications (elastic/x-pack-elasticsearch#951)
- include 'real-time' instead of now as the end time for real-time
  datafeeds
- do not notify lookback is completed when datafeed was stopped
- do not notify datafeed switch to real-time when datafeed was stopped

Relates elastic/x-pack-elasticsearch#878

Original commit: elastic/x-pack-elasticsearch@aa22f9b86f
2017-04-04 17:04:41 +01:00
Jason Tedor dee1bd1825 Respond to rename random ASCII helper methods
This commit is response to the renaming of the random ASCII helper
methods in ESTestCase. The name of this method was changed because these
methods only produce random strings generated from [a-zA-Z], not from
all ASCII characters.

Relates elastic/x-pack-elasticsearch#942

Original commit: elastic/x-pack-elasticsearch@a6085964d3
2017-04-04 11:04:37 -04:00
Martijn van Groningen 71149fb6d4 [ML] Fix NPE
Original commit: elastic/x-pack-elasticsearch@421491bce0
2017-04-04 16:15:56 +02:00
Martijn van Groningen 543319f966 [ML] Small cleanup
Original commit: elastic/x-pack-elasticsearch@78a99258dd
2017-04-04 16:10:09 +02:00
David Kyle 778e960d3e [ML] Remove periodic persistence of datacounts (elastic/x-pack-elasticsearch#944)
Original commit: elastic/x-pack-elasticsearch@756f06d316
2017-04-04 14:45:15 +01:00
Igor Motov eb79be392c Persistent Tasks: check the current state in waitForPersistentTaskStatus (elastic/x-pack-elasticsearch#935)
Add a check for the current state waitForPersistentTaskStatus before waiting for the next one. This fixes sporadic failure in testPersistentActionStatusUpdate test.

relates elastic/x-pack-elasticsearch#928

Original commit: elastic/x-pack-elasticsearch@0db4ac92d2
2017-04-04 09:44:56 -04:00
David Kyle 5716b2bf16 [ML] Create job index before adding job to cluster state (elastic/x-pack-elasticsearch#943)
Original commit: elastic/x-pack-elasticsearch@1b586c7763
2017-04-04 14:25:09 +01:00
Martijn van Groningen 8a87a91897 [ML] Renamed DatafeedJobRunner to DatafeedManager
Original commit: elastic/x-pack-elasticsearch@1228488a2e
2017-04-04 14:55:58 +02:00
Tanguy Leroux 4f1115d7f5 [Test] Reenable Monitoring Bulk tests (elastic/x-pack-elasticsearch#908)
This commit reenables the Monitoring Bulk Api REST tests. The XPackRestIT
now enables/disables the local default exporter before executing the monitoring
 tests, and also waits for the monitoring service to be started before executing
 the test.

Original commit: elastic/x-pack-elasticsearch@10b696198c
2017-04-04 14:44:40 +02:00
Martijn van Groningen 0f9bd3a08d [ML] Removed unused vars / params.
Original commit: elastic/x-pack-elasticsearch@5d3280dbec
2017-04-04 14:21:27 +02:00
David Roberts 1465711762 [ML] Wait for state processing to complete before log processing (elastic/x-pack-elasticsearch#946)
State processing can take a lot longer than log processing, even after
the C++ process has closed its end of the pipe.  The pipe has a buffer,
and indexing the state document(s) in that buffer can take more than a
second.

relates elastic/x-pack-elasticsearch#945

Original commit: elastic/x-pack-elasticsearch@65f5075028
2017-04-04 12:04:01 +01:00
Yannick Welsch e52fbdf63c Use random http port for watcher REST tests (elastic/x-pack-elasticsearch#936)
Adapts the Watcher REST tests so that they don't require a fixed http port anymore.

Original commit: elastic/x-pack-elasticsearch@14919b16e7
2017-04-04 09:02:23 +02:00
Dimitrios Athanasiou edb8c543ed [ML] Downgrade missing snapshot/quantiles log msg to warn
Original commit: elastic/x-pack-elasticsearch@fb763757d1
2017-04-03 22:18:05 +01:00
Martijn van Groningen 77dff92bef [ML] Use PersistentTasksService#waitForPersistentTaskStatus(...) to wait for job and datafeed status and
use PersistentTasksService#removeTask(...) to force close job and force stop datafeed.

Original commit: elastic/x-pack-elasticsearch@4abcf99f93
2017-04-03 20:40:31 +02:00
David Kyle 5b66c7a7ba [ML] Set job create time on server (elastic/x-pack-elasticsearch#910)
* [ML] Set job create time on server

* Job.Builder serialisation tests

* Make setCreateTime package private

Original commit: elastic/x-pack-elasticsearch@d2d75e0d7b
2017-04-03 18:30:47 +01:00
Dimitris Athanasiou 758b689f51 [ML] Improve validations for datafeed with aggregations (elastic/x-pack-elasticsearch#917)
Adds following validations:

- aggregations must contain date_histogram or histogram at the top level
- a date_histogram has to have its time_zone to UTC (or unset which
  defaults to UTC)
- a date_histogram supports calendar intervals only up to 1 week
to avoid the length variability of longer intervals
- aggregation interval must be greater than zero
- aggregation interval must be less than or equal to the bucket_span

Original commit: elastic/x-pack-elasticsearch@404496a886
2017-04-03 17:55:26 +01:00
Alexander Reelsen 84f08ee02c Tests: Ensure test does not accidentally create similar data structures
Original commit: elastic/x-pack-elasticsearch@e683de4039
2017-04-03 16:55:06 +01:00
Martijn van Groningen f0364b1218 [TEST] increased logging
Original commit: elastic/x-pack-elasticsearch@6aeac48889
2017-04-03 15:45:30 +02:00
David Roberts c0c0818d87 [ML] Fix query of all buckets with expand=true (elastic/x-pack-elasticsearch#932)
Previously if you queried all buckets with the expand=true option
you'd get an error as the code would try to do a search with
from=10000&size=10000.  This PR fixes this problem.

Followup for elastic/x-pack-elasticsearch#832

Original commit: elastic/x-pack-elasticsearch@2a5ca0998c
2017-04-03 14:15:35 +01:00