Commit Graph

353 Commits

Author SHA1 Message Date
jaymode 7efc9e1270 Test: Fix IndexAuditTrailMutedTests to set the threadpool
The IndexAuditTrailMutedTests have a threadpool but fail to set it on the test client, which causes
a NPE and tests to fail.

Original commit: elastic/x-pack-elasticsearch@d34a4ce080
2017-04-05 13:50:00 -04:00
Alexander Reelsen 16f5f44d41 Watcher: Dont check index auto creation if disabled/transportclient (elastic/x-pack-elasticsearch#975)
When watcher is disabled, no auto index creation checks should happen.

Relates elastic/elasticsearch#23919

Original commit: elastic/x-pack-elasticsearch@d924838b4d
2017-04-05 18:39:54 +01:00
jaymode a53026498f Test: mute SearchGroupsResolverInMemoryTests
This commit mutes SearchGroupsResolverInMemoryTests as they fail with a thread leak. See elastic/x-pack-elasticsearch#971

Original commit: elastic/x-pack-elasticsearch@be7c988e20
2017-04-05 13:23:53 -04:00
Dimitrios Athanasiou e97035af87 Add missing logging param
Original commit: elastic/x-pack-elasticsearch@30ee528798
2017-04-05 17:00:12 +01:00
Dimitris Athanasiou 0be4082ad7 [ML] Add notification for loading snapshot (elastic/x-pack-elasticsearch#970)
As the snapshot that is loaded is an important operational
aspect of a job, this change adds a notification that displays
the loaded snapshot with its latest_record_timestamp and the
job's latest_record_timestamp. Having both allows us to discover
when a job is recovering after a node failure.

relates elastic/x-pack-elasticsearch#872

Original commit: elastic/x-pack-elasticsearch@c2dee495a2
2017-04-05 16:43:14 +01:00
Tanguy Leroux 7b48bac9f4 [Test] Gives more time to LocalExporterTests to delete indices
The test fails on slow machines because of inflight bulk requests
that hit one node while the others are stopping. This commit adds
more time (10s), equivalent to 2 to 3 collection interval, to delete
the monitoring indices. It also add TRACE logging level for the test.

Original commit: elastic/x-pack-elasticsearch@b433937946
2017-04-05 16:38:12 +02:00
Martijn van Groningen 2edbd0efeb [ML] Removed unused methods
Original commit: elastic/x-pack-elasticsearch@8b3c606147
2017-04-05 14:30:21 +02:00
Martijn van Groningen cbbc3cfe20 [ML] Slightly adjust JobProvider#getAutodetectParams(...)
Original commit: elastic/x-pack-elasticsearch@2aaf618d78
2017-04-05 14:18:00 +02:00
Colin Goodheart-Smithe 00a5759d54 [ML] Adds jobType to Job (elastic/x-pack-elasticsearch#947)
* [ML] Adds jobType to Job

This change adds `jobType` field to teh `Job` class so that when the job is written to the index a `job_type` field is written int he document. This will help separate this type of job from other new job types in the future so migrating the index to allow those new type of jobs will be easer

relates elastic/x-pack-elasticsearch#798

* Addresses review comments

Original commit: elastic/x-pack-elasticsearch@d9fd11edb3
2017-04-05 11:12:26 +01:00
Alexander Reelsen 4b2680dac7 Tests: Ensure watch index is deleted to execute test correctly
The test requires no watch index or alias to exist, so making
sure this is deleted as part of the test.

relates elastic/x-pack-elasticsearch#920

Original commit: elastic/x-pack-elasticsearch@c1cc953ce3
2017-04-05 10:36:43 +01:00
Tim Vernum 7f0fd9e1a3 Failed LDAP SearchResults should be Exceptions (elastic/x-pack-elasticsearch#773)
When the LDAP SDK returns a SearchResult that has a non-success ResultCode, convert it to an exception and call onFailure

A configuration setting controls whether failures in referrals should be fatal (defaults to ignoring errors)

Closes: elastic/x-pack-elasticsearch#717

Original commit: elastic/x-pack-elasticsearch@4159758c2a
2017-04-05 18:40:17 +10:00
David Kyle 7c45cb7ccf Fix check style line length failure
Original commit: elastic/x-pack-elasticsearch@d37595db2b
2017-04-05 09:00:16 +01:00
David Kyle f8bb762a59 [TEST] Revert "Tests: Disable `ml/index_layout/Test unrelated index`."
This reverts commit elastic/x-pack-elasticsearch@097f559d0b.

Original commit: elastic/x-pack-elasticsearch@1f63456866
2017-04-05 08:48:23 +01:00
David Kyle 7b45460951 [ML] Wait for dataCounts to be persisted (elastic/x-pack-elasticsearch#952)
Original commit: elastic/x-pack-elasticsearch@48ca4d7363
2017-04-05 08:45:01 +01:00
Igor Motov d11fbfa70c Persistent Tasks: refactor PersistentTasksService to use ActionListener (elastic/x-pack-elasticsearch#937)
PersistentTasksService methods are not using ActionListener<PersistentTask<?>> instead of PersistentTaskOperationListener.

Original commit: elastic/x-pack-elasticsearch@f95d8bda3d
2017-04-04 13:56:22 -04:00
Zachary Tong bac8f010b4 Remove unnecessary headers in Model yaml tests
Original commit: elastic/x-pack-elasticsearch@74e2a5e29a
2017-04-04 13:51:40 -04:00
Tim Vernum 26fb1a6997 Don't assume FLS arrays are pre-sorted (elastic/x-pack-elasticsearch#939)
The `FieldPermissions` class incorrectly assumed that the `granted` and `denied` arrays were
sorted, so it could do a `binarySearch` to see if `_all` was in the arrays.

Original commit: elastic/x-pack-elasticsearch@49b5875602
2017-04-04 18:14:20 +01:00
Jay Modi 1e42473f77 Remove incorrect usages of Arrays.binarySearch (elastic/x-pack-elasticsearch#954)
This is a follow-on to elastic/x-pack-elasticsearch#939, which removes the use of Arrays.binarySearch in the FieldPermissions
class. This change removes other incorrect uses in the rest of the x-pack code and replaces them
with a stream based implementation.

Original commit: elastic/x-pack-elasticsearch@ccca7e9bad
2017-04-04 17:48:58 +01:00
Dimitris Athanasiou c9834bc826 [ML] Require max time aggregation to ensure correct datafeed restart (elastic/x-pack-elasticsearch#948)
Before this change, aggregation datafeeds used the histogram bucket
key as the record timestamp that is posted to the job. That meant
that the latest_record_timestamp at the end of a datafeed run was
the start of the latest seen histogram bucket. Upon continuing the
datafeed, the search starts from one millisecond after the
latest_record_timestamp. Hence, data may be fetched for a second time.

This change requires a max aggregation on the time_field nested in
the histogram bucket. It then reads the timestamp from that agg.
This ensures datafeed can restart without duplicating data.

relates elastic/x-pack-elasticsearch#874

Original commit: elastic/x-pack-elasticsearch@f820efa866
2017-04-04 17:15:44 +01:00
Zachary Tong 2153c71e8f Wait for datafeed index to hit yellow before proceeding (elastic/x-pack-elasticsearch#940)
relates elastic/x-pack-elasticsearch#783

Original commit: elastic/x-pack-elasticsearch@838482e645
2017-04-04 16:14:02 +00:00
Dimitris Athanasiou 4fdcedb9f9 [ML] Enable datafeed preview with future data (elastic/x-pack-elasticsearch#949)
Original commit: elastic/x-pack-elasticsearch@7f9e450edc
2017-04-04 17:08:01 +01:00
Dimitris Athanasiou 38706faec7 [ML] Improve datafeed notifications (elastic/x-pack-elasticsearch#951)
- include 'real-time' instead of now as the end time for real-time
  datafeeds
- do not notify lookback is completed when datafeed was stopped
- do not notify datafeed switch to real-time when datafeed was stopped

Relates elastic/x-pack-elasticsearch#878

Original commit: elastic/x-pack-elasticsearch@aa22f9b86f
2017-04-04 17:04:41 +01:00
Jason Tedor dee1bd1825 Respond to rename random ASCII helper methods
This commit is response to the renaming of the random ASCII helper
methods in ESTestCase. The name of this method was changed because these
methods only produce random strings generated from [a-zA-Z], not from
all ASCII characters.

Relates elastic/x-pack-elasticsearch#942

Original commit: elastic/x-pack-elasticsearch@a6085964d3
2017-04-04 11:04:37 -04:00
Martijn van Groningen 71149fb6d4 [ML] Fix NPE
Original commit: elastic/x-pack-elasticsearch@421491bce0
2017-04-04 16:15:56 +02:00
Martijn van Groningen 543319f966 [ML] Small cleanup
Original commit: elastic/x-pack-elasticsearch@78a99258dd
2017-04-04 16:10:09 +02:00
David Kyle 778e960d3e [ML] Remove periodic persistence of datacounts (elastic/x-pack-elasticsearch#944)
Original commit: elastic/x-pack-elasticsearch@756f06d316
2017-04-04 14:45:15 +01:00
Igor Motov eb79be392c Persistent Tasks: check the current state in waitForPersistentTaskStatus (elastic/x-pack-elasticsearch#935)
Add a check for the current state waitForPersistentTaskStatus before waiting for the next one. This fixes sporadic failure in testPersistentActionStatusUpdate test.

relates elastic/x-pack-elasticsearch#928

Original commit: elastic/x-pack-elasticsearch@0db4ac92d2
2017-04-04 09:44:56 -04:00
David Kyle 5716b2bf16 [ML] Create job index before adding job to cluster state (elastic/x-pack-elasticsearch#943)
Original commit: elastic/x-pack-elasticsearch@1b586c7763
2017-04-04 14:25:09 +01:00
Martijn van Groningen 8a87a91897 [ML] Renamed DatafeedJobRunner to DatafeedManager
Original commit: elastic/x-pack-elasticsearch@1228488a2e
2017-04-04 14:55:58 +02:00
Tanguy Leroux 4f1115d7f5 [Test] Reenable Monitoring Bulk tests (elastic/x-pack-elasticsearch#908)
This commit reenables the Monitoring Bulk Api REST tests. The XPackRestIT
now enables/disables the local default exporter before executing the monitoring
 tests, and also waits for the monitoring service to be started before executing
 the test.

Original commit: elastic/x-pack-elasticsearch@10b696198c
2017-04-04 14:44:40 +02:00
Martijn van Groningen 0f9bd3a08d [ML] Removed unused vars / params.
Original commit: elastic/x-pack-elasticsearch@5d3280dbec
2017-04-04 14:21:27 +02:00
David Roberts 1465711762 [ML] Wait for state processing to complete before log processing (elastic/x-pack-elasticsearch#946)
State processing can take a lot longer than log processing, even after
the C++ process has closed its end of the pipe.  The pipe has a buffer,
and indexing the state document(s) in that buffer can take more than a
second.

relates elastic/x-pack-elasticsearch#945

Original commit: elastic/x-pack-elasticsearch@65f5075028
2017-04-04 12:04:01 +01:00
Dimitrios Athanasiou edb8c543ed [ML] Downgrade missing snapshot/quantiles log msg to warn
Original commit: elastic/x-pack-elasticsearch@fb763757d1
2017-04-03 22:18:05 +01:00
Martijn van Groningen 77dff92bef [ML] Use PersistentTasksService#waitForPersistentTaskStatus(...) to wait for job and datafeed status and
use PersistentTasksService#removeTask(...) to force close job and force stop datafeed.

Original commit: elastic/x-pack-elasticsearch@4abcf99f93
2017-04-03 20:40:31 +02:00
David Kyle 5b66c7a7ba [ML] Set job create time on server (elastic/x-pack-elasticsearch#910)
* [ML] Set job create time on server

* Job.Builder serialisation tests

* Make setCreateTime package private

Original commit: elastic/x-pack-elasticsearch@d2d75e0d7b
2017-04-03 18:30:47 +01:00
Dimitris Athanasiou 758b689f51 [ML] Improve validations for datafeed with aggregations (elastic/x-pack-elasticsearch#917)
Adds following validations:

- aggregations must contain date_histogram or histogram at the top level
- a date_histogram has to have its time_zone to UTC (or unset which
  defaults to UTC)
- a date_histogram supports calendar intervals only up to 1 week
to avoid the length variability of longer intervals
- aggregation interval must be greater than zero
- aggregation interval must be less than or equal to the bucket_span

Original commit: elastic/x-pack-elasticsearch@404496a886
2017-04-03 17:55:26 +01:00
Alexander Reelsen 84f08ee02c Tests: Ensure test does not accidentally create similar data structures
Original commit: elastic/x-pack-elasticsearch@e683de4039
2017-04-03 16:55:06 +01:00
Martijn van Groningen f0364b1218 [TEST] increased logging
Original commit: elastic/x-pack-elasticsearch@6aeac48889
2017-04-03 15:45:30 +02:00
David Roberts c0c0818d87 [ML] Fix query of all buckets with expand=true (elastic/x-pack-elasticsearch#932)
Previously if you queried all buckets with the expand=true option
you'd get an error as the code would try to do a search with
from=10000&size=10000.  This PR fixes this problem.

Followup for elastic/x-pack-elasticsearch#832

Original commit: elastic/x-pack-elasticsearch@2a5ca0998c
2017-04-03 14:15:35 +01:00
Martijn van Groningen 51662c83eb [TEST] If bulk item failed then log why.
Original commit: elastic/x-pack-elasticsearch@e7cc96460d
2017-04-03 13:49:43 +02:00
David Kyle 622c5ae166 [ML] Remove JobManagers dependency on JobResultsPerister (elastic/x-pack-elasticsearch#915)
* Remove JobManagers dependency on JobResultsPerister

* Remove unneeded call to refresh the state index

Original commit: elastic/x-pack-elasticsearch@0b2351bba7
2017-04-03 12:00:52 +01:00
David Roberts 2d01c3884b [ML] Don't get stats for jobs that are being deleted (elastic/x-pack-elasticsearch#899)
If jobs are being deleted then the operations required to get stats
could fail with unexpected exceptions.  When stats for multiple jobs
were being requested, this would previously cause the whole operation
to fail.

This commit changes the stats endpoint to ignore jobs that are being
deleted.

Fixes elastic/prelert-legacy#837

Original commit: elastic/x-pack-elasticsearch@6ac141a987
2017-04-03 11:04:34 +01:00
David Roberts c5a2fba70a [ML] Ignore ResourceNotFoundException executing C++ process update (elastic/x-pack-elasticsearch#927)
When this happens it means the job has been deleted, which in turn means
the C++ process has been stopped, so there's no need to send it a message
and hence no problem worth logging a stack trace for.

This differs from elastic/x-pack-elasticsearch#896 because elastic/x-pack-elasticsearch#896 was for a similar situation with
closed jobs, whereas this one is for deleted jobs.

Original commit: elastic/x-pack-elasticsearch@9bb4e98fe7
2017-04-03 10:53:32 +01:00
David Kyle 30745b8dd7 [ML] Wait for model snapshot update when closing the results processor (elastic/x-pack-elasticsearch#916)
* [ML] Wait for model snapshot update when closing the results processor

* Review comments

Original commit: elastic/x-pack-elasticsearch@eeb6c3b8d8
2017-04-03 10:01:21 +01:00
Tanguy Leroux 73ddc2323b [Test] Relax LocalExporterTests on node_stats checks
The test is too rigid on checking the right number of node_stats documented that are collected. It happens if a node takes time to start, the node_stats count % numNodes will always be different than 0.

It also adds more logging for LocalBulk failures.

Original commit: elastic/x-pack-elasticsearch@1ebb20b6f6
2017-04-03 10:36:12 +02:00
Simon Willnauer 5e6bfb9a82 Fix cross cluster search with security (elastic/x-pack-elasticsearch#904)
This commit adds an integration test that runs basic cross cluster search
actions across 2 clusters with security installed. This commit also fixes
several issues with respect to internal actions and proxy actions in the context
of cross cluster search.

Relates to elastic/elasticsearch#23830
relates elastic/x-pack-elasticsearch#892

Original commit: elastic/x-pack-elasticsearch@2e5486c259
2017-04-03 09:53:48 +02:00
Igor Motov 4115336f5a Persistent Tasks: require correct allocation id for status updates (elastic/x-pack-elasticsearch#923)
In order to prevent tasks state updates by stale executors, this commit adds a check for correct allocation id during status update operation.

Original commit: elastic/x-pack-elasticsearch@b94eb0e863
2017-04-01 18:17:07 -04:00
Tanguy Leroux b5a285fd83 [Monitoring] Update LocalExporterTests (elastic/x-pack-elasticsearch#835)
This commit changes the LocalExporterTests so that it now test
various randomized cases in a single test. This should speed up
the test as well as minimize the failures due to multiple start
/stop of the exporter. It also uses the MonitoringBulk API
instead of calling the Exporter instances, which makes more sense
since it is the normal way to index monitoring documents.

Related  elastic/x-pack-elasticsearch#416

Original commit: elastic/x-pack-elasticsearch@f8a4af15cd
2017-04-01 10:42:09 +02:00
Chris Earle 16d963b8e3 [Monitoring] Use 'index_patterns' instead of 'template' (elastic/x-pack-elasticsearch#883)
Original commit: elastic/x-pack-elasticsearch@f1a217b597
2017-03-31 17:17:02 -04:00
Igor Motov d7e4390490 Persistent Tasks: Add waitForPersistentTaskStatus method (elastic/x-pack-elasticsearch#901)
This method allows to wait for tasks to change their status to match the supplied predicate.

Original commit: elastic/x-pack-elasticsearch@9f5d4104a0
2017-03-31 16:05:34 -04:00