Commit Graph

5586 Commits

Author SHA1 Message Date
Martijn van Groningen 295a4049a3 [ML] Use allocation id as key in `AutodetectProcessManager#autoDetectCommunicatorByJob` map instead of job id.
Relates to elastic/x-pack-elasticsearch#921

Original commit: elastic/x-pack-elasticsearch@21383fd51c
2017-04-26 19:33:41 +02:00
Dimitris Athanasiou 1f57a53b3e [ML] Correct error message for invalid job_id (elastic/x-pack-elasticsearch#1218)
relates elastic/x-pack-elasticsearch#1211

Original commit: elastic/x-pack-elasticsearch@5389a43706
2017-04-26 18:17:06 +01:00
Yannick Welsch 444cbfc283 [TEST] Fix monitoring tests that are flaky in the presence of replicas
Many tests in monitoring use the pattern of calling first awaitMonitoringDocsCount, and then doing a search that checks certain properties, assuming
that the doc count is correct at that point. In the presence of replicas, awaitMonitoringDocsCount might not wait for all shard copies to have the
desired property. A subsequent search might then hit a shard where the property does not hold. As these tests randomize the number of replicas
(through the random_index_template), it easier to constrain awaitMonitoringDocsCount just to the primary and then do subsequent checks just by
querying the primary.

Original commit: elastic/x-pack-elasticsearch@4165beb903
2017-04-26 19:12:45 +02:00
Dimitris Athanasiou 8af91aed6a [DOCS] Add missing info in datafeed resource (elastic/x-pack-elasticsearch#1215)
Original commit: elastic/x-pack-elasticsearch@c415bc92c2
2017-04-26 10:05:27 -07:00
jaymode 4a36cd77ee Test: bind regular socket instead of server socket so connection cannot be accepted
This commit creates a single server socket that will be connected to by local sockets. The local
sockets will use the port of the previously stopped ldap server as the local port. This will
prevent the ldap library from establishing a connection. The previous use of server sockets for
this did not work on all operating systems as the backlog parameter has platform specific meaning.

Original commit: elastic/x-pack-elasticsearch@03b6bf39d4
2017-04-26 12:41:12 -04:00
David Roberts f3f9cb6d74 [ML] Stop using the management thread pool unnecessarily for ML actions (elastic/x-pack-elasticsearch#1213)
The management thread pool only has 5 threads and clogging it up makes
monitoring think the cluster is dead.

relates elastic/x-pack-elasticsearch#1210

Original commit: elastic/x-pack-elasticsearch@f4ad7578d9
2017-04-26 17:17:26 +01:00
jaymode 6b4db0fc36 Test: remove usage of forbidden api (socket connect)
Original commit: elastic/x-pack-elasticsearch@5903973762
2017-04-26 11:39:27 -04:00
jaymode 123a660720 Test: add awaits fix for SessionFactoryLoadBalancingTests
Original commit: elastic/x-pack-elasticsearch@6412d12283
2017-04-26 11:25:30 -04:00
jaymode c9d039525c Test: add more debug output for SessionFactoryLoadBalancingTests failures
Original commit: elastic/x-pack-elasticsearch@206048b94d
2017-04-26 11:15:07 -04:00
jaymode c1c66f38ea Test: specify a timeout for background operation
This commit adds a timeout for the expiration of invalidated tokens so that we can expect that the
requests will have been finished before we do the assertions on the internal test cluster.

Original commit: elastic/x-pack-elasticsearch@2928706224
2017-04-26 10:34:17 -04:00
Dimitrios Athanasiou 708190f356 [TEST] Add model snapshot restore integration test
Relates elastic/x-pack-elasticsearch#882

Original commit: elastic/x-pack-elasticsearch@fbb983e63b
2017-04-26 14:29:55 +01:00
Jay Modi 295051ee8c Add a short-lived token based access mechanism (elastic/x-pack-elasticsearch#1029)
This commit adds a token based access mechanism that is a subset of the OAuth 2.0 protocol. The
token mechanism takes the same values as a OAuth 2 standard (defined in RFC 6749 and RFC 6750),
but differs in that we use XContent for the body instead of form encoded values. Additionally, this
PR provides a mechanism for expiration of a token; this can be used to implement logout
functionality that prevents the token from being used again.

The actual tokens are encrypted using AES-GCM, which also provides authentication. The key for
encryption is derived from a salt value and a passphrase that is stored on each node in the
secure settings store. By default, the tokens have an expiration time of 20 minutes and is
configurable up to a maximum of one hour.

Relates elastic/x-pack-elasticsearch#8

Original commit: elastic/x-pack-elasticsearch@3d201ac2bf
2017-04-26 08:00:03 -04:00
Martijn van Groningen c6c63c471c The xpack side of elastic/x-pack-elasticsearch#24115
Original commit: elastic/x-pack-elasticsearch@862180a169
2017-04-26 11:28:01 +02:00
Lisa Cawley aa421af2fc [DOCS] Stop and start data feeds in ML Getting Started (elastic/x-pack-elasticsearch#1206)
Original commit: elastic/x-pack-elasticsearch@b938c19695
2017-04-25 17:28:10 -07:00
Lisa Cawley 268f5a95af [DOCS] Add authorization section to ML API pages (elastic/x-pack-elasticsearch#1205)
* [DOCS] Edits for description of ML results resource

* [DOCS] Adds Authorization section to ML API pages

Original commit: elastic/x-pack-elasticsearch@797761e7ae
2017-04-25 14:21:42 -07:00
Dimitrios Athanasiou 9828161cff [TEST] Remove broken test from jobs_get_result_categories.yaml
The YAML framework randomly sends the body wrapped in a `source`
field if the API supports GET. Our API doesn't handle `source`.
Taking the test out.

relates elastic/x-pack-elasticsearch#1200

Original commit: elastic/x-pack-elasticsearch@fa77c071b2
2017-04-25 17:07:30 +01:00
Martijn van Groningen ee650b3189 [ml] Use allocation id as key in `runningDatafeeds` map instead of datafeed id
Original commit: elastic/x-pack-elasticsearch@156e3275b1
2017-04-25 17:55:27 +02:00
Dimitris Athanasiou 32128894a5 [ML] Delete interim results after job re-opening (elastic/x-pack-elasticsearch#1194)
This is an issue where a bucket can have both interim results and
non-interim results, a bucket should never have both at the same time.
The steps to cause this situation are:

1. Flush a running job and create interim results
2. Close that job (this does not delete interim results)
3. Re-open the job and POST data
4. The job will eventually emit a bucket result which mingles with the
existing interim results

Normally interim results are deleted by AutoDetectResultProcessor when a
bucket is parsed following a flush command. Because of the close and
re-opening of the job AutoDetectResultProcessor no longer knows that a
previous flush command creating interim results.

The fix is to always delete interim results the first time
AutoDetectResultProcessor sees a bucket.

relates elastic/x-pack-elasticsearch#1188

Original commit: elastic/x-pack-elasticsearch@5326455f54
2017-04-25 16:32:58 +01:00
Lisa Cawley 019b1f7ece [DOCS] Fix doc build errors for elastic/x-pack-elasticsearch#1197 (elastic/x-pack-elasticsearch#1199)
Original commit: elastic/x-pack-elasticsearch@30f69513ab
2017-04-25 08:23:51 -07:00
Sophie Chang 4b39d858b7 [DOCS] Major re-work of resultsresource (elastic/x-pack-elasticsearch#1197)
Original commit: elastic/x-pack-elasticsearch@8e9a004dd2
2017-04-25 08:23:30 -07:00
jaymode ef571568f4 Test: ensure ports that should fail are not bound to by other tests
In the SessionFactoryLoadBalancingTests, we sometime want a connection to a certain IP and Port to
fail as a way to mock an unresponsive/disconnected LDAP server. The test does this by starting up
multiple LDAP servers and then shutting some down. When the server is shut down the port that it
was bound to is open for another process or test to bind to, which can lead to sporadic failures in
CI. This change is a best effort attempt to prevent this by binding a server socket to the port and
filling its backlog so other connections should fail.

Relates elastic/x-pack-elasticsearch#1195

Original commit: elastic/x-pack-elasticsearch@b31a560c93
2017-04-25 11:17:05 -04:00
David Kyle 2d9fd0ed16 [ML] Only audit job creation after it was successful (elastic/x-pack-elasticsearch#1198)
Original commit: elastic/x-pack-elasticsearch@37af6cbf69
2017-04-25 16:07:47 +01:00
Dimitrios Athanasiou 612dcda8ae [TEST] Increase base job bucket_span to 1h
The DatafeedJobsIT.testRealtime test fails from time to time.
The test seems to take a long time to execute the flush action
after the lookback. This could make sense as the test produces
a few records over the span of a week with 5 minutes bucket_span.
Thus, flush will end up doing a lot of word to create results
for so many buckets.

This change increases the bucket_span to 1 hour. Hopefully, this
will stop the failures.

Relates elastic/x-pack-elasticsearch#1162

Original commit: elastic/x-pack-elasticsearch@4366907371
2017-04-25 15:32:44 +01:00
Dimitris Athanasiou 7f64f37c46 [ML] Fix POST version of get categories API (elastic/x-pack-elasticsearch#1184)
Also improves PageParams parsing to fill in defaults.

relates elastic/x-pack-elasticsearch#1180

Original commit: elastic/x-pack-elasticsearch@fccd7795ca
2017-04-25 14:44:10 +01:00
Martijn van Groningen 415d40e6fc test: improved assertion
Original commit: elastic/x-pack-elasticsearch@8361d2e27d
2017-04-25 12:16:01 +02:00
Martijn van Groningen a3e7c65ba4 [ML] Upon task cancel stop datafeed immediately.
Original commit: elastic/x-pack-elasticsearch@0401ca3d33
2017-04-25 11:56:44 +02:00
David Kyle bd40dd36c9 [ML] Hide partition_scores field if empty (elastic/x-pack-elasticsearch#1189)
Original commit: elastic/x-pack-elasticsearch@5dec266e0d
2017-04-25 10:48:53 +01:00
Martijn van Groningen 6c0613f36f fix mocking in tests
Original commit: elastic/x-pack-elasticsearch@b43de1ea2b
2017-04-25 09:52:12 +02:00
lcawley 85f573156f [DOCS] Merge branch 'master' of https://github.com/elastic/x-pack-elasticsearch
Original commit: elastic/x-pack-elasticsearch@d1d5ae545b
2017-04-24 16:33:46 -07:00
Lisa Cawley ee612a3dd8 [DOCS] Add ML Getting Started job analysis pages (elastic/x-pack-elasticsearch#1185)
* [DOCS] ML getting started file extraction

* [DOCS] ML Getting Started exploring job results

Original commit: elastic/x-pack-elasticsearch@7b46e7beb3
2017-04-24 16:31:48 -07:00
Ryan Ernst 105b689ec0 Templates: Convert template uses to ScriptService.compileTemplate (elastic/x-pack-elasticsearch#1170)
This is the xpack side of elastic/elasticsearch#24280

Original commit: elastic/x-pack-elasticsearch@90b7b2c6b7
2017-04-24 15:45:26 -07:00
lcawley 918f4fb962 [DOCS] Clarify force parameter for close job ML API
Original commit: elastic/x-pack-elasticsearch@c170d43757
2017-04-24 11:32:50 -07:00
Lisa Cawley af47002654 [DOCS] Edits for ML doc changes in elastic/x-pack-elasticsearch#1169 (elastic/x-pack-elasticsearch#1177)
Original commit: elastic/x-pack-elasticsearch@84f8b4b9bd
2017-04-24 10:56:26 -07:00
lcawley ee59f18d42 [DOCS] Fixed X-Pack Reference build error
Original commit: elastic/x-pack-elasticsearch@69d20f4346
2017-04-24 10:08:04 -07:00
Sophie Chang 528ac3d902 [DOCS] ML API docs review (elastic/x-pack-elasticsearch#1169)
* [DOCS] Fix for prelertcategory

* [DOCS] _preview returns a page of data

* [DOCS] Added adv options e.g. background_persist_interval"

* [DOCS] Clarify meanings of model_snapshot params

* [DOCS] Format fixes

* [DOCS] Include _all keyword

* [DOCS] Explain retain.

* [DOCS] Further explanations for model size limits

* [DOCS] Format fixes in quick ref

* [DOCS] Update for exclude_interim

* [DOCS] Update for exclude_interim

* [DOCS] Update for exclude_interim

Original commit: elastic/x-pack-elasticsearch@cdd2fcefdd
2017-04-24 10:07:51 -07:00
Jay Modi 2c2261881d Fix support for elliptic curve certificates in PEM files (elastic/x-pack-elasticsearch#1050)
This commit fixes the support for elliptic curve certificates that are specified as a PEM file.
These certificates and private keys can now be read properly and a integration test was added to
ensure that TLS also functions correctly with these certificates.

Original commit: elastic/x-pack-elasticsearch@6d6d579c88
2017-04-24 11:31:10 -04:00
Jay Modi d2b909fe5a Test: remove roles from user on remote cluster (elastic/x-pack-elasticsearch#1144)
This change removes the roles from the user in the remote cluster to show that the roles passed
from the source cluster are used in evaluating the user's permissions during multi-cluster search.

Original commit: elastic/x-pack-elasticsearch@35658168fb
2017-04-24 09:18:26 -04:00
Martijn van Groningen a98d593576 [ML] Use JobTaskStatus#staleTask(...) instead of PersistentTask#needsReassignement(...) for checking whether a job task is stale when allocation a datafeed to a node.
Original commit: elastic/x-pack-elasticsearch@0952c455fe
2017-04-24 14:56:25 +02:00
David Kyle 0b267242f1 [ML] Refresh indices after job deletion (elastic/x-pack-elasticsearch#1174)
* Refresh indices after job deletion

* Make refresh index option explicit

Original commit: elastic/x-pack-elasticsearch@d3c819966b
2017-04-24 13:55:36 +01:00
Dimitrios Athanasiou 9a4fa90ada [DOCS] Fix snippet type from json to js
Original commit: elastic/x-pack-elasticsearch@15fc02d734
2017-04-24 12:57:58 +01:00
Jay Modi 1c1837fddd Reduce spamming of logs on common SSL exceptions (elastic/x-pack-elasticsearch#1083)
This commit reduces spamming of the logs when a common SSL exception is encountered such as a
client not trusting the server's certificate or a plaintext request sent to a channel that expects
TLS traffic.

relates elastic/x-pack-elasticsearch#1062

Original commit: elastic/x-pack-elasticsearch@94959e79f6
2017-04-24 07:51:24 -04:00
Jay Modi f063af9ee3 Log INFO message when reloading SSL configuration file changes (elastic/x-pack-elasticsearch#1082)
This commit adds a INFO level log message to indicate that a file was reloaded and the SSL contexts
using the file were also updated.

relates elastic/x-pack-elasticsearch#1063

Original commit: elastic/x-pack-elasticsearch@f53f7019de
2017-04-24 07:46:14 -04:00
Dimitrios Athanasiou 5f9a48efbf [TEST] Whitelist ml getting-started doc from tests
... until we fix the whole file to run tests

Original commit: elastic/x-pack-elasticsearch@28108707ce
2017-04-24 11:54:31 +01:00
David Kyle 2c6a4d8627 [ML] Fix filter influencers by score
Add more test coverage

Original commit: elastic/x-pack-elasticsearch@45b06d2a5b
2017-04-24 11:31:51 +01:00
Martijn van Groningen 5dc8c71e65 [TEST] Increased datafeed logging
Original commit: elastic/x-pack-elasticsearch@403bc28dea
2017-04-24 10:48:53 +02:00
Lisa Cawley d3a2e34f9d [DOCS] Update ML getting started docs with server-metrics sample data (elastic/x-pack-elasticsearch#1166)
* [DOCS] Add role info to ML getting started docs

* [DOCS] Getting started with sample ML data

* [DOCS] Getting started with server-metrics sample data

Original commit: elastic/x-pack-elasticsearch@2f268f87b4
2017-04-21 19:01:59 -07:00
Ryan Ernst c5b14197d4 Remove uses of ScriptService.executable which takes Script (elastic/x-pack-elasticsearch#1164)
This is the xpack side of elastic/elasticsearch#24264

Original commit: elastic/x-pack-elasticsearch@ac36bc32aa
2017-04-21 17:52:27 -07:00
David Kyle 72a9bffff8 [ML] Proceed on conflict when deleting job docs. (elastic/x-pack-elasticsearch#1157)
Original commit: elastic/x-pack-elasticsearch@b1b98ec268
2017-04-21 16:45:14 +01:00
Dimitris Athanasiou e0b6630e3d [ML] Add the current job state in conflict error messages (elastic/x-pack-elasticsearch#1158)
Relates elastic/x-pack-elasticsearch#878

Original commit: elastic/x-pack-elasticsearch@2be8b6c9a1
2017-04-21 16:42:39 +01:00
Zachary Tong d993926ba4 [DOCS] Re-enable XPack doc testing (elastic/x-pack-elasticsearch#1146)
Adds a gradle build for documentation testing, partially stolen from the xpack meta repo. Updated to make work with the current setup

index.asciidoc was updated/enabled as a test, and various pre-existing tests appear to be working. There are a large number of tests still ignored due to missing proper // CONSOLE tags

Original commit: elastic/x-pack-elasticsearch@1d596f0be3
2017-04-21 11:23:27 -04:00