Commit Graph

5560 Commits

Author SHA1 Message Date
Zachary Tong 6147e2ba6a Remove now-unnecessary cast
Original commit: elastic/x-pack-elasticsearch@fcbbe5a919
2017-04-27 17:16:30 -04:00
Dimitrios Athanasiou f341b336a5 [ML] Correctly pass job_id from job task to logger
Original commit: elastic/x-pack-elasticsearch@1c73bccaec
2017-04-27 20:02:15 +01:00
Lisa Cawley 485be502f4 [DOCS] Doc build fixes and edits for elastic/x-pack-elasticsearch#1237 (elastic/x-pack-elasticsearch#1241)
* [DOCS] Fixing doc build error

* [DOCS] Edits on ML content for elastic/x-pack-elasticsearch#1237

Original commit: elastic/x-pack-elasticsearch@cd4d404dee
2017-04-27 11:22:59 -07:00
Sophie Chang ffb3bb6493 [DOCS] Overall review (elastic/x-pack-elasticsearch#1237)
* [DOCS] Overall review

* [DOCS] General review

* [DOCS] typo

* [DOCS] Fix for processed_record_count with aggs

* [DOCS] Added latency tbd

Original commit: elastic/x-pack-elasticsearch@9e8cf664c1
2017-04-27 11:22:42 -07:00
Lisa Cawley 642b1f7c19 [DOCS] More edits for ML Getting Started (elastic/x-pack-elasticsearch#1238)
Original commit: elastic/x-pack-elasticsearch@69be11bfd2
2017-04-27 09:15:50 -07:00
Dimitrios Athanasiou b796388431 [ML] Delete intervening results after now as well
When we revert to snapshot, if we delete intervening results
we should delete with an open end on the time range for the
case when future data has been posted to the job.

Original commit: elastic/x-pack-elasticsearch@c3f5e8f19e
2017-04-27 16:38:23 +01:00
Nik Everett 677ea404f7 Remove most usages of 1-arg Script ctor (elastic/x-pack-elasticsearch#1207)
The one argument ctor for `Script` creates a script with the
default language but most usages of are for testing and either
don't care about the language or are for use with
`MockScriptEngine`. This replaces most usages of the one argument
ctor on `Script` with calls to `ESTestCase#mockScript` to make
it clear that the tests don't need the default scripting language.

Original commit: elastic/x-pack-elasticsearch@c1d05b7357
2017-04-27 11:35:42 -04:00
Sophie Chang aa7d94ec44 [DOCS] Review getting started (elastic/x-pack-elasticsearch#1219)
* [DOCS] Initial review of getting started

* [DOCS] Completed review of getting started

Original commit: elastic/x-pack-elasticsearch@a4b800b59b
2017-04-27 08:14:17 -07:00
Yannick Welsch 779e8f6771 [TEST] Reenable IndicesStatsTests
Commit elastic/x-pack-elasticsearch@4165beb90 should have fixed the test failures.

Original commit: elastic/x-pack-elasticsearch@67b8aac14c
2017-04-27 17:04:45 +02:00
Jay Modi 1d08b4d1fb Rest endpoints for token based access (elastic/x-pack-elasticsearch#1235)
This commit adds rest endpoints for the creation of a new token and invalidation of an existing
token. This builds upon the functionality that was introduced in elastic/x-pack-elasticsearch#1029.

relates elastic/x-pack-elasticsearch#8

Original commit: elastic/x-pack-elasticsearch@d56611dfa3
2017-04-27 11:04:31 -04:00
Jay Modi f7fb02f21f Ensure we always respect a user specified filter in the AD realm (elastic/x-pack-elasticsearch#1161)
When the active directory realm was refactored to add support for authenticating against multiple
domains, only the default authenticator respected the user_search.filter setting. This commit moves
this down to the base authenticator and also changes the UPN filter to not include sAMAccountName
in the filter.

Original commit: elastic/x-pack-elasticsearch@d2c19c9bee
2017-04-27 10:20:59 -04:00
Dimitrios Athanasiou 4f12f04c65 [TEST] Fix line width checkstyle error
Original commit: elastic/x-pack-elasticsearch@0d62752c0e
2017-04-27 14:03:07 +01:00
Dimitrios Athanasiou 4c9b4132c9 [TEST] Refactor ML integration test framework
- Removes need to handle exception from action methods
- Clearly renames DatafeedJobIT to DatafeedJobsRestIT to distinguish
  from DatafeedJobsIT
- Refactors DatafeedJobsIT to reuse MlNativeAutodetectIntegTestCase

Original commit: elastic/x-pack-elasticsearch@5bd0c01391
2017-04-27 13:43:28 +01:00
Tim Vernum ddf5fd68c2 Add ClusterSearchShardsAction to "read_cross_cluster" privilege (elastic/x-pack-elasticsearch#1231)
Cross cluster search uses ClusterSearchShardsAction under the covers.
Without this change, you would need both "read_cross_cluster" and "view_index_metadata" privilegs in order to have permission to execute searches from a remote cluster.

Original commit: elastic/x-pack-elasticsearch@65a6aff329
2017-04-27 22:39:13 +10:00
Dimitrios Athanasiou 7e62eb6255 [TEST] Wait for job to close from DatafeedJobsIT.testLookbackOnly
relates elastic/x-pack-elasticsearch#1228

Original commit: elastic/x-pack-elasticsearch@77055bd128
2017-04-27 12:57:50 +01:00
Clinton Gormley 16177d46d3 Updated cross-cluster search docs to mention lack of support of wildcards
Also updated the required permissions.

Original commit: elastic/x-pack-elasticsearch@cd10ef2f81
2017-04-27 13:32:40 +02:00
Yannick Welsch c6b8542758 [TEST] Adapt wait_for_nodes condition for ClusterFormationTasks with tribes
Companion commit for elastic/elasticsearch#24351

Original commit: elastic/x-pack-elasticsearch@0f57eecbdb
2017-04-27 09:57:54 +02:00
Yannick Welsch 983fb95a2a [TEST] Disable GraphTests.testSignificanceQueryCrawl
as it sporadically fails

Relates to elastic/x-pack-elasticsearch#918

Original commit: elastic/x-pack-elasticsearch@e2a8a79e62
2017-04-27 09:34:56 +02:00
Lisa Cawley d709a64638 [DOCS] Change allows to enables in ML docs (elastic/x-pack-elasticsearch#1230)
Original commit: elastic/x-pack-elasticsearch@a408fd59f6
2017-04-26 15:29:55 -07:00
Jay Modi 69837cd817 Fix DLS query merging when there are groups with/without DLS (elastic/x-pack-elasticsearch#1203)
If a single permission set does not have a query defined then this should be considered as the user
not having document level security for the indices matching that pattern. However, the lack of
document level security was not being taken into account and document level security was being
applied when it should not have been.

Original commit: elastic/x-pack-elasticsearch@f5777c2f37
2017-04-26 16:06:15 -04:00
Luca Cavanna 2a1fd02dcf [TEST] Cross Cluster Search: specify index name when creating role (elastic/x-pack-elasticsearch#1208)
This makes sure that the user needs to have access only to the specific index that needs to be accessed on the remote clusters.

relates to elastic/x-pack-elasticsearch#1153

Original commit: elastic/x-pack-elasticsearch@7966416bd7
2017-04-26 21:46:31 +02:00
jaymode a7d9e92f25 Test: wait for all token expiration jobs to finish
This adds a best effort wait for the token expiration jobs to finish to hopefully address the
issues in tests with on going requests and locked shards.

Relates elastic/x-pack-elasticsearch#1220

Original commit: elastic/x-pack-elasticsearch@ae789c1e58
2017-04-26 15:40:56 -04:00
Lisa Cawley 577fa559f6 [DOCS] More edits in datafeed resource (elastic/x-pack-elasticsearch#1221)
Original commit: elastic/x-pack-elasticsearch@ea6abc163f
2017-04-26 11:22:15 -07:00
Dimitrios Athanasiou b54de1bb91 [TEST] Give time for results to be searchable in UpdateInterimResultsIT
Original commit: elastic/x-pack-elasticsearch@7c28860dd5
2017-04-26 18:46:33 +01:00
Martijn van Groningen 295a4049a3 [ML] Use allocation id as key in `AutodetectProcessManager#autoDetectCommunicatorByJob` map instead of job id.
Relates to elastic/x-pack-elasticsearch#921

Original commit: elastic/x-pack-elasticsearch@21383fd51c
2017-04-26 19:33:41 +02:00
Dimitris Athanasiou 1f57a53b3e [ML] Correct error message for invalid job_id (elastic/x-pack-elasticsearch#1218)
relates elastic/x-pack-elasticsearch#1211

Original commit: elastic/x-pack-elasticsearch@5389a43706
2017-04-26 18:17:06 +01:00
Yannick Welsch 444cbfc283 [TEST] Fix monitoring tests that are flaky in the presence of replicas
Many tests in monitoring use the pattern of calling first awaitMonitoringDocsCount, and then doing a search that checks certain properties, assuming
that the doc count is correct at that point. In the presence of replicas, awaitMonitoringDocsCount might not wait for all shard copies to have the
desired property. A subsequent search might then hit a shard where the property does not hold. As these tests randomize the number of replicas
(through the random_index_template), it easier to constrain awaitMonitoringDocsCount just to the primary and then do subsequent checks just by
querying the primary.

Original commit: elastic/x-pack-elasticsearch@4165beb903
2017-04-26 19:12:45 +02:00
Dimitris Athanasiou 8af91aed6a [DOCS] Add missing info in datafeed resource (elastic/x-pack-elasticsearch#1215)
Original commit: elastic/x-pack-elasticsearch@c415bc92c2
2017-04-26 10:05:27 -07:00
jaymode 4a36cd77ee Test: bind regular socket instead of server socket so connection cannot be accepted
This commit creates a single server socket that will be connected to by local sockets. The local
sockets will use the port of the previously stopped ldap server as the local port. This will
prevent the ldap library from establishing a connection. The previous use of server sockets for
this did not work on all operating systems as the backlog parameter has platform specific meaning.

Original commit: elastic/x-pack-elasticsearch@03b6bf39d4
2017-04-26 12:41:12 -04:00
David Roberts f3f9cb6d74 [ML] Stop using the management thread pool unnecessarily for ML actions (elastic/x-pack-elasticsearch#1213)
The management thread pool only has 5 threads and clogging it up makes
monitoring think the cluster is dead.

relates elastic/x-pack-elasticsearch#1210

Original commit: elastic/x-pack-elasticsearch@f4ad7578d9
2017-04-26 17:17:26 +01:00
jaymode 6b4db0fc36 Test: remove usage of forbidden api (socket connect)
Original commit: elastic/x-pack-elasticsearch@5903973762
2017-04-26 11:39:27 -04:00
jaymode 123a660720 Test: add awaits fix for SessionFactoryLoadBalancingTests
Original commit: elastic/x-pack-elasticsearch@6412d12283
2017-04-26 11:25:30 -04:00
jaymode c9d039525c Test: add more debug output for SessionFactoryLoadBalancingTests failures
Original commit: elastic/x-pack-elasticsearch@206048b94d
2017-04-26 11:15:07 -04:00
jaymode c1c66f38ea Test: specify a timeout for background operation
This commit adds a timeout for the expiration of invalidated tokens so that we can expect that the
requests will have been finished before we do the assertions on the internal test cluster.

Original commit: elastic/x-pack-elasticsearch@2928706224
2017-04-26 10:34:17 -04:00
Dimitrios Athanasiou 708190f356 [TEST] Add model snapshot restore integration test
Relates elastic/x-pack-elasticsearch#882

Original commit: elastic/x-pack-elasticsearch@fbb983e63b
2017-04-26 14:29:55 +01:00
Jay Modi 295051ee8c Add a short-lived token based access mechanism (elastic/x-pack-elasticsearch#1029)
This commit adds a token based access mechanism that is a subset of the OAuth 2.0 protocol. The
token mechanism takes the same values as a OAuth 2 standard (defined in RFC 6749 and RFC 6750),
but differs in that we use XContent for the body instead of form encoded values. Additionally, this
PR provides a mechanism for expiration of a token; this can be used to implement logout
functionality that prevents the token from being used again.

The actual tokens are encrypted using AES-GCM, which also provides authentication. The key for
encryption is derived from a salt value and a passphrase that is stored on each node in the
secure settings store. By default, the tokens have an expiration time of 20 minutes and is
configurable up to a maximum of one hour.

Relates elastic/x-pack-elasticsearch#8

Original commit: elastic/x-pack-elasticsearch@3d201ac2bf
2017-04-26 08:00:03 -04:00
Martijn van Groningen c6c63c471c The xpack side of elastic/x-pack-elasticsearch#24115
Original commit: elastic/x-pack-elasticsearch@862180a169
2017-04-26 11:28:01 +02:00
Lisa Cawley aa421af2fc [DOCS] Stop and start data feeds in ML Getting Started (elastic/x-pack-elasticsearch#1206)
Original commit: elastic/x-pack-elasticsearch@b938c19695
2017-04-25 17:28:10 -07:00
Lisa Cawley 268f5a95af [DOCS] Add authorization section to ML API pages (elastic/x-pack-elasticsearch#1205)
* [DOCS] Edits for description of ML results resource

* [DOCS] Adds Authorization section to ML API pages

Original commit: elastic/x-pack-elasticsearch@797761e7ae
2017-04-25 14:21:42 -07:00
Dimitrios Athanasiou 9828161cff [TEST] Remove broken test from jobs_get_result_categories.yaml
The YAML framework randomly sends the body wrapped in a `source`
field if the API supports GET. Our API doesn't handle `source`.
Taking the test out.

relates elastic/x-pack-elasticsearch#1200

Original commit: elastic/x-pack-elasticsearch@fa77c071b2
2017-04-25 17:07:30 +01:00
Martijn van Groningen ee650b3189 [ml] Use allocation id as key in `runningDatafeeds` map instead of datafeed id
Original commit: elastic/x-pack-elasticsearch@156e3275b1
2017-04-25 17:55:27 +02:00
Dimitris Athanasiou 32128894a5 [ML] Delete interim results after job re-opening (elastic/x-pack-elasticsearch#1194)
This is an issue where a bucket can have both interim results and
non-interim results, a bucket should never have both at the same time.
The steps to cause this situation are:

1. Flush a running job and create interim results
2. Close that job (this does not delete interim results)
3. Re-open the job and POST data
4. The job will eventually emit a bucket result which mingles with the
existing interim results

Normally interim results are deleted by AutoDetectResultProcessor when a
bucket is parsed following a flush command. Because of the close and
re-opening of the job AutoDetectResultProcessor no longer knows that a
previous flush command creating interim results.

The fix is to always delete interim results the first time
AutoDetectResultProcessor sees a bucket.

relates elastic/x-pack-elasticsearch#1188

Original commit: elastic/x-pack-elasticsearch@5326455f54
2017-04-25 16:32:58 +01:00
Lisa Cawley 019b1f7ece [DOCS] Fix doc build errors for elastic/x-pack-elasticsearch#1197 (elastic/x-pack-elasticsearch#1199)
Original commit: elastic/x-pack-elasticsearch@30f69513ab
2017-04-25 08:23:51 -07:00
Sophie Chang 4b39d858b7 [DOCS] Major re-work of resultsresource (elastic/x-pack-elasticsearch#1197)
Original commit: elastic/x-pack-elasticsearch@8e9a004dd2
2017-04-25 08:23:30 -07:00
jaymode ef571568f4 Test: ensure ports that should fail are not bound to by other tests
In the SessionFactoryLoadBalancingTests, we sometime want a connection to a certain IP and Port to
fail as a way to mock an unresponsive/disconnected LDAP server. The test does this by starting up
multiple LDAP servers and then shutting some down. When the server is shut down the port that it
was bound to is open for another process or test to bind to, which can lead to sporadic failures in
CI. This change is a best effort attempt to prevent this by binding a server socket to the port and
filling its backlog so other connections should fail.

Relates elastic/x-pack-elasticsearch#1195

Original commit: elastic/x-pack-elasticsearch@b31a560c93
2017-04-25 11:17:05 -04:00
David Kyle 2d9fd0ed16 [ML] Only audit job creation after it was successful (elastic/x-pack-elasticsearch#1198)
Original commit: elastic/x-pack-elasticsearch@37af6cbf69
2017-04-25 16:07:47 +01:00
Dimitrios Athanasiou 612dcda8ae [TEST] Increase base job bucket_span to 1h
The DatafeedJobsIT.testRealtime test fails from time to time.
The test seems to take a long time to execute the flush action
after the lookback. This could make sense as the test produces
a few records over the span of a week with 5 minutes bucket_span.
Thus, flush will end up doing a lot of word to create results
for so many buckets.

This change increases the bucket_span to 1 hour. Hopefully, this
will stop the failures.

Relates elastic/x-pack-elasticsearch#1162

Original commit: elastic/x-pack-elasticsearch@4366907371
2017-04-25 15:32:44 +01:00
Dimitris Athanasiou 7f64f37c46 [ML] Fix POST version of get categories API (elastic/x-pack-elasticsearch#1184)
Also improves PageParams parsing to fill in defaults.

relates elastic/x-pack-elasticsearch#1180

Original commit: elastic/x-pack-elasticsearch@fccd7795ca
2017-04-25 14:44:10 +01:00
Martijn van Groningen 415d40e6fc test: improved assertion
Original commit: elastic/x-pack-elasticsearch@8361d2e27d
2017-04-25 12:16:01 +02:00
Martijn van Groningen a3e7c65ba4 [ML] Upon task cancel stop datafeed immediately.
Original commit: elastic/x-pack-elasticsearch@0401ca3d33
2017-04-25 11:56:44 +02:00