Commit Graph

5618 Commits

Author SHA1 Message Date
Lisa Cawley 435338e6b9 [DOCS] Fix ML API example for elastic/x-pack-elasticsearch#1180 (elastic/x-pack-elasticsearch#1252)
Original commit: elastic/x-pack-elasticsearch@5290dd0a89
2017-04-28 09:14:53 -07:00
Nik Everett 459b77478e Test: get more information when sporadic NPE
LicenseManagerServiceTests sometimes fails in jenkins, but fairly
rarely. We don't get useful logs when it does. This cranks up
the log level and adds some more assertions so we can better track
down where the failure comes from.

Relates to elastic/x-pack-elasticsearch#222

Original commit: elastic/x-pack-elasticsearch@3e08725fc7
2017-04-28 11:34:48 -04:00
Lisa Cawley 68c3a94c35 [DOCS] Add ML limitations (elastic/x-pack-elasticsearch#1229)
* [DOCS] Add ML limitations

* [DOCS] Address feedback about ML limitations

* [DOCS] Change ML limitations capitalization

Original commit: elastic/x-pack-elasticsearch@41682d8d93
2017-04-28 08:13:47 -07:00
David Roberts 892d803a6a [ML] When putting a datafeed use runas user for index privilege check (elastic/x-pack-elasticsearch#1245)
When a user creates a datafeed, as well as checking they have permission
to create a datafeed we also check that they have permission to search the
indices they've configured the datafeed to search.

Previously this second check was erroneously done for the user who issued
the put_datafeed request, whereas it should be done as the runas user for
that request.

Original commit: elastic/x-pack-elasticsearch@4c35204c66
2017-04-28 13:38:53 +01:00
Clinton Gormley e61224778f Fixed bad link
Original commit: elastic/x-pack-elasticsearch@f7ab965eba
2017-04-28 12:35:07 +02:00
Dimitrios Athanasiou 57382390ab [TEST] Delete job from DatafeedJobsIT.testRealtime in clean up
relates elastic/x-pack-elasticsearch#1246

Original commit: elastic/x-pack-elasticsearch@9ca07f1308
2017-04-28 11:00:51 +01:00
Yannick Welsch c551bcba5c Separate publishing from applying cluster states
Companion commit to elastic/elasticsearch#24236

Original commit: elastic/x-pack-elasticsearch@d685478f5d
2017-04-28 09:35:20 +02:00
Lisa Cawley dd3578aaf0 [DOCS] Refresh ML screenshots (elastic/x-pack-elasticsearch#1242)
Original commit: elastic/x-pack-elasticsearch@e2341e49ea
2017-04-27 17:15:57 -07:00
Zachary Tong 6147e2ba6a Remove now-unnecessary cast
Original commit: elastic/x-pack-elasticsearch@fcbbe5a919
2017-04-27 17:16:30 -04:00
Dimitrios Athanasiou f341b336a5 [ML] Correctly pass job_id from job task to logger
Original commit: elastic/x-pack-elasticsearch@1c73bccaec
2017-04-27 20:02:15 +01:00
Lisa Cawley 485be502f4 [DOCS] Doc build fixes and edits for elastic/x-pack-elasticsearch#1237 (elastic/x-pack-elasticsearch#1241)
* [DOCS] Fixing doc build error

* [DOCS] Edits on ML content for elastic/x-pack-elasticsearch#1237

Original commit: elastic/x-pack-elasticsearch@cd4d404dee
2017-04-27 11:22:59 -07:00
Sophie Chang ffb3bb6493 [DOCS] Overall review (elastic/x-pack-elasticsearch#1237)
* [DOCS] Overall review

* [DOCS] General review

* [DOCS] typo

* [DOCS] Fix for processed_record_count with aggs

* [DOCS] Added latency tbd

Original commit: elastic/x-pack-elasticsearch@9e8cf664c1
2017-04-27 11:22:42 -07:00
Lisa Cawley 642b1f7c19 [DOCS] More edits for ML Getting Started (elastic/x-pack-elasticsearch#1238)
Original commit: elastic/x-pack-elasticsearch@69be11bfd2
2017-04-27 09:15:50 -07:00
Dimitrios Athanasiou b796388431 [ML] Delete intervening results after now as well
When we revert to snapshot, if we delete intervening results
we should delete with an open end on the time range for the
case when future data has been posted to the job.

Original commit: elastic/x-pack-elasticsearch@c3f5e8f19e
2017-04-27 16:38:23 +01:00
Nik Everett 677ea404f7 Remove most usages of 1-arg Script ctor (elastic/x-pack-elasticsearch#1207)
The one argument ctor for `Script` creates a script with the
default language but most usages of are for testing and either
don't care about the language or are for use with
`MockScriptEngine`. This replaces most usages of the one argument
ctor on `Script` with calls to `ESTestCase#mockScript` to make
it clear that the tests don't need the default scripting language.

Original commit: elastic/x-pack-elasticsearch@c1d05b7357
2017-04-27 11:35:42 -04:00
Sophie Chang aa7d94ec44 [DOCS] Review getting started (elastic/x-pack-elasticsearch#1219)
* [DOCS] Initial review of getting started

* [DOCS] Completed review of getting started

Original commit: elastic/x-pack-elasticsearch@a4b800b59b
2017-04-27 08:14:17 -07:00
Yannick Welsch 779e8f6771 [TEST] Reenable IndicesStatsTests
Commit elastic/x-pack-elasticsearch@4165beb90 should have fixed the test failures.

Original commit: elastic/x-pack-elasticsearch@67b8aac14c
2017-04-27 17:04:45 +02:00
Jay Modi 1d08b4d1fb Rest endpoints for token based access (elastic/x-pack-elasticsearch#1235)
This commit adds rest endpoints for the creation of a new token and invalidation of an existing
token. This builds upon the functionality that was introduced in elastic/x-pack-elasticsearch#1029.

relates elastic/x-pack-elasticsearch#8

Original commit: elastic/x-pack-elasticsearch@d56611dfa3
2017-04-27 11:04:31 -04:00
Jay Modi f7fb02f21f Ensure we always respect a user specified filter in the AD realm (elastic/x-pack-elasticsearch#1161)
When the active directory realm was refactored to add support for authenticating against multiple
domains, only the default authenticator respected the user_search.filter setting. This commit moves
this down to the base authenticator and also changes the UPN filter to not include sAMAccountName
in the filter.

Original commit: elastic/x-pack-elasticsearch@d2c19c9bee
2017-04-27 10:20:59 -04:00
Dimitrios Athanasiou 4f12f04c65 [TEST] Fix line width checkstyle error
Original commit: elastic/x-pack-elasticsearch@0d62752c0e
2017-04-27 14:03:07 +01:00
Dimitrios Athanasiou 4c9b4132c9 [TEST] Refactor ML integration test framework
- Removes need to handle exception from action methods
- Clearly renames DatafeedJobIT to DatafeedJobsRestIT to distinguish
  from DatafeedJobsIT
- Refactors DatafeedJobsIT to reuse MlNativeAutodetectIntegTestCase

Original commit: elastic/x-pack-elasticsearch@5bd0c01391
2017-04-27 13:43:28 +01:00
Tim Vernum ddf5fd68c2 Add ClusterSearchShardsAction to "read_cross_cluster" privilege (elastic/x-pack-elasticsearch#1231)
Cross cluster search uses ClusterSearchShardsAction under the covers.
Without this change, you would need both "read_cross_cluster" and "view_index_metadata" privilegs in order to have permission to execute searches from a remote cluster.

Original commit: elastic/x-pack-elasticsearch@65a6aff329
2017-04-27 22:39:13 +10:00
Dimitrios Athanasiou 7e62eb6255 [TEST] Wait for job to close from DatafeedJobsIT.testLookbackOnly
relates elastic/x-pack-elasticsearch#1228

Original commit: elastic/x-pack-elasticsearch@77055bd128
2017-04-27 12:57:50 +01:00
Clinton Gormley 16177d46d3 Updated cross-cluster search docs to mention lack of support of wildcards
Also updated the required permissions.

Original commit: elastic/x-pack-elasticsearch@cd10ef2f81
2017-04-27 13:32:40 +02:00
Yannick Welsch c6b8542758 [TEST] Adapt wait_for_nodes condition for ClusterFormationTasks with tribes
Companion commit for elastic/elasticsearch#24351

Original commit: elastic/x-pack-elasticsearch@0f57eecbdb
2017-04-27 09:57:54 +02:00
Yannick Welsch 983fb95a2a [TEST] Disable GraphTests.testSignificanceQueryCrawl
as it sporadically fails

Relates to elastic/x-pack-elasticsearch#918

Original commit: elastic/x-pack-elasticsearch@e2a8a79e62
2017-04-27 09:34:56 +02:00
Lisa Cawley d709a64638 [DOCS] Change allows to enables in ML docs (elastic/x-pack-elasticsearch#1230)
Original commit: elastic/x-pack-elasticsearch@a408fd59f6
2017-04-26 15:29:55 -07:00
Jay Modi 69837cd817 Fix DLS query merging when there are groups with/without DLS (elastic/x-pack-elasticsearch#1203)
If a single permission set does not have a query defined then this should be considered as the user
not having document level security for the indices matching that pattern. However, the lack of
document level security was not being taken into account and document level security was being
applied when it should not have been.

Original commit: elastic/x-pack-elasticsearch@f5777c2f37
2017-04-26 16:06:15 -04:00
Luca Cavanna 2a1fd02dcf [TEST] Cross Cluster Search: specify index name when creating role (elastic/x-pack-elasticsearch#1208)
This makes sure that the user needs to have access only to the specific index that needs to be accessed on the remote clusters.

relates to elastic/x-pack-elasticsearch#1153

Original commit: elastic/x-pack-elasticsearch@7966416bd7
2017-04-26 21:46:31 +02:00
jaymode a7d9e92f25 Test: wait for all token expiration jobs to finish
This adds a best effort wait for the token expiration jobs to finish to hopefully address the
issues in tests with on going requests and locked shards.

Relates elastic/x-pack-elasticsearch#1220

Original commit: elastic/x-pack-elasticsearch@ae789c1e58
2017-04-26 15:40:56 -04:00
Lisa Cawley 577fa559f6 [DOCS] More edits in datafeed resource (elastic/x-pack-elasticsearch#1221)
Original commit: elastic/x-pack-elasticsearch@ea6abc163f
2017-04-26 11:22:15 -07:00
Dimitrios Athanasiou b54de1bb91 [TEST] Give time for results to be searchable in UpdateInterimResultsIT
Original commit: elastic/x-pack-elasticsearch@7c28860dd5
2017-04-26 18:46:33 +01:00
Martijn van Groningen 295a4049a3 [ML] Use allocation id as key in `AutodetectProcessManager#autoDetectCommunicatorByJob` map instead of job id.
Relates to elastic/x-pack-elasticsearch#921

Original commit: elastic/x-pack-elasticsearch@21383fd51c
2017-04-26 19:33:41 +02:00
Dimitris Athanasiou 1f57a53b3e [ML] Correct error message for invalid job_id (elastic/x-pack-elasticsearch#1218)
relates elastic/x-pack-elasticsearch#1211

Original commit: elastic/x-pack-elasticsearch@5389a43706
2017-04-26 18:17:06 +01:00
Yannick Welsch 444cbfc283 [TEST] Fix monitoring tests that are flaky in the presence of replicas
Many tests in monitoring use the pattern of calling first awaitMonitoringDocsCount, and then doing a search that checks certain properties, assuming
that the doc count is correct at that point. In the presence of replicas, awaitMonitoringDocsCount might not wait for all shard copies to have the
desired property. A subsequent search might then hit a shard where the property does not hold. As these tests randomize the number of replicas
(through the random_index_template), it easier to constrain awaitMonitoringDocsCount just to the primary and then do subsequent checks just by
querying the primary.

Original commit: elastic/x-pack-elasticsearch@4165beb903
2017-04-26 19:12:45 +02:00
Dimitris Athanasiou 8af91aed6a [DOCS] Add missing info in datafeed resource (elastic/x-pack-elasticsearch#1215)
Original commit: elastic/x-pack-elasticsearch@c415bc92c2
2017-04-26 10:05:27 -07:00
jaymode 4a36cd77ee Test: bind regular socket instead of server socket so connection cannot be accepted
This commit creates a single server socket that will be connected to by local sockets. The local
sockets will use the port of the previously stopped ldap server as the local port. This will
prevent the ldap library from establishing a connection. The previous use of server sockets for
this did not work on all operating systems as the backlog parameter has platform specific meaning.

Original commit: elastic/x-pack-elasticsearch@03b6bf39d4
2017-04-26 12:41:12 -04:00
David Roberts f3f9cb6d74 [ML] Stop using the management thread pool unnecessarily for ML actions (elastic/x-pack-elasticsearch#1213)
The management thread pool only has 5 threads and clogging it up makes
monitoring think the cluster is dead.

relates elastic/x-pack-elasticsearch#1210

Original commit: elastic/x-pack-elasticsearch@f4ad7578d9
2017-04-26 17:17:26 +01:00
jaymode 6b4db0fc36 Test: remove usage of forbidden api (socket connect)
Original commit: elastic/x-pack-elasticsearch@5903973762
2017-04-26 11:39:27 -04:00
jaymode 123a660720 Test: add awaits fix for SessionFactoryLoadBalancingTests
Original commit: elastic/x-pack-elasticsearch@6412d12283
2017-04-26 11:25:30 -04:00
jaymode c9d039525c Test: add more debug output for SessionFactoryLoadBalancingTests failures
Original commit: elastic/x-pack-elasticsearch@206048b94d
2017-04-26 11:15:07 -04:00
jaymode c1c66f38ea Test: specify a timeout for background operation
This commit adds a timeout for the expiration of invalidated tokens so that we can expect that the
requests will have been finished before we do the assertions on the internal test cluster.

Original commit: elastic/x-pack-elasticsearch@2928706224
2017-04-26 10:34:17 -04:00
Dimitrios Athanasiou 708190f356 [TEST] Add model snapshot restore integration test
Relates elastic/x-pack-elasticsearch#882

Original commit: elastic/x-pack-elasticsearch@fbb983e63b
2017-04-26 14:29:55 +01:00
Jay Modi 295051ee8c Add a short-lived token based access mechanism (elastic/x-pack-elasticsearch#1029)
This commit adds a token based access mechanism that is a subset of the OAuth 2.0 protocol. The
token mechanism takes the same values as a OAuth 2 standard (defined in RFC 6749 and RFC 6750),
but differs in that we use XContent for the body instead of form encoded values. Additionally, this
PR provides a mechanism for expiration of a token; this can be used to implement logout
functionality that prevents the token from being used again.

The actual tokens are encrypted using AES-GCM, which also provides authentication. The key for
encryption is derived from a salt value and a passphrase that is stored on each node in the
secure settings store. By default, the tokens have an expiration time of 20 minutes and is
configurable up to a maximum of one hour.

Relates elastic/x-pack-elasticsearch#8

Original commit: elastic/x-pack-elasticsearch@3d201ac2bf
2017-04-26 08:00:03 -04:00
Martijn van Groningen c6c63c471c The xpack side of elastic/x-pack-elasticsearch#24115
Original commit: elastic/x-pack-elasticsearch@862180a169
2017-04-26 11:28:01 +02:00
Lisa Cawley aa421af2fc [DOCS] Stop and start data feeds in ML Getting Started (elastic/x-pack-elasticsearch#1206)
Original commit: elastic/x-pack-elasticsearch@b938c19695
2017-04-25 17:28:10 -07:00
Lisa Cawley 268f5a95af [DOCS] Add authorization section to ML API pages (elastic/x-pack-elasticsearch#1205)
* [DOCS] Edits for description of ML results resource

* [DOCS] Adds Authorization section to ML API pages

Original commit: elastic/x-pack-elasticsearch@797761e7ae
2017-04-25 14:21:42 -07:00
Dimitrios Athanasiou 9828161cff [TEST] Remove broken test from jobs_get_result_categories.yaml
The YAML framework randomly sends the body wrapped in a `source`
field if the API supports GET. Our API doesn't handle `source`.
Taking the test out.

relates elastic/x-pack-elasticsearch#1200

Original commit: elastic/x-pack-elasticsearch@fa77c071b2
2017-04-25 17:07:30 +01:00
Martijn van Groningen ee650b3189 [ml] Use allocation id as key in `runningDatafeeds` map instead of datafeed id
Original commit: elastic/x-pack-elasticsearch@156e3275b1
2017-04-25 17:55:27 +02:00
Dimitris Athanasiou 32128894a5 [ML] Delete interim results after job re-opening (elastic/x-pack-elasticsearch#1194)
This is an issue where a bucket can have both interim results and
non-interim results, a bucket should never have both at the same time.
The steps to cause this situation are:

1. Flush a running job and create interim results
2. Close that job (this does not delete interim results)
3. Re-open the job and POST data
4. The job will eventually emit a bucket result which mingles with the
existing interim results

Normally interim results are deleted by AutoDetectResultProcessor when a
bucket is parsed following a flush command. Because of the close and
re-opening of the job AutoDetectResultProcessor no longer knows that a
previous flush command creating interim results.

The fix is to always delete interim results the first time
AutoDetectResultProcessor sees a bucket.

relates elastic/x-pack-elasticsearch#1188

Original commit: elastic/x-pack-elasticsearch@5326455f54
2017-04-25 16:32:58 +01:00