Commit Graph

5616 Commits

Author SHA1 Message Date
Simon Willnauer 5a4b7ee98b [TEST] Add REST test to search via unfiltered alias in CrossClusterSearch
Original commit: elastic/x-pack-elasticsearch@4e1b99824c
2017-05-05 10:25:06 +02:00
Tim Vernum 384486d6dd [TESTS] Really fix MultipleAdRealmTests (I think)
Original commit: elastic/x-pack-elasticsearch@dc1eb4eb93
2017-05-05 10:45:11 +10:00
Igor Motov 9a800cac60 Switch to StreamInput.readEnum and StreamOutput.writeEnum (elastic/x-pack-elasticsearch#1313)
Start using StreamInput.readEnum and StreamOutput.writeEnum for enum serialization.

Relates to elastic/elasticsearch#24475

Original commit: elastic/x-pack-elasticsearch@67ca571458
2017-05-04 18:03:05 -04:00
Dimitris Athanasiou 9e3d5d6cf3 [ML] Provide better error message when doc values are unavailable (elastic/x-pack-elasticsearch#1312)
relates elastic/x-pack-elasticsearch#1305

Original commit: elastic/x-pack-elasticsearch@2e8ccc340b
2017-05-04 17:12:49 +01:00
Dimitris Athanasiou 4241469c89 [ML] Correctly print job state when delete job fails (elastic/x-pack-elasticsearch#1310)
relates elastic/x-pack-elasticsearch#1307

Original commit: elastic/x-pack-elasticsearch@25fb18ed0b
2017-05-04 15:17:48 +01:00
Simon Willnauer eddc7a2147 Add tests for cross cluster `_field_caps` API (elastic/x-pack-elasticsearch#1294)
This commit adds REST tests to ensure `_field_caps` works with cross cluster
search and security.

Relates to elastic/elasticsearch#24463

Original commit: elastic/x-pack-elasticsearch@2af0a8c179
2017-05-04 15:04:39 +02:00
Adrien Grand ea31227a89 Do not create the monitoring index before the exporter service is started.
This prevents the template from being applied.

relates elastic/x-pack-elasticsearch#1308

Original commit: elastic/x-pack-elasticsearch@c9fcea42ce
2017-05-04 14:40:32 +02:00
Dimitris Athanasiou 2290b93aa3 [ML] Ignore interim results during renormalization (elastic/x-pack-elasticsearch#1298)
relates elastic/x-pack-elasticsearch#1193

Original commit: elastic/x-pack-elasticsearch@6f5e354bdd
2017-05-04 11:38:52 +01:00
Dimitris Athanasiou 7f2c7dbe17 [ML] Rename datafeed indexes to indices (elastic/x-pack-elasticsearch#1297)
Renames datafeed indexes to indices but keeps indexes
as a synonym while parsing.

relates elastic/x-pack-elasticsearch#1292

Original commit: elastic/x-pack-elasticsearch@1fcdd97f88
2017-05-04 11:37:12 +01:00
Tim Vernum b922b37c3f [TEST] Fixes for randomised failures
- role_mapping.native is always present, but contains no entries if the security index is unavailable
- file based role mapping does not allow duplicate keys

Original commit: elastic/x-pack-elasticsearch@734cf7e2c0
2017-05-04 20:18:28 +10:00
Adrien Grand bbb3d008f2 Only allow indices to have a single type by default. (elastic/x-pack-elasticsearch#1212)
Sibling of elastic/elasticsearch#24317

Original commit: elastic/x-pack-elasticsearch@a57decdf9f
2017-05-04 10:23:23 +02:00
Alexander Reelsen 4078b2f1b2 Watcher: Replace _status field with status (elastic/x-pack-elasticsearch#1285)
As fields with underscores will be disallowed in master, and we have to
prepare the upgrade, this commit renames the _status field to status.

When the 5.x upgrade logic is in place in the 5.x we can remove all the
old style _status handling from the master branch.

Note: All the BWC compatibility tests, that load 5.x indices are now
faking a finished upgrade by adding the `status` field to the mapping
of the watches index.

Original commit: elastic/x-pack-elasticsearch@9d5cc9aaec
2017-05-04 10:08:34 +02:00
Tim Vernum 8633fd1f07 [TESTS] Fix problem with AD integration test
In some cases (based on the randomisation) the primary-realm would have no mappings, but the secondary-realm would.
If this occurred we wouldn't write out any mappings and the secondary-realm would not authorize the access being tested

Original commit: elastic/x-pack-elasticsearch@8e81ec9dd7
2017-05-04 18:05:26 +10:00
Jim Ferenczi 770404200e [TEST] Fix test that breaks compilation due to an invalid usage of ListenableActionFuture
Original commit: elastic/x-pack-elasticsearch@b92d06c9b1
2017-05-04 08:33:06 +02:00
Tim Vernum 6adf4fd3af Role Mapping API (elastic/x-pack-elasticsearch#925)
This introduces a role-mapping API to X-Pack security.

Features:
- A `GET`/`PUT`/`DELETE` API at `/_xpack/security/role_mapping/`
- Role-mappings are stored in the `.security` index 
- A custom expression language (in JSON) for expressing the mapping rules 
- Supported in LDAP/AD and PKI realms
- LDAP realm also supports loading arbitrary meta-data (which can be used in the mapping rules)
- A CompositeRoleMapper unifies roles from the existing file based mapper, and the new API based mapper.
- Usage stats for native role mappings

Original commit: elastic/x-pack-elasticsearch@d9972ed1da
2017-05-04 13:38:50 +10:00
Lisa Cawley cf27cb479a [DOCS] Clarify wildcard usage in ML datafeed APIs (elastic/x-pack-elasticsearch#1296)
Original commit: elastic/x-pack-elasticsearch@c18fca323b
2017-05-03 09:07:44 -07:00
Dimitris Athanasiou eae7d19d2f [DOCS] Improvements on datafeed related docs (elastic/x-pack-elasticsearch#1293)
Original commit: elastic/x-pack-elasticsearch@4bb1932f0e
2017-05-03 08:38:31 -07:00
Luca Cavanna 500170c456 [TEST] Remove more unnecessary ListenableActionFuture usages (elastic/x-pack-elasticsearch#1283)
Original commit: elastic/x-pack-elasticsearch@9dfe7ea6f9
2017-05-03 16:35:15 +02:00
Tim Brooks 106a26b399 Remove unneeded usages of listenable futures (elastic/x-pack-elasticsearch#1261)
This is related to elastic/elasticsearch#24412. That commit changed how
ListenableActionFuture implementations are created. This commit
updates x-pack to be compatible with those changes. In particular, all
the usages of ListenableActionFuture in x-pack could be replaced with
PlainActionFuture as the "listening" functionality was not being used.

Original commit: elastic/x-pack-elasticsearch@7c8d8e3df9
2017-05-03 09:23:39 -05:00
Dimitrios Athanasiou d140b3028d [ML] Reuse encapsulated delete-by-query request
Original commit: elastic/x-pack-elasticsearch@cc0f2ce815
2017-05-03 14:58:22 +01:00
Dimitris Athanasiou 3f73748d14 [ML] Use delete-by-query in JobDataDeleter (elastic/x-pack-elasticsearch#1274)
JobDataDeleter handles the deletion logic for 3 cases:

1. deleting a model snapshot and its state docs
2. deleting all results after a timestamp
3. deleting all interim results

The last 2 are currently implemented by manually performing
a search and scroll and then adding matching hits in a bulk
delete action. This operation is exactly what delete-by-query
does.

This commit changes JobDataDeleter to use delete-by-query. This
makes the code simpler and less error-prone. The downside is
losing some logging which seems non-critical. Unit tests for
JobDataDeleter are also removed as they are heavily mocked tests,
adding little value and high maintenance cost. This functionality
is tested by integration tests already.

relates elastic/x-pack-elasticsearch#821

Original commit: elastic/x-pack-elasticsearch@7da91332bd
2017-05-03 14:51:41 +01:00
Dimitris Athanasiou 3e9c36838d [ML] Remove get model snapshot by description functionality (elastic/x-pack-elasticsearch#1288)
relates elastic/x-pack-elasticsearch#1284

Original commit: elastic/x-pack-elasticsearch@780feea5c1
2017-05-03 13:20:52 +01:00
Luca Cavanna 796e23a02a Adapt to ActionRequestBuilder#execute changes (elastic/x-pack-elasticsearch#1263)
Original commit: elastic/x-pack-elasticsearch@c2d7aa562c
2017-05-03 11:21:06 +02:00
Hendrik Muhs 9d8fea9a32 [ML] expose low and high median functions (elastic/x-pack-elasticsearch#1272)
Expose low and high median functions implemented downstream in machinelearning-cpp.

Original commit: elastic/x-pack-elasticsearch@3863e9d41c
2017-05-03 09:46:10 +02:00
Daniel Mitterdorfer 240bd5bfa7 Mute EllipticCurveSSLTests
Relates elastic/x-pack-elasticsearch#1278

Original commit: elastic/x-pack-elasticsearch@3e5dbf5fe6
2017-05-03 07:57:27 +02:00
Chee Wee 1e0caee90d Corrected a mistake on disabling a user (elastic/x-pack-elasticsearch#1135)
Corrected a mistake on disabling a user, when it should be enabling.

Please review the changes.

Original commit: elastic/x-pack-elasticsearch@352997e427
2017-05-03 10:55:26 +08:00
Lisa Cawley 33c50f1201 [DOCS] Add property table for ML Update Jobs API (elastic/x-pack-elasticsearch#1268)
* [DOCS] Add property table for ML Update Jobs API

* [DOCS] Updates based on feedback for ML Update Jobs API

* [DOCS] Removed detector properties from ML Update Jobs API

* [DOCS] Fixes typos

Original commit: elastic/x-pack-elasticsearch@68d1b5598c
2017-05-02 15:38:41 -07:00
Lisa Cawley 9b2fb6ac16 [DOCS] Change "data feed" to "datafeed" in Machine Learning documentation (elastic/x-pack-elasticsearch#1277)
* [DOCS] Add xpackml attribute to XPack Reference

* [DOCS] Use attribute for datafeed terms

Original commit: elastic/x-pack-elasticsearch@f37bf48ee4
2017-05-02 12:45:42 -07:00
Alexander Reelsen 3a0bc504a9 Watcher: Exeuting a watch returns a proper 404 (elastic/x-pack-elasticsearch#1273)
A test failure uncovered that handling non existing watches in the
execute watch API could lead to 500 errors instead of 404 because
they were not handled correctly.

relates elastic/x-pack-elasticsearch#1120

Original commit: elastic/x-pack-elasticsearch@c17a07ae3e
2017-05-02 16:01:45 +02:00
Alexander Reelsen ff6283bf80 Tests: Ignore existing watches index in REST test
This test should ensure, that a missing watch on an existing index
causes an error, thus it can ignore if the .watches index already
exists.

relates elastic/x-pack-elasticsearch#1155

Original commit: elastic/x-pack-elasticsearch@89c56cb2be
2017-05-02 14:16:19 +02:00
Alexander Reelsen 50241717d3 Tests: Use execute watch API instead of persisting watch
To speed up the test and ensure no other failures occur,
simply use the execute watch API instead of persisting the watch
and executing then.

relates elastic/x-pack-elasticsearch#1350

Original commit: elastic/x-pack-elasticsearch@2888fb0e93
2017-05-02 14:04:04 +02:00
Tanguy Leroux 9e14141a8e [Test] Fix NPE in LocalExporterTests
This commit fixes a NullPointerException in the LocalExporterTests.

Original commit: elastic/x-pack-elasticsearch@e28516135f
2017-05-02 13:55:47 +02:00
David Kyle f3f387bd9c [ML] Give jobs meaningful and unique names in the yaml rest tests. (elastic/x-pack-elasticsearch#1271)
Original commit: elastic/x-pack-elasticsearch@b096de55c1
2017-05-02 11:36:33 +01:00
Alexander Reelsen c62f6f8177 Watcher: Distributed watch execution (elastic/x-pack-elasticsearch#544)
The distribution of watches now happens on the node which holds the
watches index, instead of on the master node. This requires several
changes to the current implementation.

1. Running on shards and replicas
   In order to run watches on the nodes with the watches index on its
   primaries and replicas. To ensure that watches do not run twice, there is
   a logic which checks the local shards, runs a murmurhash on the id and
   runs modulo against the number of shards and replicas, this is the way to
   find out, if a watch should run local. Reloading happens
2. Several master node actions moved to a HandledTransportAction, as they
   are basically just aliases for indexing actions, among them the
   put/delete/get watch actions, the acknowledgement action, the de/activate
   actions
3. Stats action moved to a broadcast node action, because we potentially
   have to query every node to get watcher statistics
4. Starting/Stopping watcher now is a master node action, which updates
   the cluster state and then listeners acts on those. Because of this watches
   can be running on two systems, if you those have different cluster state
   versions, until the new watcher state is propagated
5. Watcher is started on all nodes now. With the exception of the ticker
   schedule engine most classes do not need a lot of resources while running.
   However they have to run, because of the execute watch API, which can hit
   any node - it does not make sense to find the right shard for this watch
   and only then execute (as this also has to work with a watch, that has not
   been stored before)
6. By using a indexing operation listener, each storing of a watch now
   parses the watch first and only stores on successful parsing
7. Execute watch API now uses the watcher threadpool for execution
8. Getting the number of watches for the stats now simply queries the
   different execution engines, how many watches are scheduled, so this is
   not doing a search anymore

There will be follow up commits on this one, mainly to ensure BWC compatibility.

Original commit: elastic/x-pack-elasticsearch@0adb46e658
2017-05-02 10:12:46 +02:00
debadair dae762913d [DOCS] Added xpack.monitoring.ui.enabled to monitoring settings.
Original commit: elastic/x-pack-elasticsearch@3dd136ed57
2017-05-01 17:18:51 -07:00
Jason Tedor 718518fe85 Disable setting available processors in tests
Within the same JVM, setting the number of processors available to Netty
can only be done once. However, tests randomize the number of processors
and so without intervention would attempt to set this value multiple
times. Therefore, we need to use a flag that prevents setting this value
in tests.

Relates elastic/x-pack-elasticsearch#1266

Original commit: elastic/x-pack-elasticsearch@d127149725
2017-05-01 19:27:45 -04:00
Lisa Cawley a615532866 [DOCS] ML 5.4 docs final tuning (elastic/x-pack-elasticsearch#1265)
Original commit: elastic/x-pack-elasticsearch@91e4af140d
2017-05-01 11:30:19 -07:00
Jason Tedor bec3102e06 Upgrade Netty to 4.1.10.Final
This commit upgrades the Netty dependency from version 4.1.9.Final to
version 4.1.10.Final.

Relates elastic/x-pack-elasticsearch#1262

Original commit: elastic/x-pack-elasticsearch@aac7aa351c
2017-05-01 10:25:42 -04:00
Jason Tedor 5387472834 Adjust checkstyle suppressions to 140-column limit
We are back to having a 140-column limit. While at some point we will
apply auto-formatting tools to the code base, that is a down-the-road
thing and adjusting the suppressions now shaves minutes off the build
time.

Relates elastic/x-pack-elasticsearch#1260

Original commit: elastic/x-pack-elasticsearch@9a96aec9e6
2017-04-30 12:11:37 -04:00
Yannick Welsch a9aa3e2329 Revert "Test: get more information when sporadic NPE"
This reverts commit elastic/x-pack-elasticsearch@3e08725fc7.

Original commit: elastic/x-pack-elasticsearch@dc85c2b194
2017-04-29 10:51:35 +02:00
Yannick Welsch 308d5e0ae3 Don't call ClusterService.state() in a ClusterStateUpdateTask
The current state is readily available as a parameter

Original commit: elastic/x-pack-elasticsearch@a09929aa82
2017-04-29 10:50:56 +02:00
Yannick Welsch e3834da05e [TEST] Wait for trial license to be generated before running LicensesManagerServiceTests
Many of the tests assume that the trial license has already been generated before the test gets to run. As this is asynchronously triggered upon node
startup, however, there is no guarantee that trial license generation has completed before the tests get to execute, leading to null values when
checking clusterService.state().metaData().custom(LicensesMetaData.TYPE).

Original commit: elastic/x-pack-elasticsearch@d909c9ba95
2017-04-29 10:50:56 +02:00
Lisa Cawley 600aa8fc5d [DOCS] Add role=screenshot to format graphics (elastic/x-pack-elasticsearch#1259)
* [DOCS] Added role attribute to ML screenshots

* [DOCS] Fixing screenshot role in ML tutorial

Original commit: elastic/x-pack-elasticsearch@232a0632a0
2017-04-28 21:27:04 -07:00
Lisa Cawley d77f20b14e [DOCS] Update ML tutorial with latest sample data (elastic/x-pack-elasticsearch#1258)
* [DOCS] Updated mappings in ML sample data

* [DOCS] Update ML tutorial with latest sample data

Original commit: elastic/x-pack-elasticsearch@37fd75aed5
2017-04-28 12:31:24 -07:00
Lisa Cawley c4be95ae31 [DOCS] Remove unnecessary ML release notes file (elastic/x-pack-elasticsearch#1257)
Original commit: elastic/x-pack-elasticsearch@e3d2c66090
2017-04-28 11:48:08 -07:00
Lisa Cawley 4669a823cc [DOCS] Added ML sample data URLs (elastic/x-pack-elasticsearch#1256)
Original commit: elastic/x-pack-elasticsearch@528a32f26f
2017-04-28 10:28:37 -07:00
Dimitrios Athanasiou ee5e66bb0d [ML] Make updateDatafeed var in UpdateDatafeedAction volatile
Original commit: elastic/x-pack-elasticsearch@d8461d0960
2017-04-28 17:58:55 +01:00
Dimitris Athanasiou 5a70eac6e8 [ML] Delete and Update datafeed actions should use current cluster state (elastic/x-pack-elasticsearch#1254)
relates elastic/x-pack-elasticsearch#1253

Original commit: elastic/x-pack-elasticsearch@bdf695e694
2017-04-28 17:53:59 +01:00
Lisa Cawley 435338e6b9 [DOCS] Fix ML API example for elastic/x-pack-elasticsearch#1180 (elastic/x-pack-elasticsearch#1252)
Original commit: elastic/x-pack-elasticsearch@5290dd0a89
2017-04-28 09:14:53 -07:00
Nik Everett 459b77478e Test: get more information when sporadic NPE
LicenseManagerServiceTests sometimes fails in jenkins, but fairly
rarely. We don't get useful logs when it does. This cranks up
the log level and adds some more assertions so we can better track
down where the failure comes from.

Relates to elastic/x-pack-elasticsearch#222

Original commit: elastic/x-pack-elasticsearch@3e08725fc7
2017-04-28 11:34:48 -04:00