Commit Graph

5783 Commits

Author SHA1 Message Date
Lisa Cawley 0f3f22deb2 [DOCS] Add ML info_content functions (elastic/x-pack-elasticsearch#1354)
Original commit: elastic/x-pack-elasticsearch@5cfa9618b5
2017-05-09 08:12:59 -07:00
Tanguy Leroux 0a860df9f9 [Test] Restore random documents indexing in LocalExporterTests (elastic/x-pack-elasticsearch#1328)
In elastic/x-pack-elasticsearch#1212, we removed a randomized condition that inserts documents when
the exporter is enabled but before the monitoring is started. This
condition should not have been removed as it allows to test that the
exporters are correctly initialized when the Monitoring Bulk API is used
 by an external application.

This commit also fixed a failure when a search request fails because the
 monitoring indices are not yet ready (see
 https://github.com/elastic/x-pack-elasticsearch/issues/956#issuecomment-298338589)

Original commit: elastic/x-pack-elasticsearch@73ab535ae0
2017-05-09 09:39:25 +02:00
jaymode b450664766 Test: ensure system supports ECDSA before running EllicpticCurveSSLTests
Some JDKs do not support the ECDSA cipher suites that we use in the EllipticCurveSSLTests, which
is the underlying cause of some CI failures. This change ensures there is at least one enabled
ECDSA cipher before testing that a connection can be made.

relates elastic/x-pack-elasticsearch#1278

Original commit: elastic/x-pack-elasticsearch@f6c93d776c
2017-05-08 12:36:20 -04:00
David Kyle e5b11d0222 [ML] Not an error to close a job twice (elastic/x-pack-elasticsearch#1340)
* [ML] Not an error to close a job twice

* Error if job is opening

* Address review comments

* Test closed job isn’t resolved

Original commit: elastic/x-pack-elasticsearch@7da7b24c08
2017-05-08 16:34:46 +01:00
David Roberts 9264ad541e [ML] Only attempt to get the native code version when ML is enabled (elastic/x-pack-elasticsearch#1346)
Originally we used to try to get the native code version even when ML was disabled.
However, this has proved to be an annoyance in cases where people running on
unsupported platforms have disabled ML. This commit means we'll only try to get
the native code version if ML is enabled on a node.

Original commit: elastic/x-pack-elasticsearch@778d6708d2
2017-05-08 16:16:23 +01:00
Lisa Cawley 0c4a7e1930 [DOCS] More minor fixes (elastic/x-pack-elasticsearch#1343)
Original commit: elastic/x-pack-elasticsearch@351e4ea643
2017-05-08 07:12:03 -07:00
Lisa Cawley 0542d730c9 [DOCS] Fix minor ML documentation problems (elastic/x-pack-elasticsearch#1336)
Original commit: elastic/x-pack-elasticsearch@53e65b90fc
2017-05-08 06:53:04 -07:00
Dimitris Athanasiou 28f2ba3ef8 [ML] Retrieve data counts via search (elastic/x-pack-elasticsearch#1339)
Relates elastic/x-pack-elasticsearch#827

Original commit: elastic/x-pack-elasticsearch@73a6848526
2017-05-08 13:02:07 +01:00
David Kyle 4e25e1a24d [ML] Use XContentBuilders in try with resource statements (elastic/x-pack-elasticsearch#1329)
* [ML] Use XContentBuilders in try with resource statements

* Address review comments

Original commit: elastic/x-pack-elasticsearch@ef5b45e2f4
2017-05-08 11:41:23 +01:00
Tim Vernum 0a15b03395 [TESTS] A proper fix to MultipleAdRealmTests
Original commit: elastic/x-pack-elasticsearch@f04b3ebb7a
2017-05-08 00:17:00 +10:00
Chris Earle f18a7f0f68 [Monitoring] Remove unused "node" _type from .monitoring-data-2 (elastic/x-pack-elasticsearch#1333)
This removes the "node" type from `.monitoring-data-2`. This data is sent to _both_ the time-based and non-time-based indexes for Elasticsearch, but the UI only used the time-based variant already.

This is another step in the process of removing the `.monitoring-data-2` index. There is now only one `_type` left in that index: `cluster_info`, which is used by the UI and phone home stats because it contains the license details _and_ the `stack_stats` (e.g., `xpack_usage`).

Original commit: elastic/x-pack-elasticsearch@2cadb5939d
2017-05-05 17:48:30 -04:00
Tanguy Leroux 327ac9898d Fix compilation errors after elastic/elasticsearch/elastic/x-pack-elasticsearch#24492
Original commit: elastic/x-pack-elasticsearch@06d0c17165
2017-05-05 21:27:09 +02:00
Lisa Cawley 30c616a232 [DOCS] More fixes for build.gradle errors (elastic/x-pack-elasticsearch#1334)
Original commit: elastic/x-pack-elasticsearch@578f727494
2017-05-05 11:57:20 -07:00
Lisa Cawley 85cf6a1868 [DOCS] Remove incomplete pages from build.gradle (elastic/x-pack-elasticsearch#1332)
Original commit: elastic/x-pack-elasticsearch@61e236fe70
2017-05-05 11:18:24 -07:00
Lisa Cawley 45cfc17ea1 [DOCS] Add ML analytical functions (elastic/x-pack-elasticsearch#1319)
* [DOCS] Add ML analytical functions

* [DOCS] Add pages for ML analytical functions

* [DOCS] Add links to ML functions from API definitions

Original commit: elastic/x-pack-elasticsearch@ae50b431d3
2017-05-05 10:41:53 -07:00
Dimitris Athanasiou 3570eb32d3 [ML] Retrieve model_size_stats via search (elastic/x-pack-elasticsearch#1326)
This is a task towards allowing rollover.

Multiple model_size_stats are stored in order to allow
analytics of the memory usage over time. The job _stats
need to display the latest model_size_stats. Before this
commit, the latest model_size_stats was being stored with
a special ID and it was retrieved using that ID. This
does not lend itself well for rollover as we would end up
with multiple of those special IDs in the rolled indices.

This commit removes the need to store a special model_size_stats
version. Job _stats now retrieve the model_size_stats by searching
for the latest one. It also uses a manual ID for all model_size_stats
in order to maintain a single document per log_time.

Relates elastic/x-pack-elasticsearch#827

Original commit: elastic/x-pack-elasticsearch@b2796e9b08
2017-05-05 17:05:39 +01:00
Chris Earle 8ab46e800c [Monitoring] Stop Accepting .monitoring-data-N index requests via REST (elastic/x-pack-elasticsearch#1318)
Now that the Monitoring UI no longer checks the `.monitoring-data-2` index
for Kibana, Logstash, or Beats data, we can stop accepting the duplicated
data in that index (the _exact_ same documents are also indexed into the
time-based index for each product).

This ignores rather than rejects requests that contain such documents to
allow older clients to communicate with a 5.5+ monitoring cluster.

Original commit: elastic/x-pack-elasticsearch@def472cf2e
2017-05-05 11:53:43 -04:00
Simon Willnauer 5a4b7ee98b [TEST] Add REST test to search via unfiltered alias in CrossClusterSearch
Original commit: elastic/x-pack-elasticsearch@4e1b99824c
2017-05-05 10:25:06 +02:00
Tim Vernum 384486d6dd [TESTS] Really fix MultipleAdRealmTests (I think)
Original commit: elastic/x-pack-elasticsearch@dc1eb4eb93
2017-05-05 10:45:11 +10:00
Igor Motov 9a800cac60 Switch to StreamInput.readEnum and StreamOutput.writeEnum (elastic/x-pack-elasticsearch#1313)
Start using StreamInput.readEnum and StreamOutput.writeEnum for enum serialization.

Relates to elastic/elasticsearch#24475

Original commit: elastic/x-pack-elasticsearch@67ca571458
2017-05-04 18:03:05 -04:00
Dimitris Athanasiou 9e3d5d6cf3 [ML] Provide better error message when doc values are unavailable (elastic/x-pack-elasticsearch#1312)
relates elastic/x-pack-elasticsearch#1305

Original commit: elastic/x-pack-elasticsearch@2e8ccc340b
2017-05-04 17:12:49 +01:00
Dimitris Athanasiou 4241469c89 [ML] Correctly print job state when delete job fails (elastic/x-pack-elasticsearch#1310)
relates elastic/x-pack-elasticsearch#1307

Original commit: elastic/x-pack-elasticsearch@25fb18ed0b
2017-05-04 15:17:48 +01:00
Simon Willnauer eddc7a2147 Add tests for cross cluster `_field_caps` API (elastic/x-pack-elasticsearch#1294)
This commit adds REST tests to ensure `_field_caps` works with cross cluster
search and security.

Relates to elastic/elasticsearch#24463

Original commit: elastic/x-pack-elasticsearch@2af0a8c179
2017-05-04 15:04:39 +02:00
Adrien Grand ea31227a89 Do not create the monitoring index before the exporter service is started.
This prevents the template from being applied.

relates elastic/x-pack-elasticsearch#1308

Original commit: elastic/x-pack-elasticsearch@c9fcea42ce
2017-05-04 14:40:32 +02:00
Dimitris Athanasiou 2290b93aa3 [ML] Ignore interim results during renormalization (elastic/x-pack-elasticsearch#1298)
relates elastic/x-pack-elasticsearch#1193

Original commit: elastic/x-pack-elasticsearch@6f5e354bdd
2017-05-04 11:38:52 +01:00
Dimitris Athanasiou 7f2c7dbe17 [ML] Rename datafeed indexes to indices (elastic/x-pack-elasticsearch#1297)
Renames datafeed indexes to indices but keeps indexes
as a synonym while parsing.

relates elastic/x-pack-elasticsearch#1292

Original commit: elastic/x-pack-elasticsearch@1fcdd97f88
2017-05-04 11:37:12 +01:00
Tim Vernum b922b37c3f [TEST] Fixes for randomised failures
- role_mapping.native is always present, but contains no entries if the security index is unavailable
- file based role mapping does not allow duplicate keys

Original commit: elastic/x-pack-elasticsearch@734cf7e2c0
2017-05-04 20:18:28 +10:00
Adrien Grand bbb3d008f2 Only allow indices to have a single type by default. (elastic/x-pack-elasticsearch#1212)
Sibling of elastic/elasticsearch#24317

Original commit: elastic/x-pack-elasticsearch@a57decdf9f
2017-05-04 10:23:23 +02:00
Alexander Reelsen 4078b2f1b2 Watcher: Replace _status field with status (elastic/x-pack-elasticsearch#1285)
As fields with underscores will be disallowed in master, and we have to
prepare the upgrade, this commit renames the _status field to status.

When the 5.x upgrade logic is in place in the 5.x we can remove all the
old style _status handling from the master branch.

Note: All the BWC compatibility tests, that load 5.x indices are now
faking a finished upgrade by adding the `status` field to the mapping
of the watches index.

Original commit: elastic/x-pack-elasticsearch@9d5cc9aaec
2017-05-04 10:08:34 +02:00
Tim Vernum 8633fd1f07 [TESTS] Fix problem with AD integration test
In some cases (based on the randomisation) the primary-realm would have no mappings, but the secondary-realm would.
If this occurred we wouldn't write out any mappings and the secondary-realm would not authorize the access being tested

Original commit: elastic/x-pack-elasticsearch@8e81ec9dd7
2017-05-04 18:05:26 +10:00
Jim Ferenczi 770404200e [TEST] Fix test that breaks compilation due to an invalid usage of ListenableActionFuture
Original commit: elastic/x-pack-elasticsearch@b92d06c9b1
2017-05-04 08:33:06 +02:00
Tim Vernum 6adf4fd3af Role Mapping API (elastic/x-pack-elasticsearch#925)
This introduces a role-mapping API to X-Pack security.

Features:
- A `GET`/`PUT`/`DELETE` API at `/_xpack/security/role_mapping/`
- Role-mappings are stored in the `.security` index 
- A custom expression language (in JSON) for expressing the mapping rules 
- Supported in LDAP/AD and PKI realms
- LDAP realm also supports loading arbitrary meta-data (which can be used in the mapping rules)
- A CompositeRoleMapper unifies roles from the existing file based mapper, and the new API based mapper.
- Usage stats for native role mappings

Original commit: elastic/x-pack-elasticsearch@d9972ed1da
2017-05-04 13:38:50 +10:00
Lisa Cawley cf27cb479a [DOCS] Clarify wildcard usage in ML datafeed APIs (elastic/x-pack-elasticsearch#1296)
Original commit: elastic/x-pack-elasticsearch@c18fca323b
2017-05-03 09:07:44 -07:00
Dimitris Athanasiou eae7d19d2f [DOCS] Improvements on datafeed related docs (elastic/x-pack-elasticsearch#1293)
Original commit: elastic/x-pack-elasticsearch@4bb1932f0e
2017-05-03 08:38:31 -07:00
Luca Cavanna 500170c456 [TEST] Remove more unnecessary ListenableActionFuture usages (elastic/x-pack-elasticsearch#1283)
Original commit: elastic/x-pack-elasticsearch@9dfe7ea6f9
2017-05-03 16:35:15 +02:00
Tim Brooks 106a26b399 Remove unneeded usages of listenable futures (elastic/x-pack-elasticsearch#1261)
This is related to elastic/elasticsearch#24412. That commit changed how
ListenableActionFuture implementations are created. This commit
updates x-pack to be compatible with those changes. In particular, all
the usages of ListenableActionFuture in x-pack could be replaced with
PlainActionFuture as the "listening" functionality was not being used.

Original commit: elastic/x-pack-elasticsearch@7c8d8e3df9
2017-05-03 09:23:39 -05:00
Dimitrios Athanasiou d140b3028d [ML] Reuse encapsulated delete-by-query request
Original commit: elastic/x-pack-elasticsearch@cc0f2ce815
2017-05-03 14:58:22 +01:00
Dimitris Athanasiou 3f73748d14 [ML] Use delete-by-query in JobDataDeleter (elastic/x-pack-elasticsearch#1274)
JobDataDeleter handles the deletion logic for 3 cases:

1. deleting a model snapshot and its state docs
2. deleting all results after a timestamp
3. deleting all interim results

The last 2 are currently implemented by manually performing
a search and scroll and then adding matching hits in a bulk
delete action. This operation is exactly what delete-by-query
does.

This commit changes JobDataDeleter to use delete-by-query. This
makes the code simpler and less error-prone. The downside is
losing some logging which seems non-critical. Unit tests for
JobDataDeleter are also removed as they are heavily mocked tests,
adding little value and high maintenance cost. This functionality
is tested by integration tests already.

relates elastic/x-pack-elasticsearch#821

Original commit: elastic/x-pack-elasticsearch@7da91332bd
2017-05-03 14:51:41 +01:00
Dimitris Athanasiou 3e9c36838d [ML] Remove get model snapshot by description functionality (elastic/x-pack-elasticsearch#1288)
relates elastic/x-pack-elasticsearch#1284

Original commit: elastic/x-pack-elasticsearch@780feea5c1
2017-05-03 13:20:52 +01:00
Luca Cavanna 796e23a02a Adapt to ActionRequestBuilder#execute changes (elastic/x-pack-elasticsearch#1263)
Original commit: elastic/x-pack-elasticsearch@c2d7aa562c
2017-05-03 11:21:06 +02:00
Hendrik Muhs 9d8fea9a32 [ML] expose low and high median functions (elastic/x-pack-elasticsearch#1272)
Expose low and high median functions implemented downstream in machinelearning-cpp.

Original commit: elastic/x-pack-elasticsearch@3863e9d41c
2017-05-03 09:46:10 +02:00
Daniel Mitterdorfer 240bd5bfa7 Mute EllipticCurveSSLTests
Relates elastic/x-pack-elasticsearch#1278

Original commit: elastic/x-pack-elasticsearch@3e5dbf5fe6
2017-05-03 07:57:27 +02:00
Chee Wee 1e0caee90d Corrected a mistake on disabling a user (elastic/x-pack-elasticsearch#1135)
Corrected a mistake on disabling a user, when it should be enabling.

Please review the changes.

Original commit: elastic/x-pack-elasticsearch@352997e427
2017-05-03 10:55:26 +08:00
Lisa Cawley 33c50f1201 [DOCS] Add property table for ML Update Jobs API (elastic/x-pack-elasticsearch#1268)
* [DOCS] Add property table for ML Update Jobs API

* [DOCS] Updates based on feedback for ML Update Jobs API

* [DOCS] Removed detector properties from ML Update Jobs API

* [DOCS] Fixes typos

Original commit: elastic/x-pack-elasticsearch@68d1b5598c
2017-05-02 15:38:41 -07:00
Lisa Cawley 9b2fb6ac16 [DOCS] Change "data feed" to "datafeed" in Machine Learning documentation (elastic/x-pack-elasticsearch#1277)
* [DOCS] Add xpackml attribute to XPack Reference

* [DOCS] Use attribute for datafeed terms

Original commit: elastic/x-pack-elasticsearch@f37bf48ee4
2017-05-02 12:45:42 -07:00
Alexander Reelsen 3a0bc504a9 Watcher: Exeuting a watch returns a proper 404 (elastic/x-pack-elasticsearch#1273)
A test failure uncovered that handling non existing watches in the
execute watch API could lead to 500 errors instead of 404 because
they were not handled correctly.

relates elastic/x-pack-elasticsearch#1120

Original commit: elastic/x-pack-elasticsearch@c17a07ae3e
2017-05-02 16:01:45 +02:00
Alexander Reelsen ff6283bf80 Tests: Ignore existing watches index in REST test
This test should ensure, that a missing watch on an existing index
causes an error, thus it can ignore if the .watches index already
exists.

relates elastic/x-pack-elasticsearch#1155

Original commit: elastic/x-pack-elasticsearch@89c56cb2be
2017-05-02 14:16:19 +02:00
Alexander Reelsen 50241717d3 Tests: Use execute watch API instead of persisting watch
To speed up the test and ensure no other failures occur,
simply use the execute watch API instead of persisting the watch
and executing then.

relates elastic/x-pack-elasticsearch#1350

Original commit: elastic/x-pack-elasticsearch@2888fb0e93
2017-05-02 14:04:04 +02:00
Tanguy Leroux 9e14141a8e [Test] Fix NPE in LocalExporterTests
This commit fixes a NullPointerException in the LocalExporterTests.

Original commit: elastic/x-pack-elasticsearch@e28516135f
2017-05-02 13:55:47 +02:00
David Kyle f3f387bd9c [ML] Give jobs meaningful and unique names in the yaml rest tests. (elastic/x-pack-elasticsearch#1271)
Original commit: elastic/x-pack-elasticsearch@b096de55c1
2017-05-02 11:36:33 +01:00