601 Commits

Author SHA1 Message Date
David Kyle
fdf86967ce [ML] Fix bug initialising ML MetaData (elastic/x-pack-elasticsearch#1386)
Slightly different to the commit I just reverted (elastic/x-pack-elasticsearch#1352)

Original commit: elastic/x-pack-elasticsearch@46339418ae
2017-05-10 16:06:13 +01:00
David Roberts
7a32304f2c [ML] Don't try to gracefully shut down the controller on unsupported platforms (elastic/x-pack-elasticsearch#1384)
It just wastes 20 seconds while we timeout trying to open named pipes that cannot
possibly have been created.

Original commit: elastic/x-pack-elasticsearch@4e447874f6
2017-05-10 16:03:16 +01:00
David Kyle
1994b42cd5 Revert "[ML] Fix bug initialising ML Metadata (elastic/x-pack-elasticsearch#1377)"
This reverts commit elastic/x-pack-elasticsearch@4e4923634a.

Original commit: elastic/x-pack-elasticsearch@3de80a1577
2017-05-10 15:56:49 +01:00
Hendrik Muhs
c43ae014b4 Replace timestamp initialization 'now' to an explicit timestamp. (elastic/x-pack-elasticsearch#1383)
fixes build errors. Still ensures that the timestamp is set to 'now' if the parsed logfile misses it.

Original commit: elastic/x-pack-elasticsearch@cf60e8d76b
2017-05-10 16:17:30 +02:00
David Kyle
d34192ff6f [ML] Fix bug initialising ML Metadata (elastic/x-pack-elasticsearch#1377)
* Fix bug initialising ML MetaData


Original commit: elastic/x-pack-elasticsearch@4e4923634a
2017-05-10 15:01:37 +01:00
Dimitris Athanasiou
a5a44a2e2e [ML] Also revert quantiles and model_size_stats (elastic/x-pack-elasticsearch#1369)
When a model snapshot is reverted, we should also revert
quantiles and model_size_stats to the ones of the reverted
snapshot.

relates elastic/x-pack-elasticsearch#1342

Original commit: elastic/x-pack-elasticsearch@ddabe40470
2017-05-10 12:04:22 +01:00
David Roberts
cd99024599 [ML] Validate job configs before transport (elastic/x-pack-elasticsearch#1375)
If invalid job configs are transported to the master node then the root
cause of the validation exception gets reported as a remote_transport_exception,
which is extremely confusing.

This commit moves the validation of job configurations to the first node that
handles the action.

Fixes elastic/x-pack-kibana#1172

Original commit: elastic/x-pack-elasticsearch@5ed59d2a6f
2017-05-10 10:55:16 +01:00
Tanguy Leroux
f3b3df0911 [Test] Ensure monitoring indices are yellow in LocalExporterTests
Before searching for documents in monitoring indices, we need to ensure
that they exist and are available.

Original commit: elastic/x-pack-elasticsearch@29db55a1fe
2017-05-10 09:53:59 +02:00
Ryan Ernst
85deb1f8f7 Rename ScriptEngineService to ScriptEngine (elastic/x-pack-elasticsearch#1374)
This is the xpack side of elastic/elasticsearch#24574

Original commit: elastic/x-pack-elasticsearch@4d37021f6d
2017-05-10 00:47:39 -07:00
Hendrik Muhs
c1016f3c3d [ML] simple log throttler for autodetect logging (elastic/x-pack-elasticsearch#1323)
Adds a simple log throttler for autodetect logging, summarizes log messages if
they repeat often in a short time period. Throttler gets disabled for debug logging.

relates to: https://github.com/elastic/machine-learning-cpp/issues/111

Original commit: elastic/x-pack-elasticsearch@6729b1fd7c
2017-05-10 09:25:05 +02:00
Ryan Ernst
940ca229aa Fix xpack test using deprecated setting
Original commit: elastic/x-pack-elasticsearch@f98d24782b
2017-05-09 17:32:50 -07:00
Ryan Ernst
1c3d907748 Reverse runAs user setup to store authenticated user inside runAs user (elastic/x-pack-elasticsearch#1371)
Original commit: elastic/x-pack-elasticsearch@8276662298
2017-05-09 13:49:14 -07:00
Jay Modi
590eea57ac Add a base security rest handler (elastic/x-pack-elasticsearch#1239)
This commit adds a base rest handler for security that handles the license checking in the security
apis. This was done previously in some rest handlers but not all and actually had issues where a
value would be returned but we may not have consumed all of the request parameters, which could
lead to a different response being returned than what we would have expected.

relates elastic/x-pack-elasticsearch#1236

Original commit: elastic/x-pack-elasticsearch@2f1100b64a
2017-05-09 14:47:11 -04:00
Chris Earle
7965def49c [TEST] Mark test with AwaitsFix for elastic/x-pack-elasticsearch#1353
Original commit: elastic/x-pack-elasticsearch@a9705d3816
2017-05-09 11:29:39 -04:00
Nik Everett
387944b95a Add defense against broken scrolls (elastic/x-pack-elasticsearch#1327)
Adds defenseagainst broken scrolls to the fetching roles and users.
While Elasticsearch *shouldn't* send broken scrolls it has done so
in the past and when it does this causes security to consume the
entire heap and crash. This changes it so we instead fail the request
with a message about the scroll being broken.

Relates to elastic/x-pack-elasticsearch#1299

Original commit: elastic/x-pack-elasticsearch@dfef87e757
2017-05-09 11:18:15 -04:00
Tanguy Leroux
0a860df9f9 [Test] Restore random documents indexing in LocalExporterTests (elastic/x-pack-elasticsearch#1328)
In elastic/x-pack-elasticsearch#1212, we removed a randomized condition that inserts documents when
the exporter is enabled but before the monitoring is started. This
condition should not have been removed as it allows to test that the
exporters are correctly initialized when the Monitoring Bulk API is used
 by an external application.

This commit also fixed a failure when a search request fails because the
 monitoring indices are not yet ready (see
 https://github.com/elastic/x-pack-elasticsearch/issues/956#issuecomment-298338589)

Original commit: elastic/x-pack-elasticsearch@73ab535ae0
2017-05-09 09:39:25 +02:00
jaymode
b450664766 Test: ensure system supports ECDSA before running EllicpticCurveSSLTests
Some JDKs do not support the ECDSA cipher suites that we use in the EllipticCurveSSLTests, which
is the underlying cause of some CI failures. This change ensures there is at least one enabled
ECDSA cipher before testing that a connection can be made.

relates elastic/x-pack-elasticsearch#1278

Original commit: elastic/x-pack-elasticsearch@f6c93d776c
2017-05-08 12:36:20 -04:00
David Kyle
e5b11d0222 [ML] Not an error to close a job twice (elastic/x-pack-elasticsearch#1340)
* [ML] Not an error to close a job twice

* Error if job is opening

* Address review comments

* Test closed job isn’t resolved

Original commit: elastic/x-pack-elasticsearch@7da7b24c08
2017-05-08 16:34:46 +01:00
David Roberts
9264ad541e [ML] Only attempt to get the native code version when ML is enabled (elastic/x-pack-elasticsearch#1346)
Originally we used to try to get the native code version even when ML was disabled.
However, this has proved to be an annoyance in cases where people running on
unsupported platforms have disabled ML. This commit means we'll only try to get
the native code version if ML is enabled on a node.

Original commit: elastic/x-pack-elasticsearch@778d6708d2
2017-05-08 16:16:23 +01:00
Dimitris Athanasiou
28f2ba3ef8 [ML] Retrieve data counts via search (elastic/x-pack-elasticsearch#1339)
Relates elastic/x-pack-elasticsearch#827

Original commit: elastic/x-pack-elasticsearch@73a6848526
2017-05-08 13:02:07 +01:00
David Kyle
4e25e1a24d [ML] Use XContentBuilders in try with resource statements (elastic/x-pack-elasticsearch#1329)
* [ML] Use XContentBuilders in try with resource statements

* Address review comments

Original commit: elastic/x-pack-elasticsearch@ef5b45e2f4
2017-05-08 11:41:23 +01:00
Tim Vernum
0a15b03395 [TESTS] A proper fix to MultipleAdRealmTests
Original commit: elastic/x-pack-elasticsearch@f04b3ebb7a
2017-05-08 00:17:00 +10:00
Chris Earle
f18a7f0f68 [Monitoring] Remove unused "node" _type from .monitoring-data-2 (elastic/x-pack-elasticsearch#1333)
This removes the "node" type from `.monitoring-data-2`. This data is sent to _both_ the time-based and non-time-based indexes for Elasticsearch, but the UI only used the time-based variant already.

This is another step in the process of removing the `.monitoring-data-2` index. There is now only one `_type` left in that index: `cluster_info`, which is used by the UI and phone home stats because it contains the license details _and_ the `stack_stats` (e.g., `xpack_usage`).

Original commit: elastic/x-pack-elasticsearch@2cadb5939d
2017-05-05 17:48:30 -04:00
Tanguy Leroux
327ac9898d Fix compilation errors after elastic/elasticsearch/elastic/x-pack-elasticsearch#24492
Original commit: elastic/x-pack-elasticsearch@06d0c17165
2017-05-05 21:27:09 +02:00
Dimitris Athanasiou
3570eb32d3 [ML] Retrieve model_size_stats via search (elastic/x-pack-elasticsearch#1326)
This is a task towards allowing rollover.

Multiple model_size_stats are stored in order to allow
analytics of the memory usage over time. The job _stats
need to display the latest model_size_stats. Before this
commit, the latest model_size_stats was being stored with
a special ID and it was retrieved using that ID. This
does not lend itself well for rollover as we would end up
with multiple of those special IDs in the rolled indices.

This commit removes the need to store a special model_size_stats
version. Job _stats now retrieve the model_size_stats by searching
for the latest one. It also uses a manual ID for all model_size_stats
in order to maintain a single document per log_time.

Relates elastic/x-pack-elasticsearch#827

Original commit: elastic/x-pack-elasticsearch@b2796e9b08
2017-05-05 17:05:39 +01:00
Chris Earle
8ab46e800c [Monitoring] Stop Accepting .monitoring-data-N index requests via REST (elastic/x-pack-elasticsearch#1318)
Now that the Monitoring UI no longer checks the `.monitoring-data-2` index
for Kibana, Logstash, or Beats data, we can stop accepting the duplicated
data in that index (the _exact_ same documents are also indexed into the
time-based index for each product).

This ignores rather than rejects requests that contain such documents to
allow older clients to communicate with a 5.5+ monitoring cluster.

Original commit: elastic/x-pack-elasticsearch@def472cf2e
2017-05-05 11:53:43 -04:00
Tim Vernum
384486d6dd [TESTS] Really fix MultipleAdRealmTests (I think)
Original commit: elastic/x-pack-elasticsearch@dc1eb4eb93
2017-05-05 10:45:11 +10:00
Igor Motov
9a800cac60 Switch to StreamInput.readEnum and StreamOutput.writeEnum (elastic/x-pack-elasticsearch#1313)
Start using StreamInput.readEnum and StreamOutput.writeEnum for enum serialization.

Relates to elastic/elasticsearch#24475

Original commit: elastic/x-pack-elasticsearch@67ca571458
2017-05-04 18:03:05 -04:00
Dimitris Athanasiou
9e3d5d6cf3 [ML] Provide better error message when doc values are unavailable (elastic/x-pack-elasticsearch#1312)
relates elastic/x-pack-elasticsearch#1305

Original commit: elastic/x-pack-elasticsearch@2e8ccc340b
2017-05-04 17:12:49 +01:00
Dimitris Athanasiou
4241469c89 [ML] Correctly print job state when delete job fails (elastic/x-pack-elasticsearch#1310)
relates elastic/x-pack-elasticsearch#1307

Original commit: elastic/x-pack-elasticsearch@25fb18ed0b
2017-05-04 15:17:48 +01:00
Adrien Grand
ea31227a89 Do not create the monitoring index before the exporter service is started.
This prevents the template from being applied.

relates elastic/x-pack-elasticsearch#1308

Original commit: elastic/x-pack-elasticsearch@c9fcea42ce
2017-05-04 14:40:32 +02:00
Dimitris Athanasiou
2290b93aa3 [ML] Ignore interim results during renormalization (elastic/x-pack-elasticsearch#1298)
relates elastic/x-pack-elasticsearch#1193

Original commit: elastic/x-pack-elasticsearch@6f5e354bdd
2017-05-04 11:38:52 +01:00
Dimitris Athanasiou
7f2c7dbe17 [ML] Rename datafeed indexes to indices (elastic/x-pack-elasticsearch#1297)
Renames datafeed indexes to indices but keeps indexes
as a synonym while parsing.

relates elastic/x-pack-elasticsearch#1292

Original commit: elastic/x-pack-elasticsearch@1fcdd97f88
2017-05-04 11:37:12 +01:00
Tim Vernum
b922b37c3f [TEST] Fixes for randomised failures
- role_mapping.native is always present, but contains no entries if the security index is unavailable
- file based role mapping does not allow duplicate keys

Original commit: elastic/x-pack-elasticsearch@734cf7e2c0
2017-05-04 20:18:28 +10:00
Adrien Grand
bbb3d008f2 Only allow indices to have a single type by default. (elastic/x-pack-elasticsearch#1212)
Sibling of elastic/elasticsearch#24317

Original commit: elastic/x-pack-elasticsearch@a57decdf9f
2017-05-04 10:23:23 +02:00
Alexander Reelsen
4078b2f1b2 Watcher: Replace _status field with status (elastic/x-pack-elasticsearch#1285)
As fields with underscores will be disallowed in master, and we have to
prepare the upgrade, this commit renames the _status field to status.

When the 5.x upgrade logic is in place in the 5.x we can remove all the
old style _status handling from the master branch.

Note: All the BWC compatibility tests, that load 5.x indices are now
faking a finished upgrade by adding the `status` field to the mapping
of the watches index.

Original commit: elastic/x-pack-elasticsearch@9d5cc9aaec
2017-05-04 10:08:34 +02:00
Tim Vernum
8633fd1f07 [TESTS] Fix problem with AD integration test
In some cases (based on the randomisation) the primary-realm would have no mappings, but the secondary-realm would.
If this occurred we wouldn't write out any mappings and the secondary-realm would not authorize the access being tested

Original commit: elastic/x-pack-elasticsearch@8e81ec9dd7
2017-05-04 18:05:26 +10:00
Jim Ferenczi
770404200e [TEST] Fix test that breaks compilation due to an invalid usage of ListenableActionFuture
Original commit: elastic/x-pack-elasticsearch@b92d06c9b1
2017-05-04 08:33:06 +02:00
Tim Vernum
6adf4fd3af Role Mapping API (elastic/x-pack-elasticsearch#925)
This introduces a role-mapping API to X-Pack security.

Features:
- A `GET`/`PUT`/`DELETE` API at `/_xpack/security/role_mapping/`
- Role-mappings are stored in the `.security` index 
- A custom expression language (in JSON) for expressing the mapping rules 
- Supported in LDAP/AD and PKI realms
- LDAP realm also supports loading arbitrary meta-data (which can be used in the mapping rules)
- A CompositeRoleMapper unifies roles from the existing file based mapper, and the new API based mapper.
- Usage stats for native role mappings

Original commit: elastic/x-pack-elasticsearch@d9972ed1da
2017-05-04 13:38:50 +10:00
Luca Cavanna
500170c456 [TEST] Remove more unnecessary ListenableActionFuture usages (elastic/x-pack-elasticsearch#1283)
Original commit: elastic/x-pack-elasticsearch@9dfe7ea6f9
2017-05-03 16:35:15 +02:00
Tim Brooks
106a26b399 Remove unneeded usages of listenable futures (elastic/x-pack-elasticsearch#1261)
This is related to elastic/elasticsearch#24412. That commit changed how
ListenableActionFuture implementations are created. This commit
updates x-pack to be compatible with those changes. In particular, all
the usages of ListenableActionFuture in x-pack could be replaced with
PlainActionFuture as the "listening" functionality was not being used.

Original commit: elastic/x-pack-elasticsearch@7c8d8e3df9
2017-05-03 09:23:39 -05:00
Dimitrios Athanasiou
d140b3028d [ML] Reuse encapsulated delete-by-query request
Original commit: elastic/x-pack-elasticsearch@cc0f2ce815
2017-05-03 14:58:22 +01:00
Dimitris Athanasiou
3f73748d14 [ML] Use delete-by-query in JobDataDeleter (elastic/x-pack-elasticsearch#1274)
JobDataDeleter handles the deletion logic for 3 cases:

1. deleting a model snapshot and its state docs
2. deleting all results after a timestamp
3. deleting all interim results

The last 2 are currently implemented by manually performing
a search and scroll and then adding matching hits in a bulk
delete action. This operation is exactly what delete-by-query
does.

This commit changes JobDataDeleter to use delete-by-query. This
makes the code simpler and less error-prone. The downside is
losing some logging which seems non-critical. Unit tests for
JobDataDeleter are also removed as they are heavily mocked tests,
adding little value and high maintenance cost. This functionality
is tested by integration tests already.

relates elastic/x-pack-elasticsearch#821

Original commit: elastic/x-pack-elasticsearch@7da91332bd
2017-05-03 14:51:41 +01:00
Dimitris Athanasiou
3e9c36838d [ML] Remove get model snapshot by description functionality (elastic/x-pack-elasticsearch#1288)
relates elastic/x-pack-elasticsearch#1284

Original commit: elastic/x-pack-elasticsearch@780feea5c1
2017-05-03 13:20:52 +01:00
Luca Cavanna
796e23a02a Adapt to ActionRequestBuilder#execute changes (elastic/x-pack-elasticsearch#1263)
Original commit: elastic/x-pack-elasticsearch@c2d7aa562c
2017-05-03 11:21:06 +02:00
Hendrik Muhs
9d8fea9a32 [ML] expose low and high median functions (elastic/x-pack-elasticsearch#1272)
Expose low and high median functions implemented downstream in machinelearning-cpp.

Original commit: elastic/x-pack-elasticsearch@3863e9d41c
2017-05-03 09:46:10 +02:00
Daniel Mitterdorfer
240bd5bfa7 Mute EllipticCurveSSLTests
Relates elastic/x-pack-elasticsearch#1278

Original commit: elastic/x-pack-elasticsearch@3e5dbf5fe6
2017-05-03 07:57:27 +02:00
Alexander Reelsen
3a0bc504a9 Watcher: Exeuting a watch returns a proper 404 (elastic/x-pack-elasticsearch#1273)
A test failure uncovered that handling non existing watches in the
execute watch API could lead to 500 errors instead of 404 because
they were not handled correctly.

relates elastic/x-pack-elasticsearch#1120

Original commit: elastic/x-pack-elasticsearch@c17a07ae3e
2017-05-02 16:01:45 +02:00
Alexander Reelsen
ff6283bf80 Tests: Ignore existing watches index in REST test
This test should ensure, that a missing watch on an existing index
causes an error, thus it can ignore if the .watches index already
exists.

relates elastic/x-pack-elasticsearch#1155

Original commit: elastic/x-pack-elasticsearch@89c56cb2be
2017-05-02 14:16:19 +02:00
Tanguy Leroux
9e14141a8e [Test] Fix NPE in LocalExporterTests
This commit fixes a NullPointerException in the LocalExporterTests.

Original commit: elastic/x-pack-elasticsearch@e28516135f
2017-05-02 13:55:47 +02:00