Commit Graph

614 Commits

Author SHA1 Message Date
Nik Everett 3ad2da0d87 Fix ordering
Core changed how you specify bucket ordering so we need to change
too.

Original commit: elastic/x-pack-elasticsearch@73d09f64c7
2017-05-11 13:44:17 -04:00
Nik Everett 16aa600830 Fix http request round trip tests
I broke this by adding more randomization to core's randomTimeValue
method. Now it produces valid time values that don't round trip
properly over the XContent API. This change causes the test to skip
trying to round trip time these sorts of time values.

Original commit: elastic/x-pack-elasticsearch@dcdd588bdb
2017-05-11 13:29:33 -04:00
Dimitrios Athanasiou 9ccf617a95 [TEST] Add explanation for custom random time value in ChunkingConfigTests
Original commit: elastic/x-pack-elasticsearch@9e03fb833f
2017-05-11 16:26:00 +01:00
Dimitrios Athanasiou 2e1ac420d7 [TEST] Fix ChunkingConfigTests due to time value randomization
Original commit: elastic/x-pack-elasticsearch@c145b399b6
2017-05-11 16:04:38 +01:00
Dimitris Athanasiou ba40994b1f [ML] Separate read from write index for results (elastic/x-pack-elasticsearch#1397)
This is in preparation of introducing a write alias.
It adjusts all requests to persist results to do so
using a method that returns the write alias (even though
it currently returns the same as the read alias).

Relates elastic/x-pack-elasticsearch#827

Original commit: elastic/x-pack-elasticsearch@1358dd8dcf
2017-05-11 14:44:40 +01:00
jaymode 4d3bc71327 Test: randomly assign the superuser role in RunAsIntegTests
This commit updates the RunAsIntegTests to randomly assign the superuser role to the user that
is authenticating with the cluster but the request is being run as a different user. This provides
additional validation that the authorization errors are actually coming from the user the request
is running as and not due to the authenticating user's privileges.

Original commit: elastic/x-pack-elasticsearch@c6360d13e6
2017-05-11 09:18:57 -04:00
Dimitris Athanasiou 69f9fa8ae9 [ML] Map new fields as keywords in results index (elastic/x-pack-elasticsearch#1387)
Each job introduces new fields to the results index matching
the analysis terms. When the job is created, mappings for those
are added explicitly. However, when rollover is introduced, that
will not be the case. This commit prepares for that by adding
dynamic mapping of new fields as keyword.

Relates elastic/x-pack-elasticsearch#827

Original commit: elastic/x-pack-elasticsearch@8f6cd09a71
2017-05-11 13:47:31 +01:00
David Roberts c4c57e6497 [TEST] Unit test that ML handles configs with brackets in field names (elastic/x-pack-elasticsearch#1395)
Original commit: elastic/x-pack-elasticsearch@a1d12bc254
2017-05-11 10:54:39 +01:00
Tim Vernum f3d5cf229b Change user cache TTL to be based on write not access time (elastic/x-pack-elasticsearch#1373)
This was the behaviour in Shield 2.x, but it was accidentally changed during migration to X-Pack 5.x

Original commit: elastic/x-pack-elasticsearch@de0bf5e688
2017-05-11 14:02:57 +10:00
Ryan Ernst f59b71629a Fix user copied in SecurityContext.executeAfterRewritingAuthentication (elastic/x-pack-elasticsearch#1391)
Also added a unit test for this method

Original commit: elastic/x-pack-elasticsearch@637a865119
2017-05-10 18:00:04 -07:00
Dimitrios Athanasiou 25505fad95 [TEST] Add missing verification in AutoDetectResultProcessorTests
Original commit: elastic/x-pack-elasticsearch@22b620a947
2017-05-10 17:59:54 +01:00
Dimitris Athanasiou 49eb5ea136 [ML] Retrieve model snapshot via search (elastic/x-pack-elasticsearch#1376)
This removes the last remaining GET in JobProvider.

Relates elastic/x-pack-elasticsearch#827

Original commit: elastic/x-pack-elasticsearch@820344be67
2017-05-10 17:26:37 +01:00
David Kyle 0c0961134d Fix compilation
Original commit: elastic/x-pack-elasticsearch@f4dc2c5ce7
2017-05-10 16:16:49 +01:00
David Kyle fdf86967ce [ML] Fix bug initialising ML MetaData (elastic/x-pack-elasticsearch#1386)
Slightly different to the commit I just reverted (elastic/x-pack-elasticsearch#1352)

Original commit: elastic/x-pack-elasticsearch@46339418ae
2017-05-10 16:06:13 +01:00
David Roberts 7a32304f2c [ML] Don't try to gracefully shut down the controller on unsupported platforms (elastic/x-pack-elasticsearch#1384)
It just wastes 20 seconds while we timeout trying to open named pipes that cannot
possibly have been created.

Original commit: elastic/x-pack-elasticsearch@4e447874f6
2017-05-10 16:03:16 +01:00
David Kyle 1994b42cd5 Revert "[ML] Fix bug initialising ML Metadata (elastic/x-pack-elasticsearch#1377)"
This reverts commit elastic/x-pack-elasticsearch@4e4923634a.

Original commit: elastic/x-pack-elasticsearch@3de80a1577
2017-05-10 15:56:49 +01:00
Hendrik Muhs c43ae014b4 Replace timestamp initialization 'now' to an explicit timestamp. (elastic/x-pack-elasticsearch#1383)
fixes build errors. Still ensures that the timestamp is set to 'now' if the parsed logfile misses it.

Original commit: elastic/x-pack-elasticsearch@cf60e8d76b
2017-05-10 16:17:30 +02:00
David Kyle d34192ff6f [ML] Fix bug initialising ML Metadata (elastic/x-pack-elasticsearch#1377)
* Fix bug initialising ML MetaData


Original commit: elastic/x-pack-elasticsearch@4e4923634a
2017-05-10 15:01:37 +01:00
Dimitris Athanasiou a5a44a2e2e [ML] Also revert quantiles and model_size_stats (elastic/x-pack-elasticsearch#1369)
When a model snapshot is reverted, we should also revert
quantiles and model_size_stats to the ones of the reverted
snapshot.

relates elastic/x-pack-elasticsearch#1342

Original commit: elastic/x-pack-elasticsearch@ddabe40470
2017-05-10 12:04:22 +01:00
David Roberts cd99024599 [ML] Validate job configs before transport (elastic/x-pack-elasticsearch#1375)
If invalid job configs are transported to the master node then the root
cause of the validation exception gets reported as a remote_transport_exception,
which is extremely confusing.

This commit moves the validation of job configurations to the first node that
handles the action.

Fixes elastic/x-pack-kibana#1172

Original commit: elastic/x-pack-elasticsearch@5ed59d2a6f
2017-05-10 10:55:16 +01:00
Tanguy Leroux f3b3df0911 [Test] Ensure monitoring indices are yellow in LocalExporterTests
Before searching for documents in monitoring indices, we need to ensure
that they exist and are available.

Original commit: elastic/x-pack-elasticsearch@29db55a1fe
2017-05-10 09:53:59 +02:00
Ryan Ernst 85deb1f8f7 Rename ScriptEngineService to ScriptEngine (elastic/x-pack-elasticsearch#1374)
This is the xpack side of elastic/elasticsearch#24574

Original commit: elastic/x-pack-elasticsearch@4d37021f6d
2017-05-10 00:47:39 -07:00
Hendrik Muhs c1016f3c3d [ML] simple log throttler for autodetect logging (elastic/x-pack-elasticsearch#1323)
Adds a simple log throttler for autodetect logging, summarizes log messages if
they repeat often in a short time period. Throttler gets disabled for debug logging.

relates to: https://github.com/elastic/machine-learning-cpp/issues/111

Original commit: elastic/x-pack-elasticsearch@6729b1fd7c
2017-05-10 09:25:05 +02:00
Ryan Ernst 940ca229aa Fix xpack test using deprecated setting
Original commit: elastic/x-pack-elasticsearch@f98d24782b
2017-05-09 17:32:50 -07:00
Ryan Ernst 1c3d907748 Reverse runAs user setup to store authenticated user inside runAs user (elastic/x-pack-elasticsearch#1371)
Original commit: elastic/x-pack-elasticsearch@8276662298
2017-05-09 13:49:14 -07:00
Jay Modi 590eea57ac Add a base security rest handler (elastic/x-pack-elasticsearch#1239)
This commit adds a base rest handler for security that handles the license checking in the security
apis. This was done previously in some rest handlers but not all and actually had issues where a
value would be returned but we may not have consumed all of the request parameters, which could
lead to a different response being returned than what we would have expected.

relates elastic/x-pack-elasticsearch#1236

Original commit: elastic/x-pack-elasticsearch@2f1100b64a
2017-05-09 14:47:11 -04:00
Chris Earle 7965def49c [TEST] Mark test with AwaitsFix for elastic/x-pack-elasticsearch#1353
Original commit: elastic/x-pack-elasticsearch@a9705d3816
2017-05-09 11:29:39 -04:00
Nik Everett 387944b95a Add defense against broken scrolls (elastic/x-pack-elasticsearch#1327)
Adds defenseagainst broken scrolls to the fetching roles and users.
While Elasticsearch *shouldn't* send broken scrolls it has done so
in the past and when it does this causes security to consume the
entire heap and crash. This changes it so we instead fail the request
with a message about the scroll being broken.

Relates to elastic/x-pack-elasticsearch#1299

Original commit: elastic/x-pack-elasticsearch@dfef87e757
2017-05-09 11:18:15 -04:00
Tanguy Leroux 0a860df9f9 [Test] Restore random documents indexing in LocalExporterTests (elastic/x-pack-elasticsearch#1328)
In elastic/x-pack-elasticsearch#1212, we removed a randomized condition that inserts documents when
the exporter is enabled but before the monitoring is started. This
condition should not have been removed as it allows to test that the
exporters are correctly initialized when the Monitoring Bulk API is used
 by an external application.

This commit also fixed a failure when a search request fails because the
 monitoring indices are not yet ready (see
 https://github.com/elastic/x-pack-elasticsearch/issues/956#issuecomment-298338589)

Original commit: elastic/x-pack-elasticsearch@73ab535ae0
2017-05-09 09:39:25 +02:00
jaymode b450664766 Test: ensure system supports ECDSA before running EllicpticCurveSSLTests
Some JDKs do not support the ECDSA cipher suites that we use in the EllipticCurveSSLTests, which
is the underlying cause of some CI failures. This change ensures there is at least one enabled
ECDSA cipher before testing that a connection can be made.

relates elastic/x-pack-elasticsearch#1278

Original commit: elastic/x-pack-elasticsearch@f6c93d776c
2017-05-08 12:36:20 -04:00
David Kyle e5b11d0222 [ML] Not an error to close a job twice (elastic/x-pack-elasticsearch#1340)
* [ML] Not an error to close a job twice

* Error if job is opening

* Address review comments

* Test closed job isn’t resolved

Original commit: elastic/x-pack-elasticsearch@7da7b24c08
2017-05-08 16:34:46 +01:00
David Roberts 9264ad541e [ML] Only attempt to get the native code version when ML is enabled (elastic/x-pack-elasticsearch#1346)
Originally we used to try to get the native code version even when ML was disabled.
However, this has proved to be an annoyance in cases where people running on
unsupported platforms have disabled ML. This commit means we'll only try to get
the native code version if ML is enabled on a node.

Original commit: elastic/x-pack-elasticsearch@778d6708d2
2017-05-08 16:16:23 +01:00
Dimitris Athanasiou 28f2ba3ef8 [ML] Retrieve data counts via search (elastic/x-pack-elasticsearch#1339)
Relates elastic/x-pack-elasticsearch#827

Original commit: elastic/x-pack-elasticsearch@73a6848526
2017-05-08 13:02:07 +01:00
David Kyle 4e25e1a24d [ML] Use XContentBuilders in try with resource statements (elastic/x-pack-elasticsearch#1329)
* [ML] Use XContentBuilders in try with resource statements

* Address review comments

Original commit: elastic/x-pack-elasticsearch@ef5b45e2f4
2017-05-08 11:41:23 +01:00
Tim Vernum 0a15b03395 [TESTS] A proper fix to MultipleAdRealmTests
Original commit: elastic/x-pack-elasticsearch@f04b3ebb7a
2017-05-08 00:17:00 +10:00
Chris Earle f18a7f0f68 [Monitoring] Remove unused "node" _type from .monitoring-data-2 (elastic/x-pack-elasticsearch#1333)
This removes the "node" type from `.monitoring-data-2`. This data is sent to _both_ the time-based and non-time-based indexes for Elasticsearch, but the UI only used the time-based variant already.

This is another step in the process of removing the `.monitoring-data-2` index. There is now only one `_type` left in that index: `cluster_info`, which is used by the UI and phone home stats because it contains the license details _and_ the `stack_stats` (e.g., `xpack_usage`).

Original commit: elastic/x-pack-elasticsearch@2cadb5939d
2017-05-05 17:48:30 -04:00
Tanguy Leroux 327ac9898d Fix compilation errors after elastic/elasticsearch/elastic/x-pack-elasticsearch#24492
Original commit: elastic/x-pack-elasticsearch@06d0c17165
2017-05-05 21:27:09 +02:00
Dimitris Athanasiou 3570eb32d3 [ML] Retrieve model_size_stats via search (elastic/x-pack-elasticsearch#1326)
This is a task towards allowing rollover.

Multiple model_size_stats are stored in order to allow
analytics of the memory usage over time. The job _stats
need to display the latest model_size_stats. Before this
commit, the latest model_size_stats was being stored with
a special ID and it was retrieved using that ID. This
does not lend itself well for rollover as we would end up
with multiple of those special IDs in the rolled indices.

This commit removes the need to store a special model_size_stats
version. Job _stats now retrieve the model_size_stats by searching
for the latest one. It also uses a manual ID for all model_size_stats
in order to maintain a single document per log_time.

Relates elastic/x-pack-elasticsearch#827

Original commit: elastic/x-pack-elasticsearch@b2796e9b08
2017-05-05 17:05:39 +01:00
Chris Earle 8ab46e800c [Monitoring] Stop Accepting .monitoring-data-N index requests via REST (elastic/x-pack-elasticsearch#1318)
Now that the Monitoring UI no longer checks the `.monitoring-data-2` index
for Kibana, Logstash, or Beats data, we can stop accepting the duplicated
data in that index (the _exact_ same documents are also indexed into the
time-based index for each product).

This ignores rather than rejects requests that contain such documents to
allow older clients to communicate with a 5.5+ monitoring cluster.

Original commit: elastic/x-pack-elasticsearch@def472cf2e
2017-05-05 11:53:43 -04:00
Tim Vernum 384486d6dd [TESTS] Really fix MultipleAdRealmTests (I think)
Original commit: elastic/x-pack-elasticsearch@dc1eb4eb93
2017-05-05 10:45:11 +10:00
Igor Motov 9a800cac60 Switch to StreamInput.readEnum and StreamOutput.writeEnum (elastic/x-pack-elasticsearch#1313)
Start using StreamInput.readEnum and StreamOutput.writeEnum for enum serialization.

Relates to elastic/elasticsearch#24475

Original commit: elastic/x-pack-elasticsearch@67ca571458
2017-05-04 18:03:05 -04:00
Dimitris Athanasiou 9e3d5d6cf3 [ML] Provide better error message when doc values are unavailable (elastic/x-pack-elasticsearch#1312)
relates elastic/x-pack-elasticsearch#1305

Original commit: elastic/x-pack-elasticsearch@2e8ccc340b
2017-05-04 17:12:49 +01:00
Dimitris Athanasiou 4241469c89 [ML] Correctly print job state when delete job fails (elastic/x-pack-elasticsearch#1310)
relates elastic/x-pack-elasticsearch#1307

Original commit: elastic/x-pack-elasticsearch@25fb18ed0b
2017-05-04 15:17:48 +01:00
Adrien Grand ea31227a89 Do not create the monitoring index before the exporter service is started.
This prevents the template from being applied.

relates elastic/x-pack-elasticsearch#1308

Original commit: elastic/x-pack-elasticsearch@c9fcea42ce
2017-05-04 14:40:32 +02:00
Dimitris Athanasiou 2290b93aa3 [ML] Ignore interim results during renormalization (elastic/x-pack-elasticsearch#1298)
relates elastic/x-pack-elasticsearch#1193

Original commit: elastic/x-pack-elasticsearch@6f5e354bdd
2017-05-04 11:38:52 +01:00
Dimitris Athanasiou 7f2c7dbe17 [ML] Rename datafeed indexes to indices (elastic/x-pack-elasticsearch#1297)
Renames datafeed indexes to indices but keeps indexes
as a synonym while parsing.

relates elastic/x-pack-elasticsearch#1292

Original commit: elastic/x-pack-elasticsearch@1fcdd97f88
2017-05-04 11:37:12 +01:00
Tim Vernum b922b37c3f [TEST] Fixes for randomised failures
- role_mapping.native is always present, but contains no entries if the security index is unavailable
- file based role mapping does not allow duplicate keys

Original commit: elastic/x-pack-elasticsearch@734cf7e2c0
2017-05-04 20:18:28 +10:00
Adrien Grand bbb3d008f2 Only allow indices to have a single type by default. (elastic/x-pack-elasticsearch#1212)
Sibling of elastic/elasticsearch#24317

Original commit: elastic/x-pack-elasticsearch@a57decdf9f
2017-05-04 10:23:23 +02:00
Alexander Reelsen 4078b2f1b2 Watcher: Replace _status field with status (elastic/x-pack-elasticsearch#1285)
As fields with underscores will be disallowed in master, and we have to
prepare the upgrade, this commit renames the _status field to status.

When the 5.x upgrade logic is in place in the 5.x we can remove all the
old style _status handling from the master branch.

Note: All the BWC compatibility tests, that load 5.x indices are now
faking a finished upgrade by adding the `status` field to the mapping
of the watches index.

Original commit: elastic/x-pack-elasticsearch@9d5cc9aaec
2017-05-04 10:08:34 +02:00
Tim Vernum 8633fd1f07 [TESTS] Fix problem with AD integration test
In some cases (based on the randomisation) the primary-realm would have no mappings, but the secondary-realm would.
If this occurred we wouldn't write out any mappings and the secondary-realm would not authorize the access being tested

Original commit: elastic/x-pack-elasticsearch@8e81ec9dd7
2017-05-04 18:05:26 +10:00