Commit Graph

634 Commits

Author SHA1 Message Date
David Kyle abbdf232aa [ML] Test ML with the Transport Client (elastic/x-pack-elasticsearch#1440)
* Hide ML actions for tribe node client
* Remove unused parameters
* Enable ML actions and rest endpoints for the transport client
* Create the ML components for the transport client
* Add ml transport client tests

Original commit: elastic/x-pack-elasticsearch@509007ca29
2017-05-16 14:34:44 +01:00
Dimitris Athanasiou dda456fb76 [ML] Add job version (elastic/x-pack-elasticsearch#1437)
relates elastic/x-pack-elasticsearch#1396

Original commit: elastic/x-pack-elasticsearch@3148c76d7f
2017-05-16 13:28:52 +01:00
Dimitrios Athanasiou e047598ca9 [ML] Fix timestamp comparison in scroll extractor cancellation logic
Original commit: elastic/x-pack-elasticsearch@1b3f88adf0
2017-05-16 13:26:57 +01:00
David Kyle 9f23c2c171 [ML] Wait for closing jobs in a close request (elastic/x-pack-elasticsearch#1398)
* Remove repeated calls to validateAndReturnJobTask

* Wait for closing job

* Refactor resolving job ids

* More close job unit tests

* Don’t finalise closing jobs twice

Original commit: elastic/x-pack-elasticsearch@20616d6f0a
2017-05-16 10:55:05 +01:00
Tanguy Leroux 9a06b5ee77 [Test] Converts LocalExporterTests.testLocalExporterFlush() as REST test (elastic/x-pack-elasticsearch#969)
The LocalExporterTests.testLocalExporterFlush() test was removed in elastic/x-pack-elasticsearch#835 when
LocalExporterTests was changed. This test verified that export exceptions are
thrown when monitoring documents are exported using the Monitoring Bulk API but
the underlying  monitoring indices are closed.

This commit reintroduces this test, but as a REST test this time.

relates elastic/x-pack-elasticsearch#416

Original commit: elastic/x-pack-elasticsearch@0a42f9a1be
2017-05-16 10:08:40 +02:00
Ryan Ernst ef3b954f63 Scripts: Convert template uses to return String instead of BytesReference (elastic/x-pack-elasticsearch#1279)
This is the xpack side of elastic/elasticsearch#24447. The one caveat is
there are a number of places within the xpack api that use
BytesReference to pass down to templates. These are convert to encode
into a BytesArray when necessary, so as not to require changing all of
those apis here at once, but they should all be convert to String as
well.

Original commit: elastic/x-pack-elasticsearch@8399b9d8c3
2017-05-15 22:37:51 -07:00
Jay Modi f5cd833af4 Use epoch millis for token expiration instead of ZonedDateTime (elastic/x-pack-elasticsearch#1417)
For user tokens, we were storing the expiration time as a date with the date time format in the
mappings. Occasionally, we would get CI failures due to parsing the date and having an invalid
format. This change simplifies the user tokens to simply use an Instant and convert it to the epoch
millis when we index the document. This eliminates the need for a date formatter and should ensure
we no longer have issues with parsing the dates.

Additionally, the TokenAuthIntegTests#testExpireMultipleTimes could fail if the token was expired
but the approximation for the expiration time was after the current time. In order to resolve this
we get the exact expiration time from the token and use that in the test.

relates elastic/x-pack-elasticsearch#1255

Original commit: elastic/x-pack-elasticsearch@d4bf53e7bc
2017-05-15 14:20:12 -06:00
Igor Motov 53753311f3 Move ReindexAction class to core (elastic/x-pack-elasticsearch#1426)
X-Pack counterpart of elastic/elasticsearch#24684

Relates to elastic/x-pack-elasticsearch#24578

Original commit: elastic/x-pack-elasticsearch@e5cd09cf4b
2017-05-15 14:29:17 -04:00
Tanguy Leroux ffcee524c5 Make SignificantTerms.Bucket an interface rather than an abstract class
Related to elastic/elasticsearch#24670

Original commit: elastic/x-pack-elasticsearch@e5b0123fbc
2017-05-15 15:19:19 +02:00
Dimitris Athanasiou e36e2c604d [ML] Write updated model snapshot to its original index (elastic/x-pack-elasticsearch#1415)
When we update a model snapshot we need to write it back to
the exact index where we read it from. This is necessary
for rollover as otherwise we would end up with two different
versions of the same snapshot.

Relates elastic/x-pack-elasticsearch#827

Original commit: elastic/x-pack-elasticsearch@b5d1ab38a7
2017-05-15 12:00:15 +01:00
Tim Vernum 463133b7de [Security] Cross cluster wildcard security (elastic/x-pack-elasticsearch#1290)
Support the resolution of remote index names, including those that contain wildcards in the cluster name or index part)

Specifically these work:
- `GET /remote*:foo/_search`
- `GET /*:foo/_search`
- `GET /*:foo,*/_search`
- `GET /remote:*/_search`
- `GET /*:*/_search`

This change assumes that every user is allowed to attempt a cross-cluster search against any remote index, and the actual authorisation of indices happens on the remote nodes. Thus ` GET /*:foo/_search` will expand to search the `foo` index on every registered remote without consideration of the roles and privileges that the user has on the source cluster.

Original commit: elastic/x-pack-elasticsearch@b45041aaa3
2017-05-15 15:02:13 +10:00
Andrew Cholakian 6413dcd759 [Logstash] New mapping for pipeline viewer (elastic/x-pack-elasticsearch#573)
Add `logstash-states` + other mapping fields for logstash pipeline viewer
This is part of https://github.com/elastic/x-pack-logstash/pull/25

Original commit: elastic/x-pack-elasticsearch@1940d79d13
2017-05-12 18:31:59 -05:00
Jason Tedor 879b4457f3 Update Netty to 4.1.11.Final
This commit upgrades the Netty dependency from 4.1.10.Final to
4.1.11.Final.

Relates elastic/x-pack-elasticsearch#1418

Original commit: elastic/x-pack-elasticsearch@1944a6f5f9
2017-05-12 15:54:05 -04:00
Dimitris Athanasiou 0ac38b05e4 [ML] Require job to have explicit data_description (elastic/x-pack-elasticsearch#1411)
relates elastic/x-pack-elasticsearch#1187

Original commit: elastic/x-pack-elasticsearch@aa2051f959
2017-05-12 17:24:27 +01:00
Alexander Reelsen 50e9e413da Watcher: Make watch history use doc type instead of watch_record (elastic/x-pack-elasticsearch#1311)
As this does not require any reindexing this is easy to fix by just
changing the watch history template.

In addition the old templates are deleted on start up and the new ones
are instantiated.

Original commit: elastic/x-pack-elasticsearch@7e1ad495ad
2017-05-12 16:52:57 +02:00
Jim Ferenczi de1d98b135 Adapt x-pack after the parent/child modularisation (elastic/x-pack-elasticsearch#1407)
This is the x-pack side of elastic/elasticsearch#24634
The hasChild, hasParent queries and the children agg are now in a module.

Original commit: elastic/x-pack-elasticsearch@e9b1296fc3
2017-05-12 15:59:40 +02:00
javanna 4f073cd4bd fix typo in AuthorizationService comment
Original commit: elastic/x-pack-elasticsearch@f747df0311
2017-05-12 11:18:01 +02:00
Simon Willnauer 0215356f12 Remove XPackDeleteByQueryAction BWC (elastic/x-pack-elasticsearch#1400)
5.5 will use delete by query from the module directly and has the
BWC layer in-place. This change therefore removes the BWC layer from 6.0

Relates to elastic/x-pack-elasticsearch#1378

Original commit: elastic/x-pack-elasticsearch@d4d4d6bc61
2017-05-12 10:19:21 +02:00
Simon Willnauer 2e34f160ff [TEST] Add ReindexPlugin to AutodetectResultProcessorIT
Original commit: elastic/x-pack-elasticsearch@acd19bf25c
2017-05-11 21:51:02 +02:00
Simon Willnauer 891c2a6c3f Replace XPacks delete_by_query impl with the core impl (elastic/x-pack-elasticsearch#1378)
This can now be shared with core once elastic/elasticsearch#24578

Original commit: elastic/x-pack-elasticsearch@42bbd75aee
2017-05-11 20:23:55 +02:00
Nik Everett 3ad2da0d87 Fix ordering
Core changed how you specify bucket ordering so we need to change
too.

Original commit: elastic/x-pack-elasticsearch@73d09f64c7
2017-05-11 13:44:17 -04:00
Nik Everett 16aa600830 Fix http request round trip tests
I broke this by adding more randomization to core's randomTimeValue
method. Now it produces valid time values that don't round trip
properly over the XContent API. This change causes the test to skip
trying to round trip time these sorts of time values.

Original commit: elastic/x-pack-elasticsearch@dcdd588bdb
2017-05-11 13:29:33 -04:00
Dimitrios Athanasiou 9ccf617a95 [TEST] Add explanation for custom random time value in ChunkingConfigTests
Original commit: elastic/x-pack-elasticsearch@9e03fb833f
2017-05-11 16:26:00 +01:00
Dimitrios Athanasiou 2e1ac420d7 [TEST] Fix ChunkingConfigTests due to time value randomization
Original commit: elastic/x-pack-elasticsearch@c145b399b6
2017-05-11 16:04:38 +01:00
Dimitris Athanasiou ba40994b1f [ML] Separate read from write index for results (elastic/x-pack-elasticsearch#1397)
This is in preparation of introducing a write alias.
It adjusts all requests to persist results to do so
using a method that returns the write alias (even though
it currently returns the same as the read alias).

Relates elastic/x-pack-elasticsearch#827

Original commit: elastic/x-pack-elasticsearch@1358dd8dcf
2017-05-11 14:44:40 +01:00
jaymode 4d3bc71327 Test: randomly assign the superuser role in RunAsIntegTests
This commit updates the RunAsIntegTests to randomly assign the superuser role to the user that
is authenticating with the cluster but the request is being run as a different user. This provides
additional validation that the authorization errors are actually coming from the user the request
is running as and not due to the authenticating user's privileges.

Original commit: elastic/x-pack-elasticsearch@c6360d13e6
2017-05-11 09:18:57 -04:00
Dimitris Athanasiou 69f9fa8ae9 [ML] Map new fields as keywords in results index (elastic/x-pack-elasticsearch#1387)
Each job introduces new fields to the results index matching
the analysis terms. When the job is created, mappings for those
are added explicitly. However, when rollover is introduced, that
will not be the case. This commit prepares for that by adding
dynamic mapping of new fields as keyword.

Relates elastic/x-pack-elasticsearch#827

Original commit: elastic/x-pack-elasticsearch@8f6cd09a71
2017-05-11 13:47:31 +01:00
David Roberts c4c57e6497 [TEST] Unit test that ML handles configs with brackets in field names (elastic/x-pack-elasticsearch#1395)
Original commit: elastic/x-pack-elasticsearch@a1d12bc254
2017-05-11 10:54:39 +01:00
Tim Vernum f3d5cf229b Change user cache TTL to be based on write not access time (elastic/x-pack-elasticsearch#1373)
This was the behaviour in Shield 2.x, but it was accidentally changed during migration to X-Pack 5.x

Original commit: elastic/x-pack-elasticsearch@de0bf5e688
2017-05-11 14:02:57 +10:00
Ryan Ernst f59b71629a Fix user copied in SecurityContext.executeAfterRewritingAuthentication (elastic/x-pack-elasticsearch#1391)
Also added a unit test for this method

Original commit: elastic/x-pack-elasticsearch@637a865119
2017-05-10 18:00:04 -07:00
Dimitrios Athanasiou 25505fad95 [TEST] Add missing verification in AutoDetectResultProcessorTests
Original commit: elastic/x-pack-elasticsearch@22b620a947
2017-05-10 17:59:54 +01:00
Dimitris Athanasiou 49eb5ea136 [ML] Retrieve model snapshot via search (elastic/x-pack-elasticsearch#1376)
This removes the last remaining GET in JobProvider.

Relates elastic/x-pack-elasticsearch#827

Original commit: elastic/x-pack-elasticsearch@820344be67
2017-05-10 17:26:37 +01:00
David Kyle 0c0961134d Fix compilation
Original commit: elastic/x-pack-elasticsearch@f4dc2c5ce7
2017-05-10 16:16:49 +01:00
David Kyle fdf86967ce [ML] Fix bug initialising ML MetaData (elastic/x-pack-elasticsearch#1386)
Slightly different to the commit I just reverted (elastic/x-pack-elasticsearch#1352)

Original commit: elastic/x-pack-elasticsearch@46339418ae
2017-05-10 16:06:13 +01:00
David Roberts 7a32304f2c [ML] Don't try to gracefully shut down the controller on unsupported platforms (elastic/x-pack-elasticsearch#1384)
It just wastes 20 seconds while we timeout trying to open named pipes that cannot
possibly have been created.

Original commit: elastic/x-pack-elasticsearch@4e447874f6
2017-05-10 16:03:16 +01:00
David Kyle 1994b42cd5 Revert "[ML] Fix bug initialising ML Metadata (elastic/x-pack-elasticsearch#1377)"
This reverts commit elastic/x-pack-elasticsearch@4e4923634a.

Original commit: elastic/x-pack-elasticsearch@3de80a1577
2017-05-10 15:56:49 +01:00
Hendrik Muhs c43ae014b4 Replace timestamp initialization 'now' to an explicit timestamp. (elastic/x-pack-elasticsearch#1383)
fixes build errors. Still ensures that the timestamp is set to 'now' if the parsed logfile misses it.

Original commit: elastic/x-pack-elasticsearch@cf60e8d76b
2017-05-10 16:17:30 +02:00
David Kyle d34192ff6f [ML] Fix bug initialising ML Metadata (elastic/x-pack-elasticsearch#1377)
* Fix bug initialising ML MetaData


Original commit: elastic/x-pack-elasticsearch@4e4923634a
2017-05-10 15:01:37 +01:00
Dimitris Athanasiou a5a44a2e2e [ML] Also revert quantiles and model_size_stats (elastic/x-pack-elasticsearch#1369)
When a model snapshot is reverted, we should also revert
quantiles and model_size_stats to the ones of the reverted
snapshot.

relates elastic/x-pack-elasticsearch#1342

Original commit: elastic/x-pack-elasticsearch@ddabe40470
2017-05-10 12:04:22 +01:00
David Roberts cd99024599 [ML] Validate job configs before transport (elastic/x-pack-elasticsearch#1375)
If invalid job configs are transported to the master node then the root
cause of the validation exception gets reported as a remote_transport_exception,
which is extremely confusing.

This commit moves the validation of job configurations to the first node that
handles the action.

Fixes elastic/x-pack-kibana#1172

Original commit: elastic/x-pack-elasticsearch@5ed59d2a6f
2017-05-10 10:55:16 +01:00
Tanguy Leroux f3b3df0911 [Test] Ensure monitoring indices are yellow in LocalExporterTests
Before searching for documents in monitoring indices, we need to ensure
that they exist and are available.

Original commit: elastic/x-pack-elasticsearch@29db55a1fe
2017-05-10 09:53:59 +02:00
Ryan Ernst 85deb1f8f7 Rename ScriptEngineService to ScriptEngine (elastic/x-pack-elasticsearch#1374)
This is the xpack side of elastic/elasticsearch#24574

Original commit: elastic/x-pack-elasticsearch@4d37021f6d
2017-05-10 00:47:39 -07:00
Hendrik Muhs c1016f3c3d [ML] simple log throttler for autodetect logging (elastic/x-pack-elasticsearch#1323)
Adds a simple log throttler for autodetect logging, summarizes log messages if
they repeat often in a short time period. Throttler gets disabled for debug logging.

relates to: https://github.com/elastic/machine-learning-cpp/issues/111

Original commit: elastic/x-pack-elasticsearch@6729b1fd7c
2017-05-10 09:25:05 +02:00
Ryan Ernst 940ca229aa Fix xpack test using deprecated setting
Original commit: elastic/x-pack-elasticsearch@f98d24782b
2017-05-09 17:32:50 -07:00
Ryan Ernst 1c3d907748 Reverse runAs user setup to store authenticated user inside runAs user (elastic/x-pack-elasticsearch#1371)
Original commit: elastic/x-pack-elasticsearch@8276662298
2017-05-09 13:49:14 -07:00
Jay Modi 590eea57ac Add a base security rest handler (elastic/x-pack-elasticsearch#1239)
This commit adds a base rest handler for security that handles the license checking in the security
apis. This was done previously in some rest handlers but not all and actually had issues where a
value would be returned but we may not have consumed all of the request parameters, which could
lead to a different response being returned than what we would have expected.

relates elastic/x-pack-elasticsearch#1236

Original commit: elastic/x-pack-elasticsearch@2f1100b64a
2017-05-09 14:47:11 -04:00
Chris Earle 7965def49c [TEST] Mark test with AwaitsFix for elastic/x-pack-elasticsearch#1353
Original commit: elastic/x-pack-elasticsearch@a9705d3816
2017-05-09 11:29:39 -04:00
Nik Everett 387944b95a Add defense against broken scrolls (elastic/x-pack-elasticsearch#1327)
Adds defenseagainst broken scrolls to the fetching roles and users.
While Elasticsearch *shouldn't* send broken scrolls it has done so
in the past and when it does this causes security to consume the
entire heap and crash. This changes it so we instead fail the request
with a message about the scroll being broken.

Relates to elastic/x-pack-elasticsearch#1299

Original commit: elastic/x-pack-elasticsearch@dfef87e757
2017-05-09 11:18:15 -04:00
Tanguy Leroux 0a860df9f9 [Test] Restore random documents indexing in LocalExporterTests (elastic/x-pack-elasticsearch#1328)
In elastic/x-pack-elasticsearch#1212, we removed a randomized condition that inserts documents when
the exporter is enabled but before the monitoring is started. This
condition should not have been removed as it allows to test that the
exporters are correctly initialized when the Monitoring Bulk API is used
 by an external application.

This commit also fixed a failure when a search request fails because the
 monitoring indices are not yet ready (see
 https://github.com/elastic/x-pack-elasticsearch/issues/956#issuecomment-298338589)

Original commit: elastic/x-pack-elasticsearch@73ab535ae0
2017-05-09 09:39:25 +02:00
jaymode b450664766 Test: ensure system supports ECDSA before running EllicpticCurveSSLTests
Some JDKs do not support the ECDSA cipher suites that we use in the EllipticCurveSSLTests, which
is the underlying cause of some CI failures. This change ensures there is at least one enabled
ECDSA cipher before testing that a connection can be made.

relates elastic/x-pack-elasticsearch#1278

Original commit: elastic/x-pack-elasticsearch@f6c93d776c
2017-05-08 12:36:20 -04:00