Commit Graph

6595 Commits

Author SHA1 Message Date
David Roberts 2e3aca414b [ML] Remove obsolete dynamic mapper setting (elastic/x-pack-elasticsearch#2477)
Since 5.6 we have only used one type per ML index, so this setting
is not necessary.

Original commit: elastic/x-pack-elasticsearch@64c434adec
2017-09-12 19:25:30 +01:00
jaymode 940e699e31 Remove index.mapper.dynamic from watcher and security templates
index.mapper.dynamic should not be used for 6.0+ indices, so this commit removes it from the
templates used by security and watcher.

Relates elastic/elasticsearch#25734

Original commit: elastic/x-pack-elasticsearch@766eebe660
2017-09-12 11:12:25 -06:00
Simon Willnauer 2f5aeb6c6f Remove token passphrase setting (elastic/x-pack-elasticsearch#2318)
This change removes `xpack.security.authc.token.passphrase` entirely since from
6.0 onwards we use randomly generated keys by the master there is no need for
this setting anymore. This setting will be deprecated from 6.0 onwards.

Original commit: elastic/x-pack-elasticsearch@37ba90359e
2017-09-12 15:34:41 +02:00
Alexander Reelsen c3f3ae5391 Watcher: Remove all traces from execution on master node (elastic/x-pack-elasticsearch#2383)
As there are no master node operations anymore.

* TransportActions are regular Actions now
* Watcher requests are now ActionRequests, no MasterNodeRequests anymore
* REST spec does not contain master node timeout parameters anymore
* WatcherLifeCycleService does not have a check anymore if watcher is able to run distributed, this will be a given in 7.0
* Some serialization BWC checks against version 5 have been removed

Original commit: elastic/x-pack-elasticsearch@4607dd538c
2017-09-12 15:05:26 +02:00
Dimitris Athanasiou e4882b36b7 [ML] Ensure datafeed runs on time (elastic/x-pack-elasticsearch#2471)
The datafeed runs on frequency-aligned intervals behind
query_delay. Currently, when a real-time run is triggered,
we subtract query_delay from now and then we take the aligned
interval. This results into running frequency + query_delay
behind now. The fix involves simply adding the query_delay
into the time real-time runs occur.

Relates elastic/x-pack-elasticsearch#2426

Original commit: elastic/x-pack-elasticsearch@61ceaaca8f
2017-09-12 13:24:55 +01:00
David Roberts 59d94eba40 [TEST] Mute failing test: CategorizationIT testBasicCategorization
See elastic/machine-learning-cpp#279

Original commit: elastic/x-pack-elasticsearch@03f0c307b7
2017-09-12 11:04:19 +01:00
David Roberts 1500074cb2 [ML] Add method to find the established memory use for a job (elastic/x-pack-elasticsearch#2449)
"Established" memory use will be one of the building blocks for smarter node
allocation.

In order for a job to be considered to have established memory usage it must:
- Have generated at least 20 buckets of results
- Have generated at least one model size stats document
- Have low variability of model bytes in model size stats documents in the
  time period covered by the last 20 buckets, which is defined as having a
  coefficient of variation of no more than 0.1

Relates elastic/x-pack-elasticsearch#546

Original commit: elastic/x-pack-elasticsearch@5032eb01d8
2017-09-12 10:25:53 +01:00
lcawley a979f33252 [DOCS] Added Dev Tools settings
Original commit: elastic/x-pack-elasticsearch@1538d28c21
2017-09-11 16:07:52 -07:00
Lisa Cawley 26148c91fe [DOCS] Format X-Pack migration APIs (elastic/x-pack-elasticsearch#2378)
Original commit: elastic/x-pack-elasticsearch@502d77b975
2017-09-11 14:02:23 -07:00
Lisa Cawley 90c6b93897 Add troubleshooting for ML shared index problem (elastic/x-pack-elasticsearch#2347)
* [DOCS] Add troubleshooting for ML shared index problem

* [DOCS] Address troubleshooting feedback

Original commit: elastic/x-pack-elasticsearch@3116524177
2017-09-11 08:39:14 -07:00
David Kyle bad65b4186 [ML] Add setting for job max model memory limit (elastic/x-pack-elasticsearch#2460)
* Add setting for job max model memory limit

* Address review comments

Original commit: elastic/x-pack-elasticsearch@5cec3a1abf
2017-09-11 14:53:46 +01:00
Jay Modi aaa0510821 Run core's full cluster restart tests with x-pack (elastic/x-pack-elasticsearch#2433)
This change pulls in the o.e.u.FullClusterRestartIT class from core and runs it as part of the
x-pack full cluster restart tests.

Relates elastic/x-pack-elasticsearch#1629

Original commit: elastic/x-pack-elasticsearch@87da59485f
2017-09-08 13:33:33 -06:00
Lisa Cawley 27a8041804 [DOCS] CCS no longer needs local *:* permission (elastic/x-pack-elasticsearch#2445)
Original commit: elastic/x-pack-elasticsearch@fb7f6eaeb2
2017-09-08 08:41:32 -07:00
David Kyle 51603620ee Mute ML rolling upgrade tests. Awaits fix elastic/x-pack-elasticsearch#1760
Original commit: elastic/x-pack-elasticsearch@deaf060818
2017-09-07 14:25:52 +01:00
Ryan Ernst 53294f217c Use versionless alias to ES rest client codebase (elastic/x-pack-elasticsearch#2441)
This is the xpack side of
https://github.com/elastic/elasticsearch/pull/26521

Original commit: elastic/x-pack-elasticsearch@b650e9e433
2017-09-06 18:58:17 -07:00
Lisa Cawley e4a008f9ee [DOCS] Update passphrase security setting (elastic/x-pack-elasticsearch#2431)
Original commit: elastic/x-pack-elasticsearch@8834d64f10
2017-09-06 08:13:59 -07:00
Lisa Cawley 0cd24a9283 [DOCS] Added kibana_dashboard_only_user role (elastic/x-pack-elasticsearch#2427)
Original commit: elastic/x-pack-elasticsearch@e6ab2238eb
2017-09-05 10:40:58 -07:00
Tim Brooks 84a47e2690 Disable `HipChatServiceTests` integration tests
This is related to elastic/x-pack-elasticsearch#2429. These tests are currently disabled due to
calls to the hipchat integration api failing. There is an open infra
issue for this (elastic/infra#2726).

Original commit: elastic/x-pack-elasticsearch@4aa9fe0387
2017-09-05 11:13:39 -06:00
David Roberts c73d70491a [TEST] Fix error if named pipe already connected (elastic/x-pack-elasticsearch#2423)
On Windows a named pipe server must call ConnectNamedPipe() before using
a named pipe.  However, if the client has already connected then this
function returns a failure code, with detailed error code
ERROR_PIPE_CONNECTED.  The server must check for this, as it means the
connection will work fine.  The Java test that emulates what the C++
would do in production did not have this logic.

This was purely a test problem.  The C++ code used in production already
does the right thing.

relates elastic/x-pack-elasticsearch#2359

Original commit: elastic/x-pack-elasticsearch@e162887f28
2017-09-05 13:39:22 +01:00
David Roberts 500b4ac6b9 [TEST] Improve ML security tests (elastic/x-pack-elasticsearch#2417)
The changes made for elastic/x-pack-elasticsearch#2369 showed that the ML security tests were seriously
weakened by the decision to grant many "minimal" privileges to all users
involved in the tests.  A better solution is to override the auth header
such that a superuser runs setup actions and assertions that work by
querying raw documents in ways that an end user wouldn't.  Then the ML
endpoints can be called with the privileges provided by the ML roles and
nothing else.

Original commit: elastic/x-pack-elasticsearch@4de42d9e54
2017-09-05 10:49:41 +01:00
Jason Tedor d9dce1afcd Fix typo in CI script
This commit fixes a typo in the CI script, the command-line flag for
setting tests.badapples had a typo.

Original commit: elastic/x-pack-elasticsearch@c67a620485
2017-09-03 18:38:40 -04:00
Jason Tedor bf68b5a907 Add intake option to CI script
This commit adds an option for running the intake build from the CI
scripts. The intake build differs from check in that it runs compilation
and pre-commit checks before running tests so that common build failures
causes will happen earlier than otherwise.

Relates elastic/x-pack-elasticsearch#2420

Original commit: elastic/x-pack-elasticsearch@b179ce1087
2017-09-03 16:43:15 -04:00
Alexander Reelsen cd5e001ca6 Watcher: Only load active watches on load (elastic/x-pack-elasticsearch#2408)
When watcher is loading it must only load the watches
which are active instead of all possible watches.

This loading happens on start up as well as when shards
relocate.

Original commit: elastic/x-pack-elasticsearch@29df56b99d
2017-09-01 16:20:42 +02:00
David Roberts 32b4c18ea3 [ML] Ensure internal client is used where appropriate (elastic/x-pack-elasticsearch#2415)
Implementation details of ML endpoints should be performed using the
internal client, so that the end user only requires permissions for
the public ML endpoints and does not need to know how they are
implemented.  This change fixes some instances where this rule was
not adhered to.

Original commit: elastic/x-pack-elasticsearch@01c8f5172c
2017-09-01 13:16:48 +01:00
David Kyle 6ba02e8087 [ML] Use the internal client for DeleteModelSnapshotAction (elastic/x-pack-elasticsearch#2407)
Original commit: elastic/x-pack-elasticsearch@7d3cbfa0cf
2017-08-31 18:16:30 +01:00
lcawley 7b13c4adc0 [DOCS] Copy relabel script from elasticsearch
Original commit: elastic/x-pack-elasticsearch@61423dbc94
2017-08-31 09:32:30 -07:00
Tim Vernum 57a07d6b5a Authorize on shard requests for bulk actions (elastic/x-pack-elasticsearch#2369)
* Add support for authz checks at on shard requests

* Add Rest Tests for authorization

* Bulk security - Only reject individual items, rather than a whole shard

* Sync with core change

* Grant "delete" priv in ML smoketest

This role had index and+bulk privileges but it also needs delete (in order to delete ML model-snapshots)

Original commit: elastic/x-pack-elasticsearch@830e89e652
2017-08-31 11:49:46 -04:00
Alexander Reelsen 2033b027ed Tests: Ensure watcher is enabled/disabled during tests (elastic/x-pack-elasticsearch#2392)
The method to check if watcher was enabled was returning
`randomBoolean()` and thus could change during test runs.

This fixes the test to ensure that always the same value
is returned and documents this requirement.

relates elastic/x-pack-elasticsearch#1783

Original commit: elastic/x-pack-elasticsearch@97bf3cfc29
2017-08-31 12:55:05 +02:00
Alexander Reelsen 4cada797d7 Watcher: Replace System.currentTimeMillis() with nanotime() (elastic/x-pack-elasticsearch#2393)
Enjoy the luxury of monotonically increasing clocks. So that
the duration will never be zero.

Original commit: elastic/x-pack-elasticsearch@c934ff0adb
2017-08-31 11:57:25 +02:00
Alexander Reelsen 1121a15eb7 Watcher: Ensure emit_stacktraces parameter works in watcher stats (elastic/x-pack-elasticsearch#2399)
This parameter ceased to work when Elasticsearch 5 introduced strict
parameter handling, because of a missing test.

This commit adds the parameter to the rest handler responseParams()
and adds a test along with the needed YAML definition.

relates elastic/x-pack-elasticsearch#2396

Original commit: elastic/x-pack-elasticsearch@8638df336c
2017-08-31 09:19:07 +02:00
Hendrik Muhs 381bb4ed7d add a test case to ensure NaN's aren't accepted (elastic/x-pack-elasticsearch#2395)
Original commit: elastic/x-pack-elasticsearch@72879c4f7c
2017-08-31 07:41:48 +02:00
David Kyle 91635608ef [ML] Rolling upgrade test job configurations with empty strings (elastic/x-pack-elasticsearch#2333)
Original commit: elastic/x-pack-elasticsearch@b61947cca7
2017-08-30 15:20:14 +01:00
David Kyle b818ee285b Revert "[ML] Check influencer names are valid (elastic/x-pack-elasticsearch#2073)"
This reverts commit elastic/x-pack-elasticsearch@75869cacb3.

Original commit: elastic/x-pack-elasticsearch@1f582ed260
2017-08-30 15:19:49 +01:00
Alexander Reelsen 03ec9df1d8 Tests: Remove is_true assertion that can return zero
Original commit: elastic/x-pack-elasticsearch@220d87fe4e
2017-08-30 15:46:38 +02:00
Dimitrios Athanasiou 5c145224a5 [ML] Add integrations tests for memory limit compliance
Relates elastic/x-pack-elasticsearch#2098

Original commit: elastic/x-pack-elasticsearch@730835edfa
2017-08-30 14:27:06 +01:00
Alexander Reelsen 200cb6a5b3 Tests: Remove is_true check in REST test
If the duration time was 0 (and this might happen due to
using System.currentTimeMillis), the is_true check
still returns false.

The correct fix will be done later to replace the offending
measurement calls and replace them. Then we can add back this
line.

Original commit: elastic/x-pack-elasticsearch@076a9a37cc
2017-08-30 14:04:24 +02:00
Alexander Reelsen 58d16c3579 Tests: Disable watcher in tribe TribeTransportTestCase
Watcher does not need to run in those tests. Disabling it,
no need for careful stopping then.

Relates elastic/x-pack-elasticsearch#1783

Original commit: elastic/x-pack-elasticsearch@ecca7f2ebd
2017-08-30 13:48:54 +02:00
Alexander Reelsen 80df309bbd Tests: Dont check for execution speed, only that field exists
Execution might be fast in combination with the clock not being counted
fast enough. As we are only interested in the field existing, we dont
need to check for the value.

relates elastic/x-pack-elasticsearch#2391

Original commit: elastic/x-pack-elasticsearch@8f3df967d9
2017-08-30 13:38:50 +02:00
Alexander Reelsen 4ac13fa9c5 Watcher: Replace integration tests with unit and REST tests (elastic/x-pack-elasticsearch#2322)
A bunch of integration tests should have been built as unit tests
or already have unit test equivalents.

This commit removes integration tests as well as adding REST equivalents
or creating unit tests instead of extending from AbstractWatcherIntegrationTestCase

Original commit: elastic/x-pack-elasticsearch@a97b99467d
2017-08-30 11:46:28 +02:00
Alexander Reelsen 17de601e21 Watcher: Remove the BWC compatible old watcher stats (elastic/x-pack-elasticsearch#2323)
Those were needed for the 5.x to 6.x transition phase, but can
be removed for 7.x.

Original commit: elastic/x-pack-elasticsearch@75572bd0c8
2017-08-30 11:45:57 +02:00
Lisa Cawley a56312a8e9 Update security info in X-Pack installation (elastic/x-pack-elasticsearch#2389)
* [DOCS] Update security info in X-Pack installation

* [DOCS] Remove bootstrap from security info

Original commit: elastic/x-pack-elasticsearch@fc272747b1
2017-08-29 13:17:20 -07:00
Jason Tedor 5cd92ffbbf Remove extraneous newlines from keystore.bash
This commit removes some extraneous trailing newlines from
keystore.bash, the packaging test cases for the interaction between
installing X-Pack and the keystore.

Original commit: elastic/x-pack-elasticsearch@86250ecfbc
2017-08-28 21:09:47 -04:00
Jason Tedor fb7118fde2 Add packaging tests for keystore creation
This commit adds a packaging test that the keystore is created when
X-Pack is installed, and that it has the correct ownership and
permissions.

Relates elastic/x-pack-elasticsearch#2380

Original commit: elastic/x-pack-elasticsearch@27e181d2f6
2017-08-28 20:47:58 -04:00
Lisa Cawley 895d73efe0 Format X-Pack Info API reference (elastic/x-pack-elasticsearch#2376)
* [DOCS] Format X-Pack Info API

* [DOCS] Augmented parameter descriptions for X-Pack Info API

Original commit: elastic/x-pack-elasticsearch@3ad97def27
2017-08-28 15:34:32 -07:00
Ryan Ernst 3ef93c55b0 Build: Add build metadata generation and usage to xpack (elastic/x-pack-elasticsearch#2368)
This commit adds generation of build_metadata files from bwc checkouts
as well as the sibling ES checkout of the xpack ci script.

relates elastic/elasticsearch#26397

Original commit: elastic/x-pack-elasticsearch@e17d904beb
2017-08-28 14:03:02 -07:00
debadair c85627f506 Update README.asciidoc
Original commit: elastic/x-pack-elasticsearch@c0f87f36a8
2017-08-28 11:35:42 -07:00
Jim Ferenczi 27d8b4c79c Remove the _all metadata field (elastic/x-pack-elasticsearch#2356)
This change removes the `_all` metadata field. This field is deprecated in 6
and cannot be activated for indices created in 6 so it can be safely removed in
the next major version (e.g. 7).

Relates https://github.com/elastic/elasticsearch/pull/26356

Original commit: elastic/x-pack-elasticsearch@a47133c94e
2017-08-28 13:01:27 +02:00
Michael Basnight e18f04f3eb Revert "Use shaded rest client dependencies" (elastic/x-pack-elasticsearch#2352)
This reverts commit elastic/x-pack-elasticsearch@8605560232.

Relates elastic/elasticsearch#26367

Original commit: elastic/x-pack-elasticsearch@e4cd960504
2017-08-25 14:13:16 -05:00
Alexander Reelsen 9b0a1a34e0 Upgrade: Remove watcher/security upgrade checks (elastic/x-pack-elasticsearch#2338)
The checks are used for the 5.6 to 6.x transition, thus they do
not make sense to keep in 7.x.

Original commit: elastic/x-pack-elasticsearch@c6c6fa819e
2017-08-25 17:24:49 +02:00
David Kyle c6b6a5c804 Fix failing ML test after bucket count change (elastic/x-pack-elasticsearch#2351)
Original commit: elastic/x-pack-elasticsearch@c215ba1c16
2017-08-25 10:25:18 +01:00