Commit Graph

6722 Commits

Author SHA1 Message Date
Jay Modi 53d6d945f0 Update documentation to reflect the latest TLS changes and licensing (elastic/x-pack-elasticsearch#2508)
This commit updates to documentation and adds notes about TLS being required to install a
license.

Relates elastic/x-pack-elasticsearch#2463

Original commit: elastic/x-pack-elasticsearch@0d8bfb98ea
2017-09-15 08:44:03 -06:00
Simon Willnauer 4d20586b24 [TEST] add integration test that ensures we reject license upgrades if TLS is not enabled
Original commit: elastic/x-pack-elasticsearch@dfbadb5e5f
2017-09-15 14:47:28 +02:00
Simon Willnauer c3066d1a51 Merge branch 'master' into tls_6.0
Original commit: elastic/x-pack-elasticsearch@9ce33bc7c3
2017-09-15 09:51:16 +02:00
Simon Willnauer 023bdb72b2 Add common-analysis plugin to several xpack integ tests (elastic/x-pack-elasticsearch#2501)
Several tests miss the common-analysis plugin in the old-style integ
tests causing odd exceptions in the test logs. This adds the missing plugin reference.

relates elastic/x-pack-elasticsearch#2363

Original commit: elastic/x-pack-elasticsearch@e4e6735408
2017-09-15 09:25:36 +02:00
Lisa Cawley 895d28f462 [DOCS] Remove redundant users command info (elastic/x-pack-elasticsearch#2504)
Original commit: elastic/x-pack-elasticsearch@1c9fa91293
2017-09-14 15:47:21 -07:00
jaymode 8997792875 Test: use TLS for plugin integ tests
Original commit: elastic/x-pack-elasticsearch@99971d7256
2017-09-14 15:57:28 -06:00
lcawley c3b658df88 [DOCS] Fixed broken link to put watch API
Original commit: elastic/x-pack-elasticsearch@f36caaa371
2017-09-14 14:12:20 -07:00
Michael Basnight fa0b854fb6 Update rest-api-spec to use bad_request (elastic/x-pack-elasticsearch#2507)
ref #elastic/elasticsearch#26539

Original commit: elastic/x-pack-elasticsearch@8b79a0769a
2017-09-14 15:59:29 -05:00
Lisa Cawley df1e4e85a4 Format Watcher APIs (elastic/x-pack-elasticsearch#2382)
* [DOCS] Format Watcher APIs

* [DOCS] Removed master_timeout from Watcher APIs

* [DOCS] Added authority info to watcher APIs

Original commit: elastic/x-pack-elasticsearch@1e6de3b036
2017-09-14 13:01:47 -07:00
Andy Bristol 279c7e14fd [TEST] fix security template version check in rest tests (elastic/x-pack-elasticsearch#2506)
Since the template upgrade service was added, upgrades should
be performed by a node with the highest version in the cluster,
which may not be the master node.

Original commit: elastic/x-pack-elasticsearch@d66145de54
2017-09-14 12:16:20 -07:00
Jay Modi 57de66476c Disable TLS by default (elastic/x-pack-elasticsearch#2481)
This commit adds back the ability to disable TLS on the transport layer and also disables TLS by
default to restore the 5.x behavior. The auto generation of key/cert and bundled CA certificate
have also been removed.

Relates elastic/x-pack-elasticsearch#2463

Original commit: elastic/x-pack-elasticsearch@abc66ec67d
2017-09-14 12:18:54 -06:00
Simon Willnauer 1e14e14571 Prevent licenses to be upgraded to production unless TLS is configured (elastic/x-pack-elasticsearch#2502)
if a user tries to upgrade a license to a production license and has security
enabled we prevent the upgrade unless TLS is setup. This is a requirement now
if a cluster with security is running in prodcution.

Relates to elastic/x-pack-elasticsearch#2463

Original commit: elastic/x-pack-elasticsearch@d61ef3bcb1
2017-09-14 20:14:27 +02:00
Jay Modi 8d8baffe24 Add specific client and user for security index access (elastic/x-pack-elasticsearch#2492)
This change removes security index access from the xpack user by creating its own specific role
and adds a xpack security user that maintains the superuser role so that it can perform all
operations necessary for security.

Original commit: elastic/x-pack-elasticsearch@ad906bc913
2017-09-14 08:09:14 -06:00
David Roberts 104a3a323f [BUILD] Make AWS error message more informative
Original commit: elastic/x-pack-elasticsearch@42cca7ed82
2017-09-14 14:46:48 +01:00
Simon Willnauer 91b57ee63f Add bootstrap check that enforces TLS if a production license is in the local clusterstate (elastic/x-pack-elasticsearch#2499)
This change will enforce transport SSL to be enforced if security is enabled and the
license in the clusterstate is a production license. The cluster state is loaded from
local storage such that we don't need to join a cluster to make these checks. Yet, the cluster
might have already got a different license if the node got disconnected while the license got
downgraded and then TLS got disabled. This corner case requires manual intervention which
we consider ok given the simplicity of this change.

Relates to elastic/x-pack-elasticsearch#2463

Original commit: elastic/x-pack-elasticsearch@5765b7cd21
2017-09-14 13:52:53 +02:00
Hendrik Muhs 7d19264363 [ML-FC] Branch landing feature/ml (elastic/x-pack-elasticsearch#2500)
integrate forecasting feature branch into master

    - add endpoint xpack/ml/job/forecast to request forecasting on data of ml-jobs
       - current parameters: end time
    - persists forecast results into shared or own index
       - different runs are separated by a 'forecast id'

relates elastic/x-pack-elasticsearch#1838

Original commit: elastic/x-pack-elasticsearch@f9d701a6bc
2017-09-14 12:31:20 +02:00
Simon Willnauer 3b00251a96 Merge branch 'master' into tls_6.0
Original commit: elastic/x-pack-elasticsearch@4a36f0c2be
2017-09-14 07:43:19 +02:00
Jason Tedor 4f3e740ba8 Refactor bootstrap check results and error messages
This commit refactors the X-Pack bootstrap checks to respond to a change
in core Elasticsearch where the checks now return a single result
object.

Relates elastic/x-pack-elasticsearch#2495

Original commit: elastic/x-pack-elasticsearch@230b050529
2017-09-13 21:30:51 -04:00
lcawley 9ea36ef771 [DOCS] Added tip in users command
Original commit: elastic/x-pack-elasticsearch@3fb4e1819c
2017-09-13 17:21:15 -07:00
Lisa Cawley 89d6c7e01e [DOCS] Create reference for users command (elastic/x-pack-elasticsearch#2480)
Original commit: elastic/x-pack-elasticsearch@d0afe8a20d
2017-09-13 17:16:06 -07:00
Aaron Bull Schaefer 447f224677 Add releaseTest option to CI script (elastic/x-pack-elasticsearch#2482)
This option runs a normal check but with `-Dsnapshot=false` (the flag
used to indicate a release build).

Related to https://github.com/elastic/infra/issues/2759 and
https://github.com/elastic/infra/issues/2739 from the ES side.

Original commit: elastic/x-pack-elasticsearch@e674e68905
2017-09-13 16:01:21 -07:00
Nik Everett f15666f82e Fix links in deprecation checks (elastic/x-pack-elasticsearch#2490)
Some links must have moved since we wrote the tests and released
5.6.0.

relates elastic/x-pack-elasticsearch#2488

Original commit: elastic/x-pack-elasticsearch@ebceee7f3d
2017-09-13 16:32:40 -04:00
Simon Willnauer 01a921a8e3 Accept BootstrapContext in xpack (elastic/x-pack-elasticsearch#2486)
This is the xpack side of elastic/elasticsearch#26628

Original commit: elastic/x-pack-elasticsearch@f6c0599ee2
2017-09-13 22:14:29 +02:00
Jay Modi f30e5c3fee Register the legacy truststore password setting for the PKI realm (elastic/x-pack-elasticsearch#2487)
After the addition of the secure settings in 5.6, the truststore.password setting for the PKI realm
was no longer registered. This would cause new nodes to fail for customers that were upgrading and
had configured a PKI realm with a truststore. This change registers the setting and adds a test to
ensure a realm configuration with the old setting passes validation.

Relates elastic/support-dev-help#2505

Original commit: elastic/x-pack-elasticsearch@54da044a27
2017-09-13 13:11:54 -06:00
Simon Willnauer 0680e41f36 Prevent nodes from joining a non-TLS enabled cluster with a production license (elastic/x-pack-elasticsearch#2484)
This change prevents a node from joining a cluster with a production license (gold, platinum, standard) iff the cluster doesn't have TLS setup. This is mainly a BWC oriented change that prevents joining old 5.x clusters without a TLS setup.

Relates to elastic/x-pack-elasticsearch#2463

Original commit: elastic/x-pack-elasticsearch@21f5a58472
2017-09-13 20:40:35 +02:00
Dimitris Athanasiou 99ffbb1cd6 [ML] Add random offset to the maintenance task execution time (elastic/x-pack-elasticsearch#2483)
Currently the maintenance task is executed at 30 minutes past
midnight of each day. In the scenario where multiple clusters
are running on the same hardware infrastructure they all will
be running at the same time, competing for resources.

This commit changes this by adding a random offset to the
execution time which ranges from 0 to 119 minutes. The
minute granularity means that different offsets give at
least 1 minute for the maintenance task to end. Moreover,
the 2 hour window gives enough slots for different offsets
to occur and remains within what most people would think
as "middle of the night".

relates elastic/x-pack-elasticsearch#2273

Original commit: elastic/x-pack-elasticsearch@b538923aca
2017-09-13 14:53:44 +01:00
Dimitris Athanasiou e4753656bc [ML] Randomize default datafeed query delay (elastic/x-pack-elasticsearch#2475)
Changes the default query delay from 1m to a random
value between 1m and 2m. The motivation is to avoid
having multiple jobs firing their searches at the same
time which may potentially lead to increased load
on the machine.

relates elastic/x-pack-elasticsearch#2472

Original commit: elastic/x-pack-elasticsearch@3224e836fa
2017-09-13 09:12:39 +01:00
David Roberts 2e3aca414b [ML] Remove obsolete dynamic mapper setting (elastic/x-pack-elasticsearch#2477)
Since 5.6 we have only used one type per ML index, so this setting
is not necessary.

Original commit: elastic/x-pack-elasticsearch@64c434adec
2017-09-12 19:25:30 +01:00
jaymode 940e699e31 Remove index.mapper.dynamic from watcher and security templates
index.mapper.dynamic should not be used for 6.0+ indices, so this commit removes it from the
templates used by security and watcher.

Relates elastic/elasticsearch#25734

Original commit: elastic/x-pack-elasticsearch@766eebe660
2017-09-12 11:12:25 -06:00
Simon Willnauer 2f5aeb6c6f Remove token passphrase setting (elastic/x-pack-elasticsearch#2318)
This change removes `xpack.security.authc.token.passphrase` entirely since from
6.0 onwards we use randomly generated keys by the master there is no need for
this setting anymore. This setting will be deprecated from 6.0 onwards.

Original commit: elastic/x-pack-elasticsearch@37ba90359e
2017-09-12 15:34:41 +02:00
Alexander Reelsen c3f3ae5391 Watcher: Remove all traces from execution on master node (elastic/x-pack-elasticsearch#2383)
As there are no master node operations anymore.

* TransportActions are regular Actions now
* Watcher requests are now ActionRequests, no MasterNodeRequests anymore
* REST spec does not contain master node timeout parameters anymore
* WatcherLifeCycleService does not have a check anymore if watcher is able to run distributed, this will be a given in 7.0
* Some serialization BWC checks against version 5 have been removed

Original commit: elastic/x-pack-elasticsearch@4607dd538c
2017-09-12 15:05:26 +02:00
Dimitris Athanasiou e4882b36b7 [ML] Ensure datafeed runs on time (elastic/x-pack-elasticsearch#2471)
The datafeed runs on frequency-aligned intervals behind
query_delay. Currently, when a real-time run is triggered,
we subtract query_delay from now and then we take the aligned
interval. This results into running frequency + query_delay
behind now. The fix involves simply adding the query_delay
into the time real-time runs occur.

Relates elastic/x-pack-elasticsearch#2426

Original commit: elastic/x-pack-elasticsearch@61ceaaca8f
2017-09-12 13:24:55 +01:00
David Roberts 59d94eba40 [TEST] Mute failing test: CategorizationIT testBasicCategorization
See elastic/machine-learning-cpp#279

Original commit: elastic/x-pack-elasticsearch@03f0c307b7
2017-09-12 11:04:19 +01:00
David Roberts 1500074cb2 [ML] Add method to find the established memory use for a job (elastic/x-pack-elasticsearch#2449)
"Established" memory use will be one of the building blocks for smarter node
allocation.

In order for a job to be considered to have established memory usage it must:
- Have generated at least 20 buckets of results
- Have generated at least one model size stats document
- Have low variability of model bytes in model size stats documents in the
  time period covered by the last 20 buckets, which is defined as having a
  coefficient of variation of no more than 0.1

Relates elastic/x-pack-elasticsearch#546

Original commit: elastic/x-pack-elasticsearch@5032eb01d8
2017-09-12 10:25:53 +01:00
lcawley a979f33252 [DOCS] Added Dev Tools settings
Original commit: elastic/x-pack-elasticsearch@1538d28c21
2017-09-11 16:07:52 -07:00
Lisa Cawley 26148c91fe [DOCS] Format X-Pack migration APIs (elastic/x-pack-elasticsearch#2378)
Original commit: elastic/x-pack-elasticsearch@502d77b975
2017-09-11 14:02:23 -07:00
Lisa Cawley 90c6b93897 Add troubleshooting for ML shared index problem (elastic/x-pack-elasticsearch#2347)
* [DOCS] Add troubleshooting for ML shared index problem

* [DOCS] Address troubleshooting feedback

Original commit: elastic/x-pack-elasticsearch@3116524177
2017-09-11 08:39:14 -07:00
David Kyle bad65b4186 [ML] Add setting for job max model memory limit (elastic/x-pack-elasticsearch#2460)
* Add setting for job max model memory limit

* Address review comments

Original commit: elastic/x-pack-elasticsearch@5cec3a1abf
2017-09-11 14:53:46 +01:00
Jay Modi aaa0510821 Run core's full cluster restart tests with x-pack (elastic/x-pack-elasticsearch#2433)
This change pulls in the o.e.u.FullClusterRestartIT class from core and runs it as part of the
x-pack full cluster restart tests.

Relates elastic/x-pack-elasticsearch#1629

Original commit: elastic/x-pack-elasticsearch@87da59485f
2017-09-08 13:33:33 -06:00
Lisa Cawley 27a8041804 [DOCS] CCS no longer needs local *:* permission (elastic/x-pack-elasticsearch#2445)
Original commit: elastic/x-pack-elasticsearch@fb7f6eaeb2
2017-09-08 08:41:32 -07:00
David Kyle 51603620ee Mute ML rolling upgrade tests. Awaits fix elastic/x-pack-elasticsearch#1760
Original commit: elastic/x-pack-elasticsearch@deaf060818
2017-09-07 14:25:52 +01:00
Ryan Ernst 53294f217c Use versionless alias to ES rest client codebase (elastic/x-pack-elasticsearch#2441)
This is the xpack side of
https://github.com/elastic/elasticsearch/pull/26521

Original commit: elastic/x-pack-elasticsearch@b650e9e433
2017-09-06 18:58:17 -07:00
Lisa Cawley e4a008f9ee [DOCS] Update passphrase security setting (elastic/x-pack-elasticsearch#2431)
Original commit: elastic/x-pack-elasticsearch@8834d64f10
2017-09-06 08:13:59 -07:00
Lisa Cawley 0cd24a9283 [DOCS] Added kibana_dashboard_only_user role (elastic/x-pack-elasticsearch#2427)
Original commit: elastic/x-pack-elasticsearch@e6ab2238eb
2017-09-05 10:40:58 -07:00
Tim Brooks 84a47e2690 Disable `HipChatServiceTests` integration tests
This is related to elastic/x-pack-elasticsearch#2429. These tests are currently disabled due to
calls to the hipchat integration api failing. There is an open infra
issue for this (elastic/infra#2726).

Original commit: elastic/x-pack-elasticsearch@4aa9fe0387
2017-09-05 11:13:39 -06:00
David Roberts c73d70491a [TEST] Fix error if named pipe already connected (elastic/x-pack-elasticsearch#2423)
On Windows a named pipe server must call ConnectNamedPipe() before using
a named pipe.  However, if the client has already connected then this
function returns a failure code, with detailed error code
ERROR_PIPE_CONNECTED.  The server must check for this, as it means the
connection will work fine.  The Java test that emulates what the C++
would do in production did not have this logic.

This was purely a test problem.  The C++ code used in production already
does the right thing.

relates elastic/x-pack-elasticsearch#2359

Original commit: elastic/x-pack-elasticsearch@e162887f28
2017-09-05 13:39:22 +01:00
David Roberts 500b4ac6b9 [TEST] Improve ML security tests (elastic/x-pack-elasticsearch#2417)
The changes made for elastic/x-pack-elasticsearch#2369 showed that the ML security tests were seriously
weakened by the decision to grant many "minimal" privileges to all users
involved in the tests.  A better solution is to override the auth header
such that a superuser runs setup actions and assertions that work by
querying raw documents in ways that an end user wouldn't.  Then the ML
endpoints can be called with the privileges provided by the ML roles and
nothing else.

Original commit: elastic/x-pack-elasticsearch@4de42d9e54
2017-09-05 10:49:41 +01:00
Jason Tedor d9dce1afcd Fix typo in CI script
This commit fixes a typo in the CI script, the command-line flag for
setting tests.badapples had a typo.

Original commit: elastic/x-pack-elasticsearch@c67a620485
2017-09-03 18:38:40 -04:00
Jason Tedor bf68b5a907 Add intake option to CI script
This commit adds an option for running the intake build from the CI
scripts. The intake build differs from check in that it runs compilation
and pre-commit checks before running tests so that common build failures
causes will happen earlier than otherwise.

Relates elastic/x-pack-elasticsearch#2420

Original commit: elastic/x-pack-elasticsearch@b179ce1087
2017-09-03 16:43:15 -04:00
Alexander Reelsen cd5e001ca6 Watcher: Only load active watches on load (elastic/x-pack-elasticsearch#2408)
When watcher is loading it must only load the watches
which are active instead of all possible watches.

This loading happens on start up as well as when shards
relocate.

Original commit: elastic/x-pack-elasticsearch@29df56b99d
2017-09-01 16:20:42 +02:00