Commit Graph

7713 Commits

Author SHA1 Message Date
Chris Earle 309adaf38e [Monitoring] Disable Monitoring Collection by Default (elastic/x-pack-elasticsearch#3962)
This adds a new setting, `xpack.monitoring.collection.enabled`, and
disables it by default (`false`).

Original commit: elastic/x-pack-elasticsearch@4b3a5a1161
2018-02-21 23:10:20 -05:00
Tim Vernum 828387a25a Add more info logging when loading roles.yml (elastic/x-pack-elasticsearch#3992)
Although not frequently used in production, we make heavy use of the
FileRolesStore within integration tests. This change adds a little bit
more logging at INFO level when the roles.yml file is (re)loaded.

Original commit: elastic/x-pack-elasticsearch@bbacd46e28
2018-02-22 11:10:27 +11:00
Lisa Cawley 1eca36bda9 [DOCS] Clarified model snapshot retention (elastic/x-pack-elasticsearch#4000)
Original commit: elastic/x-pack-elasticsearch@f1bdf5454d
2018-02-21 08:58:17 -08:00
David Kyle 92a9fc8b48 Mute failing ML datafeed docs code snippet
Original commit: elastic/x-pack-elasticsearch@9cfea037bc
2018-02-21 10:03:20 +00:00
Alexander Reelsen 90c360d9d0 Watcher: Replace YAML tests needing to wait for watch execution (elastic/x-pack-elasticsearch#3995)
In order to check for the REST tests if triggering of watches with
security enabled works as expected, we have to add a watch and wait for
its background execution. In the REST tests the only wait is to wait for
this with a timeout. If the timeout is reached but the watch has not
been executed yet, the test will fail.

This commit replaces the YAML with a java based REST test, so that
helper methods like assertBusy() can be used and waiting for a watch to
be executed now works as expected.

relates elastic/x-pack-elasticsearch#3753

Original commit: elastic/x-pack-elasticsearch@fc39636ef7
2018-02-21 09:58:11 +01:00
Alexander Reelsen 0bf354eb38 Watcher: Fix hipchat message failure serialization (elastic/x-pack-elasticsearch#3939)
The current toXContent serialization of a failed hipchat message writes
the same field called status twice and thus cannot be stored in the
watch history.

This commit ensures the field gets only written once.

relates elastic/x-pack-elasticsearch#3919

Original commit: elastic/x-pack-elasticsearch@fb499e8055
2018-02-21 09:38:53 +01:00
lcawley 0e2a39603e [DOCS] Fixed ml.machine_memory example testing
Original commit: elastic/x-pack-elasticsearch@d0fa44ab20
2018-02-20 18:36:56 -08:00
Lisa Cawley e9b4a2d063 [DOCS] Enabled code snippet testing for more ML APIs (elastic/x-pack-elasticsearch#3990)
Original commit: elastic/x-pack-elasticsearch@1b631adff6
2018-02-20 11:08:37 -08:00
Jay Modi e3e82ccb2d Security: IndexAuditTrail updates mappings when indexing locally (elastic/x-pack-elasticsearch#3944)
This change makes the IndexAuditTrail update mappings for the current
index even when indexing locally. This is needed in a couple of
scenarios; the first being upgrades from 5x to 6x where the document
type was changed and the second is when upgrading minor versions and
a new field has been introduced. A new field was introduced in 6.2.

Additionally, writing tests for the rolling upgrade revealed a bug
where the actual node name was never being set as it was using the name
key instead of the node.name key.

relates elastic/x-pack-elasticsearch#3901
relates elastic/x-pack-elasticsearch#3718

Original commit: elastic/x-pack-elasticsearch@7374a62ff1
2018-02-20 11:09:56 -07:00
Jay Modi 0fc0034509 Test: remove outdated logic for file writing in security tests (elastic/x-pack-elasticsearch#3947)
This commit removes some outdated logic in the SecurityTestUtils class
for writing files and creating directories. A long, long time ago there
was a global cluster for tests that was used across suites and because
of this there were calls to delete files if they already existed. The
global cluster has been removed, so we no longer need the code that
deletes the content of a directory if it already exists.

Additionally, the file writing used in SecurityTestUtils did not use
atomic moves when possible and this commit changes the code such that a
temp file is written and we try to atomically move it to the correct
path; if atomic moves are not supported a regular move is performed.

relates elastic/x-pack-elasticsearch#3912

Original commit: elastic/x-pack-elasticsearch@973fcfe2e1
2018-02-20 09:34:17 -07:00
Lee Hinman 6728912c87 Handle removing log4j dependency from elasticsearch-core jar (elastic/x-pack-elasticsearch#3976)
This handles the removal of the log4j dependency from our elasticsearch-core
project. It was originally necessary only for our jar classpath checking. It is
now replaced by a `Consumer<String>`.

This is the x-pack side of https://github.com/elastic/elasticsearch/pull/28705

Original commit: elastic/x-pack-elasticsearch@1186a7eaf6
2018-02-20 09:15:56 -07:00
David Kyle e365a255c7 [ML] Filter non-ml attributes from data feed stats (elastic/x-pack-elasticsearch#3994)
Original commit: elastic/x-pack-elasticsearch@9cdc78c000
2018-02-20 12:22:28 +00:00
Alexander Reelsen c9d77d20fd Watcher: Never return credentials after watch creation... (elastic/x-pack-elasticsearch#3581)
... yet support updates. This commit introduces a few changes of how
watches are put.

The GET Watch API will never return credentials like basic auth
passwords, but a placeholder instead now. If the watcher is enabled to
encrypt sensitive settings, then the original encrypted value is
returned otherwise a "::es_redacted::" place holder.

There have been several Put Watch API changes.

The API now internally uses the Update API and versioning. This has
several implications. First if no version is supplied, we assume an
initial creation. This will work as before, however if a credential is
marked as redacted we will reject storing the watch, so users do not
accidentally store the wrong watch.

The watch xcontent parser now has an additional methods to tell the
caller if redacted passwords have been found. Based on this information
an error can be thrown.

If the user now wants to store a watch that contains a password marked
as redacted, this password will not be part of the toXContent
representation of the watch and in combinatination with update request
the existing password will be merged in. If the encrypted password is
supplied this one will be stored.

The serialization for GetWatchResponse/PutWatchRequest has changed.
The version checks for this will be put into the 6.x branch.

The Watcher UI now needs specify the version, when it wants to store a
watch. This also prevents last-write-wins scenarios and is the reason
why the put/get watch response now contains the internal version.

relates elastic/x-pack-elasticsearch#3089

Original commit: elastic/x-pack-elasticsearch@bb63be9f79
2018-02-20 10:09:27 +01:00
Tim Vernum 56c761f241 Log roles.yml changes in TokenAuthIntegTests
This test recently failed and it appears that the cause was that
roles.yml was deleted while the cluster was still running.
This turns on logging for the FileRolesStore so that changes to file
based roles can traced more easily.

Original commit: elastic/x-pack-elasticsearch@fbce5a8027
2018-02-20 17:19:39 +11:00
Tim Vernum c4582cdcd0 Additional settings for SAML NameID policy (elastic/x-pack-elasticsearch#3969)
* Additional settings for SAML NameID policy

We should not be populating SPNameQualifier by default as it is
intended to be used to specify an alternate SP EntityID rather than
our own. Some IdPs (ADFS) fail when presented with this value.

This commit
- makes the SPNameQualifier a setting that defaults to blank
- adds a setting for "AllowCreate"
- documents the above

Original commit: elastic/x-pack-elasticsearch@093557e88f
2018-02-20 13:51:42 +11:00
Lisa Cawley 64653e525a [DOCS] Identify informational ML properties (elastic/x-pack-elasticsearch#3773)
Original commit: elastic/x-pack-elasticsearch@cb310b360d
2018-02-19 11:48:09 -08:00
Lisa Cawley 530b709948 [DOCS] Add skip_time to Flush Jobs API (elastic/x-pack-elasticsearch#1955)
Original commit: elastic/x-pack-elasticsearch@352bd336d8
2018-02-19 11:04:12 -08:00
Lisa Cawley ecfd8892b1 [DOCS] Expanded conceptual information about jobs (elastic/x-pack-elasticsearch#3332)
Original commit: elastic/x-pack-elasticsearch@18b32bd7b0
2018-02-19 10:38:38 -08:00
Albert Zaharovits 5833ed296e Fix embarrassing NPE in audit trail.
relates elastic/x-pack-elasticsearch#3988

Original commit: elastic/x-pack-elasticsearch@72c16ea50f
2018-02-19 20:14:37 +02:00
Martijn van Groningen c621be626f test: index documents with the same type and increase root log level
Original commit: elastic/x-pack-elasticsearch@41153e52d4
2018-02-19 16:51:21 +01:00
markharwood dd9962cff2 [TEST] Add deterministic routing of docs to shards to prevent
randomisation of scores produced in results.
Reinstated failing assertions from elastic/x-pack-elasticsearch#918

Original commit: elastic/x-pack-elasticsearch@42318f1d8a
2018-02-19 15:46:51 +00:00
David Kyle 137488c9f9 [ML] Kill process after executor has shutdown (elastic/x-pack-elasticsearch#3958)
Original commit: elastic/x-pack-elasticsearch@4c58aa51bf
2018-02-19 12:11:22 +00:00
Tim Vernum 206f47707c Test: use trial license for qa/rolling-upgrade
This updates the rolling-upgrade cluster tests to use a trial license so
that security and watcher functionality can be tested. This is needed
as a result of changing the default license type to basic.

Relates elastic/x-pack-elasticsearch#3878
Relates elastic/x-pack-elasticsearch@f899d48

Original commit: elastic/x-pack-elasticsearch@b1cdd52600
2018-02-19 17:58:50 +11:00
jaymode 809d6ec08c Test: use trial license for full cluster restart tests
This updates the full cluster restart tests to use a trial license so
that security and watcher functionality can be tested. This is needed
as a result of changing the default license type to basic.

Relates elastic/x-pack-elasticsearch#3878
relates elastic/x-pack-elasticsearch#3975

Original commit: elastic/x-pack-elasticsearch@f899d48449
2018-02-16 13:05:54 -07:00
Tal Levy e22b9aed86 [TEST] awaitFix FullClusterRestartIT testWatcher&testSecurityNativeRealm
Original commit: elastic/x-pack-elasticsearch@5dae5e9b07
2018-02-16 11:49:42 -08:00
lcawley 5a445c82fb [DOCS] Fixed broken role mapping link
Original commit: elastic/x-pack-elasticsearch@97b8fae993
2018-02-16 09:59:36 -08:00
Lisa Cawley 3890875a88 [DOCS] Role Mapping API improvements (elastic/x-pack-elasticsearch#3951)
Original commit: elastic/x-pack-elasticsearch@d300c96c7a
2018-02-16 09:29:19 -08:00
Shaunak Kashyap 50be43bcd3 Adding new queue metric fields to mapping (elastic/x-pack-elasticsearch#3949)
This will allow these fields to be used in aggregations by Monitoring UI code

Original commit: elastic/x-pack-elasticsearch@6ce0aecb0c
2018-02-16 08:51:15 -08:00
Tanguy Leroux eb11cb96a1 [Tests] Ensure that Watcher templates are created before creating alias (elastic/x-pack-elasticsearch#3956)
* [Tests] Ensure that Watcher templates are created before creating alias

The createWatcherIndicesOrAliases() method randomly created watcher
indices with or without aliases. On slow machines it is possible that
the indices get created before WatcherIndexTemplateRegistry put the
templates. When it happens, the indices will be created without the
right index.format value, preventing the WatcherService to start (as
it checks the index.format before starting, in case an upgrade is
required)

relates elastic/x-pack-elasticsearch#3965

Original commit: elastic/x-pack-elasticsearch@82946a2be0
2018-02-16 11:48:42 +01:00
Lee Hinman fbd9d6c7f5 Call createParser with deprecation handler (elastic/x-pack-elasticsearch#3943)
There are only two remaining callers from x-pack that don't pass the deprecation
handler in.

Relates to elastic/x-pack-elasticsearch#28504

Original commit: elastic/x-pack-elasticsearch@cb542613fd
2018-02-15 15:38:25 -07:00
Costin Leau bc150c2521 SQL: Fix failing percentiles in JdbcCsvSpecIT
Original commit: elastic/x-pack-elasticsearch@da332afa9a
2018-02-15 22:34:13 +02:00
Costin Leau 749d0e3880 SQL: Regen grammar
Original commit: elastic/x-pack-elasticsearch@7cac9f5ea0
2018-02-15 22:34:12 +02:00
lcawley 90b1dec14b [DOCS] Fixed broken TLS link
Original commit: elastic/x-pack-elasticsearch@34ec651dc9
2018-02-15 12:32:37 -08:00
Costin Leau ae8152b55b SQL: add catalog and type arguments to SYS TABLES (elastic/x-pack-elasticsearch#3920)
Extend the SYS TABLES command to allow filtering by catalog and types
Add creation of EnumSet inside node tests
Handle EnumSet inside test for replaceChildren

Original commit: elastic/x-pack-elasticsearch@407ce7b006
2018-02-15 22:07:51 +02:00
Costin Leau 844c485e0d SQL: introduce SYS CATALOGS and TABLE TYPES commands (elastic/x-pack-elasticsearch#3941)
These are the equivalent of JDBC/ODBC getCatalogs and getTableTypes on
the server side.

Original commit: elastic/x-pack-elasticsearch@a6fa733680
2018-02-15 21:51:23 +02:00
Lisa Cawley 42f9a990d1 [DOCS] Split TLS instructions for HTTP and transport layers (elastic/x-pack-elasticsearch#3895)
Original commit: elastic/x-pack-elasticsearch@77fe30f7d3
2018-02-15 11:41:01 -08:00
Andy Bristol b042afdfdc [TEST] AwaitsFix JdbcCsvSpecIT
For elastic/x-pack-elasticsearch#3960

Original commit: elastic/x-pack-elasticsearch@c0c1ff7d07
2018-02-15 11:25:57 -08:00
Andy Bristol 3eeb38762a [TEST] packaging: enable trial license
x-pack used to generate a trial platinum license on startup, but now it
generates a basic license. Configure security tests to use a trial
license at startup

Original commit: elastic/x-pack-elasticsearch@29382fe5ce
2018-02-15 10:39:05 -08:00
David Roberts 54b0fca88e [TEST] Update strings in ML categorization test (elastic/x-pack-elasticsearch#3959)
This makes the samples match the C++ code

Original commit: elastic/x-pack-elasticsearch@54b71f9299
2018-02-15 16:41:16 +00:00
Colin Goodheart-Smithe 2994dd15be [TEST] mute failing watcher with security yaml test
Original commit: elastic/x-pack-elasticsearch@034396693e
2018-02-15 16:02:22 +00:00
Costin Leau f04036a1bd SQL: update percentile tests to reflect the t-digest upgrade
The commit to t-digest 3.2 elastic/x-pack-elasticsearch#28305, changed the results of percentiles
(more information https://github.com/elastic/elasticsearch/pull/28305
#issuecomment-362542140)
This commit updates the tests accordingly.

Original commit: elastic/x-pack-elasticsearch@f2351ce747
2018-02-15 17:44:03 +02:00
Tim Vernum bad812d920 [TEST] Silence LocalExporterIntegTests.testExport
See: elastic/x-pack-elasticsearch#3954, elastic/x-pack-elasticsearch#3910

Original commit: elastic/x-pack-elasticsearch@a298c21579
2018-02-15 18:04:20 +11:00
Tim Brooks d45f28339b Support configuring license type for gradle run (elastic/x-pack-elasticsearch#3942)
This commit is related to elastic/x-pack-elasticsearch#3877. It allows the passing of a system
property to gradle run indicating whether a `basic` or `trial` license
should be generated. If the `trial` license is indicated, platinum
features will be enabled.

Original commit: elastic/x-pack-elasticsearch@b2b89a4606
2018-02-14 19:34:55 -07:00
Chris Earle 176411e55e [Monitoring] Ignore _bulk if Collection is Disabled (elastic/x-pack-elasticsearch#3910)
This blocks incoming requests from Kibana, Logstash, and Beats when X-Pack monitoring is effectively disabled by setting `xpack.monitoring.collection.interval: -1`.

Original commit: elastic/x-pack-elasticsearch@016a9472f1
2018-02-14 16:24:03 -05:00
Tanguy Leroux 00dec27d9f SQL: Relax RemoteFailure parsing so that it also parses error metadata (elastic/x-pack-elasticsearch#3938)
The current parsing of errors in the RemoteFailure class is strict and
fails on any field that is not one of [caused_by, reason, root_cause,
stack_trace, type]. Sadly some exceptions adds more headers or
metadata when they are printed as XContent and such fields can't be
easily ignored at parsing time.

This commit changes the RemoteFailure.parseFromResponse() method so that
it parses errors using the same behavior as the high level rest client:
it parses any unknown field as a metadata if it's string value or an
array of string and just ignores and skips everything else without
throwing an exception.

Original commit: elastic/x-pack-elasticsearch@1348706807
2018-02-14 21:05:23 +01:00
Igor Motov 89f15ed9d3 Tests: suppress testIndexUpgradeInfoLicense on release builds (elastic/x-pack-elasticsearch#3930)
The testIndexUpgradeInfoLicense test is generating a license, which
fails on release builds because they are using production public keys
for license verification.

relates elastic/x-pack-elasticsearch#3921

Original commit: elastic/x-pack-elasticsearch@e1c37bfd91
2018-02-14 13:56:40 -05:00
Lisa Cawley 22c3060fd6 [DOCS] Add ML limitation and troubleshooting (elastic/x-pack-elasticsearch#3929)
Original commit: elastic/x-pack-elasticsearch@bfabcbb456
2018-02-14 08:58:29 -08:00
Lee Hinman affc9e3563 Move more XContent.createParser calls to non-deprecated version (elastic/x-pack-elasticsearch#3928)
Part 2

This moves more of the callers to pass in the DeprecationHandler.

Relates to elastic/x-pack-elasticsearch#28504

Original commit: elastic/x-pack-elasticsearch@e0b52bf050
2018-02-14 09:41:26 -07:00
Lee Hinman e8ef20b219 Move more XContent.createParser calls to non-deprecated version (elastic/x-pack-elasticsearch#3927)
* Move more XContent.createParser calls to non-deprecated version

This moves more of the callers to pass in the DeprecationHandler.

Relates to https://github.com/elastic/elasticsearch/issues/28504

* Mockito ಠ_ಠ

Original commit: elastic/x-pack-elasticsearch@81cb99ba52
2018-02-14 09:24:47 -07:00
David Kyle 0023572acb [ML] Fix test logic in MlJobIT::testDelete_multipleRequest (elastic/x-pack-elasticsearch#3937)
Original commit: elastic/x-pack-elasticsearch@a9a457e286
2018-02-14 15:09:31 +00:00