Commit Graph

7867 Commits

Author SHA1 Message Date
Dimitris Athanasiou e34cb2085f [ML] Also adjust bwc version in job serialization
Original commit: elastic/x-pack-elasticsearch@e6231ba6d3
2018-03-27 16:13:51 +01:00
Andy Bristol f3cd9a69a2 [test] packaging: renamed packaging configuration (elastic/x-pack-elasticsearch#4112)
For elastic/elasticsearch#26741

Original commit: elastic/x-pack-elasticsearch@401e9bb0e4
2018-03-26 13:43:29 -07:00
Dimitris Athanasiou afb6a06f61 [ML] Model snapshot min_version is now present since 7.0.0
Original commit: elastic/x-pack-elasticsearch@39d193461d
2018-03-26 17:09:11 +01:00
Chris Earle a600350d4c [Monitoring] Remove 202 responses in favor of 200 responses (elastic/x-pack-elasticsearch#4213)
This changes `_xpack/monitoring/_bulk` to fundamentally behave in the same
way as `_bulk` and never return 202 when data is ignored (something
`_bulk` cannot do). Instead, anyone interested will have to inspect the
returned response for the ignored flag.

Original commit: elastic/x-pack-elasticsearch@07254a006d
2018-03-26 11:36:04 -04:00
Alexander Reelsen 67badaadb0 Docs: Fix secure settings link
Original commit: elastic/x-pack-elasticsearch@f98a8dabc6
2018-03-26 15:32:27 +02:00
Alexander Reelsen c2764cef98 Docs: Fix deprecation notices and typo to build docs
Original commit: elastic/x-pack-elasticsearch@6e5504efd9
2018-03-26 14:25:42 +02:00
Jim Ferenczi 3a75435980 Fix IndexerUtilsTests that relies on indexed fields
This test creates doc values fields only but does not set the index options to none.
This commit fixes this discrepancy by adding an indexed point field for all doc values field.

relates elastic/x-pack-elasticsearch#4223

Original commit: elastic/x-pack-elasticsearch@8adab7c849
2018-03-26 13:37:18 +02:00
David Turner 8c8de0a774 Mute failing IndexerUtilsTests
Awaiting a fix of elastic/x-pack-elasticsearch#4223

Original commit: elastic/x-pack-elasticsearch@d385099719
2018-03-26 10:57:34 +01:00
Alexander Reelsen 6eeacf339c Build: Use environment variables for credentials (elastic/x-pack-elasticsearch#4058)
The credentials now get injected via environment variables, so that
external services can pull those.

As soon as the specified environment variables are set, the tests are run. No need to check for the @Network annotation

This also introduces new secret store settings for the secure settings in order to be sure to not leak them in the configuration files, that get dumped.

Relates elastic/x-pack-elasticsearch#3800

Original commit: elastic/x-pack-elasticsearch@a2cfb9cb86
2018-03-26 09:10:04 +02:00
Jason Tedor e66072c09f Enable security in packaging tests (elastic/x-pack-elasticsearch#4216)
Now that security is not enabled by default for a trial license, the
packaging tests are failing because they expect security to be
enabled. This commit adds enabling security in all instances started
during the packaging tests.

Original commit: elastic/x-pack-elasticsearch@9838393ecb
2018-03-24 15:36:05 -04:00
Tim Sullivan 05a0d6273c [Monitoring/Beats] Add new CPU fields, remove old CPU fields (elastic/x-pack-elasticsearch#3991)
* [Monitoring/Beats] Add new CPU fields, remove old CPU fields

* use long instead of double for cpu counters

* time => time.ms

Original commit: elastic/x-pack-elasticsearch@244b08a574
2018-03-23 16:19:40 -07:00
Dimitris Athanasiou 67c64a6dfd [ML] Return error when process cause has been killed (elastic/x-pack-elasticsearch#4211)
relates elastic/x-pack-elasticsearch#4210

Original commit: elastic/x-pack-elasticsearch@c5169328ee
2018-03-23 17:30:10 +00:00
Dimitris Athanasiou 5f219bd70f [ML][DOCS] Remove empty rules from docs
Original commit: elastic/x-pack-elasticsearch@dee88e1161
2018-03-23 12:31:36 +00:00
Dimitris Athanasiou c4ff5ad3ed [ML] Do not serialize rules when empty (elastic/x-pack-elasticsearch#4203)
Original commit: elastic/x-pack-elasticsearch@18d731cb35
2018-03-23 11:21:27 +00:00
Alexander Reelsen f6d318a782 Watcher: Prevent question mark in HttpClient with empty params (elastic/x-pack-elasticsearch#4206)
The HTTPClient in watcher always appended a question mark at the end of
an URL, regardless if parameters were used or not. This commit adds a
check to only pass valid parameters to the URI construction.

Original commit: elastic/x-pack-elasticsearch@184f8f441c
2018-03-23 12:16:34 +01:00
Costin Leau 264c88f445 SQL: Introduce CSV and TSV tabular output (elastic/x-pack-elasticsearch#4190)
When running SQL REST queries, a client can ask (through Accept header) for
the data to be returned in CSV or TSV format in addition to plain text,
json & co.

Original commit: elastic/x-pack-elasticsearch@12d87b3033
2018-03-23 12:23:00 +02:00
javanna d143d26bbd Adapt to RecoveryResponse change upstream
See https://github.com/elastic/elasticsearch/pull/28878 , RecoveryResponse doesn't accept the detailed boolean flag anymore in its constructor as it was unused.

Original commit: elastic/x-pack-elasticsearch@d96df3448e
2018-03-23 10:48:12 +01:00
Zachary Tong 8296dad5ec [TEST] disable Upgrade YAML tests
Tracking issue: elastic/x-pack-elasticsearch#4197

Original commit: elastic/x-pack-elasticsearch@cc2c7ad788
2018-03-22 18:39:27 +00:00
Alexander Reelsen 23b4368fe4 Docs: Fix encrypt watcher sensitive data documentation (elastic/x-pack-elasticsearch#4198)
The documentation mentions that the xpack.watcher.encrypt_sensitive_data
setting needs to be set in the keystore. This is wrong however, it needs
to be set in the standard elasticsearch yaml file.

relates elastic/x-pack-elasticsearch#4195


Original commit: elastic/x-pack-elasticsearch@613d63da85
2018-03-22 18:57:31 +01:00
David Kyle 179090c840 [ML] Unclutter failed job assignment explanations (elastic/x-pack-elasticsearch#4179)
Unclutter failed job assignment explanations

Original commit: elastic/x-pack-elasticsearch@1c3deebaac
2018-03-22 17:45:57 +00:00
Lee Hinman cd4a073bb5 Decouple XContentBuilder classes for strict building (elastic/x-pack-elasticsearch#4188)
This is the x-pack side of https://github.com/elastic/elasticsearch/pull/29197

Original commit: elastic/x-pack-elasticsearch@7237622dee
2018-03-22 08:19:08 -06:00
Dimitris Athanasiou 25b1a444cf [ML] Respect datafeed frequency when less or equal than query_delay (elastic/x-pack-elasticsearch#4168)
In order to deal with the most anticipated scenario, when datafeed
frequency is greater than the query_delay, we add the query_delay
to the frequency in order to determine the next time we will trigger
a real-time run. For example, if frequency is 10s and query_delay 1s,
we make sure to trigger the real-time run at a 10s + 1s = 11s offset.

However, this is not correct in the case the frequency is less or
equal to the query_delay. For example, if frequency is 1s and
query_delay is 10s. we would also end up triggering at 11s offset.
But the right behaviour would be to trigger every second while
ensuring we are searching for up to 10seconds ago.

This commit fixes this issue.

relates elastic/x-pack-elasticsearch#4167

Original commit: elastic/x-pack-elasticsearch@f605885167
2018-03-22 14:05:22 +00:00
Dimitris Athanasiou 42eae8b3be [ML] Notify job memory status changes (elastic/x-pack-elasticsearch#4187)
This commit adds job notifications when the memory status
changes. This ensures a job reaching its memory limit is
communicated more visibly to the user so action can be taken.

relates elastic/x-pack-elasticsearch#4173

Original commit: elastic/x-pack-elasticsearch@c7362bd4bc
2018-03-22 14:04:37 +00:00
Christoph Büscher d12ee3898d Add rest test for rank_eval with security (elastic/x-pack-elasticsearch#4185)
Original commit: elastic/x-pack-elasticsearch@5f9001ec65
2018-03-22 12:01:12 +01:00
Alexander Reelsen 0a525d4707 Watcher: Hide credentials/secret data of integrations in toXContent (elastic/x-pack-elasticsearch#4162)
If a user specifies an 'Authorization' header in an HTTPRequest we,
which might be needed instead of using HTTP Basic Auth due to using
Bearer Authentication, then in case of an failure, the request gets
logged including that Authorization header.

In addition, each implementation of a sent message for jira/hipchat/slack filters
out special fields when a HTTP request is written in case of a failed
response in order to not leak secret data.

Relates elastic/x-pack-elasticsearch#3800

Original commit: elastic/x-pack-elasticsearch@66efdd9b36
2018-03-22 11:00:27 +01:00
Jay Modi b0cc5afe8b Disable security for trial licenses by default (elastic/x-pack-elasticsearch#4120)
This change disables security for trial licenses unless security is
explicitly enabled in the settings. This is done to facilitate users
getting started and not having to deal with some of the complexities
involved in getting security configured. In order to do this and avoid
disabling security for existing users that have gold or platinum
licenses, we have to disable security after cluster formation so that
the license can be retrieved.

relates elastic/x-pack-elasticsearch#4078

Original commit: elastic/x-pack-elasticsearch@96bdb889fc
2018-03-21 21:09:44 -06:00
Tim Brooks e139b7ee31 Do not allow registering basic licenses (elastic/x-pack-elasticsearch#4087)
This is related to elastic/x-pack-elasticsearch#3877. This commit introduces validation that blocks
a user from registering a new basic license. Instead, the user should
use the /start_basic route to install a basic license.

Original commit: elastic/x-pack-elasticsearch@ebfcf6d8b2
2018-03-21 15:25:32 -04:00
Tim Brooks 2cb044849f Improve license expiration log line (elastic/x-pack-elasticsearch#4183)
This is related to elastic/x-pack-elasticsearch#266. It modifies the license expiration warning log
line to include "License expiration notice" on the first line so that it
is more easily seen when grepping logs.

Original commit: elastic/x-pack-elasticsearch@c75df8eafb
2018-03-21 15:09:51 -04:00
David Kyle aa566a55e3 [ML] Persist realtime datafeed job after lookback (elastic/x-pack-elasticsearch#4146)
* Wire in new Persist job action

* Persist after loopback if realtime job

Original commit: elastic/x-pack-elasticsearch@c24101e9cf
2018-03-21 16:31:59 +00:00
Dimitris Athanasiou 97703bf0da [ML] Refactor method to process model size stats (elastic/x-pack-elasticsearch#4175)
Original commit: elastic/x-pack-elasticsearch@6262ff33a9
2018-03-21 14:54:55 +00:00
Dimitris Athanasiou 506694c180 [ML] Add ML info endpoint providing defaults and limits (elastic/x-pack-elasticsearch#4154)
This commit adds an info API to ML. The API returns information
about default values and limits so that implementors can be
aware of such values and deal with them accordingly.

relates elastic/x-pack-elasticsearch#4135

Original commit: elastic/x-pack-elasticsearch@a969221032
2018-03-21 10:23:20 +00:00
Ryan Ernst e7e7e53fad Build: Remove public license key and inject it during release builds (elastic/x-pack-elasticsearch#3916)
This commit moves the dev key into core and renames to make it clear it
is for snapshots, and makes the production key a required parameter of
release builds.

Original commit: elastic/x-pack-elasticsearch@ea299bd5a2
2018-03-20 22:06:05 -07:00
Yogesh Gaikwad 57946a97df X-Pack-Security: Making setup-passwords work with protected keystores (elastic/x-pack-elasticsearch#3918)
Changes are done in SetupPasswordTool to load the keystore
and set security settings to load password protected keys in SSL configuration.
Check for the verification mode and appropriately use hostname verifier.
Close the keystore after setup-password is complete.
Unit test for CommandLineHttpClient

TODO: TLS documentation needs to be fixed, which will be taken up as a separate
fix due to documentation refactoring in progress.

relates elastic/x-pack-elasticsearch#3760 

Original commit: elastic/x-pack-elasticsearch@72e8666e21
2018-03-21 09:48:01 +11:00
Lee Hinman 7cb5378f82 Fix compilation after merging XContent changes
A backport missed these usages that had been removed.

Original commit: elastic/x-pack-elasticsearch@9e47562dd1
2018-03-20 12:49:41 -06:00
Lisa Cawley 9cfc759cfc [DOCS] Removed xpack installation comments (elastic/x-pack-elasticsearch#4153)
Original commit: elastic/x-pack-elasticsearch@1cd7982061
2018-03-20 11:26:34 -07:00
jaymode 1de3e45824 Test: use trial license for security core rest tests
This commit adds the setting to use a trial license in the core rest
tests with security project. This is needed so that security is
actually enabled during these tests.

Original commit: elastic/x-pack-elasticsearch@b08bdaeb6f
2018-03-20 12:21:07 -06:00
Lee Hinman 418eefbcf1 Remove BytesArray and BytesReference usage from XContentFactory (elastic/x-pack-elasticsearch#4158)
This is the x-pack side of https://github.com/elastic/elasticsearch/pull/29151

Original commit: elastic/x-pack-elasticsearch@be71338057
2018-03-20 11:52:31 -06:00
Tim Brooks 0a1e09c644 Remove date from rest resp for non-exp licenses (elastic/x-pack-elasticsearch#4149)
This is related to elastic/x-pack-elasticsearch#3877. This commit removes the expiration from the
json rest response for licenses that do not expire.

Original commit: elastic/x-pack-elasticsearch@f767e9d756
2018-03-20 12:19:07 -04:00
Alan Woodward c16e5f1f92 Watcher comparisons don't deal with NaN correctly (elastic/x-pack-elasticsearch#4133)
Aggregations may return `NaN`, and the comparison code would return `true` if this result was passed to a `gte` or `lte` condition.

Original commit: elastic/x-pack-elasticsearch@3b16ae6675
2018-03-20 15:55:59 +00:00
Tim Vernum 063ed78c42 Provide clearer errors if SAML is not licensed (elastic/x-pack-elasticsearch#4096)
SAML is only available on a Platinum license. If you try and use SAML
on a Gold license, then the error message is misleading - it gives the
equivalent of "cannot find saml realm".

This change adds a standard license error of
    "current license is non-compliant for [saml]"
if SAML rest actions are used when SAML is not licensed.

Original commit: elastic/x-pack-elasticsearch@7c0e26d58e
2018-03-20 18:54:51 +10:00
Yogesh Gaikwad 0de6376452 [DOCS] `xpack.ssl.client_authentication` setting does not apply to HTTP (elastic/x-pack-elasticsearch#4113)
Fix the documentation to mention the global setting does not apply to HTTP.

relates elastic/x-pack-elasticsearch#3413

Original commit: elastic/x-pack-elasticsearch@f9dc545b4a
2018-03-20 18:45:36 +11:00
Tim Vernum 7cd5e1d516 Check cluster heath before setup-passwords (elastic/x-pack-elasticsearch#4104)
Trying to setup passwords on a red cluster (or a cluster that cannot
reach a quorum) is generally not a good idea.

This commit:
- Adds a check for RED cluster status
- Prompts to confirm execution if the cluster is red
- Prints out the reason/type is an error response is received
- Increases the HTTP read timeout so that master election failures are
  reported correctly.

Original commit: elastic/x-pack-elasticsearch@4ffbda23db
2018-03-20 17:03:17 +10:00
Tim Vernum bc95ad80ce Add beats_system user to security (elastic/x-pack-elasticsearch#4103)
This creates a new "beats_system" user and role with the same
privileges as the existing "logstash_system" user/role.

The "beat_system" user is also added as a managed user within
the "setup-passwords" command.

Users who upgrade from an earlier version of Elasticsearch/X-Pack
will need to manually set a password for the beats_system user via
the change password API (or Kibana UI)

Original commit: elastic/x-pack-elasticsearch@6087d3a18e
2018-03-20 17:01:53 +10:00
Yogesh Gaikwad 10bb78c3d6 X-Pack-Security: Correct attribute_patterns regex in saml guide (elastic/x-pack-elasticsearch#4159)
X-Pack-Security: Correct attribute_patterns regex in saml guide

relates elastic/x-pack-elasticsearch#4157

Original commit: elastic/x-pack-elasticsearch@546d408e5a
2018-03-20 17:37:05 +11:00
Lisa Cawley 9d697f3c8b [DOCS] Add monitoring overview (elastic/x-pack-elasticsearch#4148)
Original commit: elastic/x-pack-elasticsearch@750ca4abc1
2018-03-19 16:48:07 -07:00
jaymode 0493f64e3f [Test] mute MonitoringIT#testMonitoringService
The MonitoringIT#testMonitoringService test fails occasionally in CI
due to a failed wait for documents condition, so this commit adds an
awaits fix.

See elastic/x-pack-elasticsearch#4150

Original commit: elastic/x-pack-elasticsearch@c9415cba63
2018-03-19 12:47:51 -06:00
Tim Brooks 762bacbc03 Update start_trial version compatibility (elastic/x-pack-elasticsearch#4122)
This is related to elastic/x-pack-elasticsearch#3877. With the backport of elastic/x-pack-elasticsearch#4102 the type
parameter for start_trial is now on 6.3. This commit updates the
version check during stream serialization / deserialization.

Original commit: elastic/x-pack-elasticsearch@75e8537ab6
2018-03-19 12:08:13 -04:00
Lisa Cawley b6f16a6b0d [DOCS] Add aggregation limitation details (elastic/x-pack-elasticsearch#4143)
Original commit: elastic/x-pack-elasticsearch@953103bcd7
2018-03-19 07:55:14 -07:00
Alexander Reelsen 955b0dddad Watcher: Ignore deactivated state with execute watch API (elastic/x-pack-elasticsearch#4054)
If a watch is not active, it should still be executed, if it is called
via the execute watch API.

This commit adds an additional method to the execution context to check
for this, which returns true for a manual execution context but checks
the watch status for the triggered one.

Original commit: elastic/x-pack-elasticsearch@18f3f9e84b
2018-03-19 07:31:06 -07:00
Jason Tedor a68051405a Fix Javadocs for MonitoringTempalteUtils
This commit fixes the Javadoc build for MonitoringTemplateUtils after
changes to core removed the string and bytes methods from
XContentBuilder.

Original commit: elastic/x-pack-elasticsearch@49f3b95b22
2018-03-19 10:19:50 -04:00