Commit Graph

1990 Commits

Author SHA1 Message Date
Zachary Tong 54539a1eb0 [Rollup] Make Rollup a Basic license feature (elastic/x-pack-elasticsearch#4246)
* Make Rollup a Basic license feature

Original commit: elastic/x-pack-elasticsearch@ef1ee98855
2018-03-30 06:23:08 -07:00
Dimitris Athanasiou 2aeff7713c [ML] Parsing objects from internal indices should be lenient (elastic/x-pack-elasticsearch#4256)
All ML objects stored in internal indices are currently parsed
strictly. This means unknown fields lead to parsing failures.
In turn, this means we cannot add new fields in any of those
objects (e.g. bucket, record, calendar, etc.) as it is not
backwards compatible.

This commit changes this by introducing lenient parsing when
it comes to reading those objects from the internal indices.
Note we still use strict parsing for the objects we read from
the c++ process, which is nice as it guarantees we would detect
if any of the fields were renamed on one side but not the other.

Also note that even though this is going in from 6.3, we cannot
introduce new fields until 7.0.

relates elastic/x-pack-elasticsearch#4232

Original commit: elastic/x-pack-elasticsearch@3f95d3c7b9
2018-03-29 17:32:57 +01:00
Jay Modi 1776905a2b Set order of audit log template to 1000 (elastic/x-pack-elasticsearch#4156)
This commit sets the order of the audit log template to 1000 instead of
using the max value. This will allow a user to define a template that
adds an alias.

Original commit: elastic/x-pack-elasticsearch@2267322755
2018-03-29 10:30:09 -06:00
Tim Vernum e69c5d4d48 Add secure_bind_password to LDAP realm (elastic/x-pack-elasticsearch#4192)
Adds a SecureSetting option for the "bind_password" in LDAP/AD realms
and deprecates the non-secure version.

LDAP bind passwords should now be configured with the setting
`xpack.security.authc.realms.REALM_NAME.secure_bind_password`
in the elasticsearch keystore.

Original commit: elastic/x-pack-elasticsearch@1a0cebd77e
2018-03-29 16:31:45 +10:00
Yogesh Gaikwad 01ab4782a1 SAML: Enhance tests to randomly select crypto algorithms (elastic/x-pack-elasticsearch#4217)
- Changes in CertUtils to add algorithm parameter to
  generateSignedCertificates
- Changes in Tests to randomly pick signature algorithms
- Changes in Tests to randomly pick encryption algorithms

relates elastic/x-pack-elasticsearch#3983

Original commit: elastic/x-pack-elasticsearch@d1b5f3a166
2018-03-29 12:36:40 +11:00
Tim Vernum 94b6f637a6 Improve error if Indices Permission is too complex (elastic/x-pack-elasticsearch#4239)
If a user has roles that grant access to a large number of disparate
index patterns, then the resulting Automaton can become large and
too costly to determinise. This happens rarely, and is usually a sign
of a poorly implemented security model, so we have no immediate plans
to change the implementation. However the resulting error message is
not clear and does not provide sufficient information for users to
resolve the underlying problem.

This commit catches the underlying exception and provides a more
specific error message, with DEBUG logging of the offending index
patterns.

Original commit: elastic/x-pack-elasticsearch@532be70efc
2018-03-29 10:55:48 +10:00
Albert Zaharovits 53436450c4 All logging audit settings updateable (elastic/x-pack-elasticsearch#4227)
All logging audit settings are update-able via cluster settings
update API (prefix.emit_node_host_address,
prefix.emit_node_host_name, prefix.emit_node_name, events.include,
events.exclude).

Original commit: elastic/x-pack-elasticsearch@96adbd0ae2
2018-03-28 21:46:28 +03:00
Dimitris Athanasiou 00c391602d [ML] Cap default model_memory_limit in the ML info API (elastic/x-pack-elasticsearch#4249)
Original commit: elastic/x-pack-elasticsearch@dc614ad370
2018-03-28 16:05:42 +01:00
Zachary Tong df88ba4ed7 [Rollup] Don't persist state if aborting
`doSaveState` can be invoked on different types of failure.  Some of
these failures are recoverable (e.g. search exception) which just cause
the job to reset until the next trigger time.  Other exceptions might
be caused by an Abort request.

Previously `doSaveState` assumed that the indexer state would be
INDEXING, STOPPED or STARTED and asserted that.  But if we are ABORTING
it failed the assertion, and in production would try to persist
that aborting state which is not needed (and may complicate matters later).

This commit removes the assertion and only tries to persist if we
are not aborting.  If we're aborting, we just invoke the next handler
which is likely an onFailure handler.

Relates to elastic/x-pack-elasticsearch#4243

Original commit: elastic/x-pack-elasticsearch@3643b7c0e4
2018-03-28 13:01:58 +00:00
Dimitris Athanasiou bcfc8442f9 [ML] Preview of datafeed with aggs should respect chunking config (elastic/x-pack-elasticsearch#4241)
While it makes sense to apply auto-chunking in order to limit
the time range of the search for previewing datafeeds without aggs,
the same is not the case when aggs are used. In contrary, we should
do the preview the same way it would be if the datafeed run, as this
can reveal problems with regard to the datafeed configuration.

In addition, by default datafeeds with aggs have a manual chunking config
that limits the cost of each search. So, setting the chunking to auto
in those cases may lead to the datafeed preview failing even though
actually running the datafeed would work fine.

Original commit: elastic/x-pack-elasticsearch@79e317efb2
2018-03-28 13:43:09 +01:00
Ioannis Kakavas febb46b702 [SAML] Saml metadata signing (elastic/x-pack-elasticsearch#4184)
Adds option to sign generated Service Provider SAML metadata
- Using a (possibly password protected) PEM encoded keypair
- Using a keypair stored in a (possibly password protected) PKCSelastic/x-pack-elasticsearch#12 keystore

Resolves elastic/x-pack-elasticsearch#3982


Original commit: elastic/x-pack-elasticsearch@7b806d76f8
2018-03-28 13:43:29 +03:00
David Roberts c63d32482f [ML] Avoid timeout if ML persistent task assignment fails on master node (elastic/x-pack-elasticsearch#4236)
The ML open_job and start_datafeed endpoints start persistent tasks and
wait for these to be successfully assigned before returning.  Since the
setup sequence is complex they do a "fast fail" validation step on the
coordinating node before the setup sequence.  However, this leads to the
possibility of the "fast fail" validation succeeding and the eventual
persistent task assignment failing due to other changes during the setup
sequence.  Previously when this happened the endpoints would time out,
which in the case of the open_job action takes 30 minutes by default.
The start_datafeed endpoint has a shorter default timeout of 20 seconds,
but in both cases the result of a timeout is an unfriendly HTTP 500
status.

This change adjusts the criteria used to wait for the persistent tasks to
be assigned to account for the possibility of assignment failure and, if
this happens, return an error identical to what the "fast fail"
validation would have returned.  Additionally in this case the unassigned
persistent task is cancelled, leaving the system in the same state as if
the "fast fail" validation had failed.

Original commit: elastic/x-pack-elasticsearch@16916cbc13
2018-03-28 10:06:14 +01:00
Albert Zaharovits b7515f03cf LdapUserSearch rebind with bind DN after user bind (elastic/x-pack-elasticsearch#4209)
Fixes an inconsistency bug in which `LdapSession`s built by
`LdapUserSearchSessionFactory` are different if the factory is
configured to use a connection pool or not. The bind status of the
connection, or the connection(s) from the pool, passed through to
the newly minted `LdapSession` are now identical. Connections are
bind to the bind_dn configuration entry in the realm config.

Original commit: elastic/x-pack-elasticsearch@094af063ea
2018-03-28 09:36:02 +03:00
Ioannis Kakavas d1ed4e0bff Adds missing SAML Realm Settings (elastic/x-pack-elasticsearch#4221)
Adds idp.use_single_logout and populate_user_metadata 
in the SAML Realm Settings Set.

Resolves elastic/x-pack-elasticsearch#4219

Original commit: elastic/x-pack-elasticsearch@360f1f744e
2018-03-28 09:20:28 +03:00
Lee Hinman b646abd12c Adjust to XContentBuilder decoupling (elastic/x-pack-elasticsearch#4212)
This is the x-pack side of https://github.com/elastic/elasticsearch/pull/29225
where some methods were renamed or take different arguments.

Original commit: elastic/x-pack-elasticsearch@525e118381
2018-03-27 12:58:26 -06:00
Zachary Tong 9cc33f4e29 [Rollup] Select best jobs then execute msearch-per-job (elastic/x-pack-elasticsearch#4152)
If there are multiple jobs that are all the "best" (e.g. share the
best interval) we have no way of knowing which is actually the best.
Unfortunately, we cannot just filter for all the jobs in a single
search because their doc_counts can potentially overlap.

To solve this, we execute an msearch-per-job so that the results
stay isolated.  When rewriting the response, we iteratively
unroll and reduce the independent msearch responses into a single
"working tree".  This allows us to intervene if there are
overlapping buckets and manually choose a doc_count.

Job selection is found by recursively descending through the aggregation
tree and independently pruning the list of valid job caps in each branch.
When a leaf node is reached in the branch, the remaining jobs are
sorted by "best'ness" (see comparator in RollupJobIdentifierUtils for the
implementation) and added to a global set of "best jobs". Once
all branches have been evaluated, the final set is returned to the
calling code.

Job "best'ness" is, briefly, the job(s) that have
 - The largest compatible date interval
 - Fewer and larger interval histograms
 - Fewer terms groups

Note: the final set of "best" jobs is not guaranteed to be minimal,
there may be redundant effort due to independent branches choosing
jobs that are subsets of other branches.

Related changes:
- We have to include the job's ID in the rollup doc's
hash, so that different jobs don't overwrite the same summary
document.
- Now that we iteratively reduce the agg tree, the agg framework
injects empty buckets while we're working.  In most cases this
is harmless, but for `avg` aggs the empty bucket is a SumAgg while
any unrolled versions are converted into AvgAggs... causing a cast
exception.  To get around this, avg's are renamed to
`{source_name}.value` to prevent a conflict
- The job filtering has been pushed up into a query filter, since it
applies to the entire msearch rather than just individual agg components
- We no longer add a filter agg clause about the date_histo's interval, because 
that is handled by the job validation and pruning.

Original commit: elastic/x-pack-elasticsearch@995be2a039
2018-03-27 10:33:59 -07:00
David Roberts f1a948bc54 [ML] More corrections to BWC version for model min version in job serialization
Original commit: elastic/x-pack-elasticsearch@408f35f784
2018-03-27 16:19:39 +01:00
Dimitris Athanasiou e34cb2085f [ML] Also adjust bwc version in job serialization
Original commit: elastic/x-pack-elasticsearch@e6231ba6d3
2018-03-27 16:13:51 +01:00
Dimitris Athanasiou afb6a06f61 [ML] Model snapshot min_version is now present since 7.0.0
Original commit: elastic/x-pack-elasticsearch@39d193461d
2018-03-26 17:09:11 +01:00
Chris Earle a600350d4c [Monitoring] Remove 202 responses in favor of 200 responses (elastic/x-pack-elasticsearch#4213)
This changes `_xpack/monitoring/_bulk` to fundamentally behave in the same
way as `_bulk` and never return 202 when data is ignored (something
`_bulk` cannot do). Instead, anyone interested will have to inspect the
returned response for the ignored flag.

Original commit: elastic/x-pack-elasticsearch@07254a006d
2018-03-26 11:36:04 -04:00
Jim Ferenczi 3a75435980 Fix IndexerUtilsTests that relies on indexed fields
This test creates doc values fields only but does not set the index options to none.
This commit fixes this discrepancy by adding an indexed point field for all doc values field.

relates elastic/x-pack-elasticsearch#4223

Original commit: elastic/x-pack-elasticsearch@8adab7c849
2018-03-26 13:37:18 +02:00
David Turner 8c8de0a774 Mute failing IndexerUtilsTests
Awaiting a fix of elastic/x-pack-elasticsearch#4223

Original commit: elastic/x-pack-elasticsearch@d385099719
2018-03-26 10:57:34 +01:00
Alexander Reelsen 6eeacf339c Build: Use environment variables for credentials (elastic/x-pack-elasticsearch#4058)
The credentials now get injected via environment variables, so that
external services can pull those.

As soon as the specified environment variables are set, the tests are run. No need to check for the @Network annotation

This also introduces new secret store settings for the secure settings in order to be sure to not leak them in the configuration files, that get dumped.

Relates elastic/x-pack-elasticsearch#3800

Original commit: elastic/x-pack-elasticsearch@a2cfb9cb86
2018-03-26 09:10:04 +02:00
Tim Sullivan 05a0d6273c [Monitoring/Beats] Add new CPU fields, remove old CPU fields (elastic/x-pack-elasticsearch#3991)
* [Monitoring/Beats] Add new CPU fields, remove old CPU fields

* use long instead of double for cpu counters

* time => time.ms

Original commit: elastic/x-pack-elasticsearch@244b08a574
2018-03-23 16:19:40 -07:00
Dimitris Athanasiou 67c64a6dfd [ML] Return error when process cause has been killed (elastic/x-pack-elasticsearch#4211)
relates elastic/x-pack-elasticsearch#4210

Original commit: elastic/x-pack-elasticsearch@c5169328ee
2018-03-23 17:30:10 +00:00
Dimitris Athanasiou c4ff5ad3ed [ML] Do not serialize rules when empty (elastic/x-pack-elasticsearch#4203)
Original commit: elastic/x-pack-elasticsearch@18d731cb35
2018-03-23 11:21:27 +00:00
Alexander Reelsen f6d318a782 Watcher: Prevent question mark in HttpClient with empty params (elastic/x-pack-elasticsearch#4206)
The HTTPClient in watcher always appended a question mark at the end of
an URL, regardless if parameters were used or not. This commit adds a
check to only pass valid parameters to the URI construction.

Original commit: elastic/x-pack-elasticsearch@184f8f441c
2018-03-23 12:16:34 +01:00
Costin Leau 264c88f445 SQL: Introduce CSV and TSV tabular output (elastic/x-pack-elasticsearch#4190)
When running SQL REST queries, a client can ask (through Accept header) for
the data to be returned in CSV or TSV format in addition to plain text,
json & co.

Original commit: elastic/x-pack-elasticsearch@12d87b3033
2018-03-23 12:23:00 +02:00
javanna d143d26bbd Adapt to RecoveryResponse change upstream
See https://github.com/elastic/elasticsearch/pull/28878 , RecoveryResponse doesn't accept the detailed boolean flag anymore in its constructor as it was unused.

Original commit: elastic/x-pack-elasticsearch@d96df3448e
2018-03-23 10:48:12 +01:00
Zachary Tong 8296dad5ec [TEST] disable Upgrade YAML tests
Tracking issue: elastic/x-pack-elasticsearch#4197

Original commit: elastic/x-pack-elasticsearch@cc2c7ad788
2018-03-22 18:39:27 +00:00
David Kyle 179090c840 [ML] Unclutter failed job assignment explanations (elastic/x-pack-elasticsearch#4179)
Unclutter failed job assignment explanations

Original commit: elastic/x-pack-elasticsearch@1c3deebaac
2018-03-22 17:45:57 +00:00
Lee Hinman cd4a073bb5 Decouple XContentBuilder classes for strict building (elastic/x-pack-elasticsearch#4188)
This is the x-pack side of https://github.com/elastic/elasticsearch/pull/29197

Original commit: elastic/x-pack-elasticsearch@7237622dee
2018-03-22 08:19:08 -06:00
Dimitris Athanasiou 25b1a444cf [ML] Respect datafeed frequency when less or equal than query_delay (elastic/x-pack-elasticsearch#4168)
In order to deal with the most anticipated scenario, when datafeed
frequency is greater than the query_delay, we add the query_delay
to the frequency in order to determine the next time we will trigger
a real-time run. For example, if frequency is 10s and query_delay 1s,
we make sure to trigger the real-time run at a 10s + 1s = 11s offset.

However, this is not correct in the case the frequency is less or
equal to the query_delay. For example, if frequency is 1s and
query_delay is 10s. we would also end up triggering at 11s offset.
But the right behaviour would be to trigger every second while
ensuring we are searching for up to 10seconds ago.

This commit fixes this issue.

relates elastic/x-pack-elasticsearch#4167

Original commit: elastic/x-pack-elasticsearch@f605885167
2018-03-22 14:05:22 +00:00
Dimitris Athanasiou 42eae8b3be [ML] Notify job memory status changes (elastic/x-pack-elasticsearch#4187)
This commit adds job notifications when the memory status
changes. This ensures a job reaching its memory limit is
communicated more visibly to the user so action can be taken.

relates elastic/x-pack-elasticsearch#4173

Original commit: elastic/x-pack-elasticsearch@c7362bd4bc
2018-03-22 14:04:37 +00:00
Alexander Reelsen 0a525d4707 Watcher: Hide credentials/secret data of integrations in toXContent (elastic/x-pack-elasticsearch#4162)
If a user specifies an 'Authorization' header in an HTTPRequest we,
which might be needed instead of using HTTP Basic Auth due to using
Bearer Authentication, then in case of an failure, the request gets
logged including that Authorization header.

In addition, each implementation of a sent message for jira/hipchat/slack filters
out special fields when a HTTP request is written in case of a failed
response in order to not leak secret data.

Relates elastic/x-pack-elasticsearch#3800

Original commit: elastic/x-pack-elasticsearch@66efdd9b36
2018-03-22 11:00:27 +01:00
Jay Modi b0cc5afe8b Disable security for trial licenses by default (elastic/x-pack-elasticsearch#4120)
This change disables security for trial licenses unless security is
explicitly enabled in the settings. This is done to facilitate users
getting started and not having to deal with some of the complexities
involved in getting security configured. In order to do this and avoid
disabling security for existing users that have gold or platinum
licenses, we have to disable security after cluster formation so that
the license can be retrieved.

relates elastic/x-pack-elasticsearch#4078

Original commit: elastic/x-pack-elasticsearch@96bdb889fc
2018-03-21 21:09:44 -06:00
Tim Brooks e139b7ee31 Do not allow registering basic licenses (elastic/x-pack-elasticsearch#4087)
This is related to elastic/x-pack-elasticsearch#3877. This commit introduces validation that blocks
a user from registering a new basic license. Instead, the user should
use the /start_basic route to install a basic license.

Original commit: elastic/x-pack-elasticsearch@ebfcf6d8b2
2018-03-21 15:25:32 -04:00
Tim Brooks 2cb044849f Improve license expiration log line (elastic/x-pack-elasticsearch#4183)
This is related to elastic/x-pack-elasticsearch#266. It modifies the license expiration warning log
line to include "License expiration notice" on the first line so that it
is more easily seen when grepping logs.

Original commit: elastic/x-pack-elasticsearch@c75df8eafb
2018-03-21 15:09:51 -04:00
David Kyle aa566a55e3 [ML] Persist realtime datafeed job after lookback (elastic/x-pack-elasticsearch#4146)
* Wire in new Persist job action

* Persist after loopback if realtime job

Original commit: elastic/x-pack-elasticsearch@c24101e9cf
2018-03-21 16:31:59 +00:00
Dimitris Athanasiou 97703bf0da [ML] Refactor method to process model size stats (elastic/x-pack-elasticsearch#4175)
Original commit: elastic/x-pack-elasticsearch@6262ff33a9
2018-03-21 14:54:55 +00:00
Dimitris Athanasiou 506694c180 [ML] Add ML info endpoint providing defaults and limits (elastic/x-pack-elasticsearch#4154)
This commit adds an info API to ML. The API returns information
about default values and limits so that implementors can be
aware of such values and deal with them accordingly.

relates elastic/x-pack-elasticsearch#4135

Original commit: elastic/x-pack-elasticsearch@a969221032
2018-03-21 10:23:20 +00:00
Ryan Ernst e7e7e53fad Build: Remove public license key and inject it during release builds (elastic/x-pack-elasticsearch#3916)
This commit moves the dev key into core and renames to make it clear it
is for snapshots, and makes the production key a required parameter of
release builds.

Original commit: elastic/x-pack-elasticsearch@ea299bd5a2
2018-03-20 22:06:05 -07:00
Yogesh Gaikwad 57946a97df X-Pack-Security: Making setup-passwords work with protected keystores (elastic/x-pack-elasticsearch#3918)
Changes are done in SetupPasswordTool to load the keystore
and set security settings to load password protected keys in SSL configuration.
Check for the verification mode and appropriately use hostname verifier.
Close the keystore after setup-password is complete.
Unit test for CommandLineHttpClient

TODO: TLS documentation needs to be fixed, which will be taken up as a separate
fix due to documentation refactoring in progress.

relates elastic/x-pack-elasticsearch#3760 

Original commit: elastic/x-pack-elasticsearch@72e8666e21
2018-03-21 09:48:01 +11:00
Lee Hinman 7cb5378f82 Fix compilation after merging XContent changes
A backport missed these usages that had been removed.

Original commit: elastic/x-pack-elasticsearch@9e47562dd1
2018-03-20 12:49:41 -06:00
Lee Hinman 418eefbcf1 Remove BytesArray and BytesReference usage from XContentFactory (elastic/x-pack-elasticsearch#4158)
This is the x-pack side of https://github.com/elastic/elasticsearch/pull/29151

Original commit: elastic/x-pack-elasticsearch@be71338057
2018-03-20 11:52:31 -06:00
Tim Brooks 0a1e09c644 Remove date from rest resp for non-exp licenses (elastic/x-pack-elasticsearch#4149)
This is related to elastic/x-pack-elasticsearch#3877. This commit removes the expiration from the
json rest response for licenses that do not expire.

Original commit: elastic/x-pack-elasticsearch@f767e9d756
2018-03-20 12:19:07 -04:00
Alan Woodward c16e5f1f92 Watcher comparisons don't deal with NaN correctly (elastic/x-pack-elasticsearch#4133)
Aggregations may return `NaN`, and the comparison code would return `true` if this result was passed to a `gte` or `lte` condition.

Original commit: elastic/x-pack-elasticsearch@3b16ae6675
2018-03-20 15:55:59 +00:00
Tim Vernum 063ed78c42 Provide clearer errors if SAML is not licensed (elastic/x-pack-elasticsearch#4096)
SAML is only available on a Platinum license. If you try and use SAML
on a Gold license, then the error message is misleading - it gives the
equivalent of "cannot find saml realm".

This change adds a standard license error of
    "current license is non-compliant for [saml]"
if SAML rest actions are used when SAML is not licensed.

Original commit: elastic/x-pack-elasticsearch@7c0e26d58e
2018-03-20 18:54:51 +10:00
Tim Vernum 7cd5e1d516 Check cluster heath before setup-passwords (elastic/x-pack-elasticsearch#4104)
Trying to setup passwords on a red cluster (or a cluster that cannot
reach a quorum) is generally not a good idea.

This commit:
- Adds a check for RED cluster status
- Prompts to confirm execution if the cluster is red
- Prints out the reason/type is an error response is received
- Increases the HTTP read timeout so that master election failures are
  reported correctly.

Original commit: elastic/x-pack-elasticsearch@4ffbda23db
2018-03-20 17:03:17 +10:00
Tim Vernum bc95ad80ce Add beats_system user to security (elastic/x-pack-elasticsearch#4103)
This creates a new "beats_system" user and role with the same
privileges as the existing "logstash_system" user/role.

The "beat_system" user is also added as a managed user within
the "setup-passwords" command.

Users who upgrade from an earlier version of Elasticsearch/X-Pack
will need to manually set a password for the beats_system user via
the change password API (or Kibana UI)

Original commit: elastic/x-pack-elasticsearch@6087d3a18e
2018-03-20 17:01:53 +10:00