Commit Graph

2610 Commits

Author SHA1 Message Date
Adrien Grand 86e56590a7 Revert "Disable CcrRetentionLeaseIT#testRetentionLeasesAreNotBeingRenewedAfterRecoveryCompletes."
This reverts commit 343039e200.
2019-04-18 11:31:00 +02:00
Adrien Grand 343039e200 Disable CcrRetentionLeaseIT#testRetentionLeasesAreNotBeingRenewedAfterRecoveryCompletes.
Relates #39331.
2019-04-18 11:29:11 +02:00
Gordon Brown 66366d0307
Extract template management from Watcher (#41169)
This commit extracts the template management from Watcher into an
abstract class, so that templates and lifecycle policies can be managed
in the same way across multiple plugins. This will be useful for SLM, as
well as potentially ILM and any other plugins which need to manage index
templates.
2019-04-17 13:42:36 -06:00
Zachary Tong 7e62ff2823 [Rollup] Validate timezones based on rules not string comparision (#36237)
The date_histogram internally converts obsolete timezones (such as
"Canada/Mountain") into their modern equivalent ("America/Edmonton").
But rollup just stored the TZ as provided by the user.

When checking the TZ for query validation we used a string comparison,
which would fail due to the date_histo's upgrading behavior.

Instead, we should convert both to a TimeZone object and check if their
rules are compatible.
2019-04-17 13:46:44 -04:00
Iana Bondarska e090176f17 [ML] Exclude analysis fields with core field names from anomaly results (#41093)
Added "_index", "_type", "_id" to list of reserved fields.

Closes #39406
2019-04-17 16:08:03 +01:00
David Kyle 1d2365f5b6 [ML-DataFrame] Refactorings and tidying (#41248)
Remove unnecessary generic params from SingleGroupSource
and unused code from the HLRC
2019-04-17 14:58:26 +01:00
David Kyle 711d2545aa [ML-DataFrame] Resolve random test failure using deterministic name (#41262) 2019-04-17 09:04:20 +01:00
Marios Trivyzas 3fd081528d
SQL: Allow current_date/time/timestamp to be also used as a function escape pattern (#41254)
CURRENT_DATE/CURRENT_TIME/CURRENT_TIMESTAMP can be used as SQL keywords
(without parentheses) and therefore there is a special rule in the
grammar to accommodate this.

Previously, this rule was also catching the parenthesised version of those functions too,
not allowing the {fn <functionName>()} to be used. E.g.:
{fn current_time(2)} or {fn current_timestamp()}

Now, the grammar rule catches only the keyword versions and all the parenthesised
versions go through the normal function resolution. As a consequence the validation
of the precision is moved from the parser lever (ExpressionBuilder) to the function
implementations.

Fixes: #41240
(cherry picked from commit bfbc9f140144b5a35aa29008b58bf58074419853)
2019-04-17 10:13:16 +03:00
Yogesh Gaikwad 0d1178fca6
put mapping authorization for alias with write-index and multiple read indices (#40834) (#41287)
When the same alias points to multiple indices we can write to only one index
with `is_write_index` value `true`. The special handling in case of the put
mapping request(to resolve authorized indices) has a check on indices size
for a concrete index. If multiple indices existed then it marked the request
as unauthorized.

The check has been modified to consider write index flag and only when the
requested index matches with the one with write index alias, the alias is considered
for authorization.

Closes #40831
2019-04-17 14:25:33 +10:00
Yogesh Gaikwad 6a552c05fe
Use alias name from rollover request to query indices stats (#40774) (#41284)
In `TransportRolloverAction` before doing rollover we resolve
source index name (write index) from the alias in the rollover request.
Before evaluating the conditions and executing rollover action, we
retrieve stats, but to do so we used the source index name
resolved from the alias instead of alias from the index.
This fails when the user is assigned a role with index privilege on the
alias instead of the concrete index. This commit fixes this by using
the alias from the request.
After this change, verified that when we retrieve all the stats (including write + read indexes)
we are considering only source index.

Closes #40771
2019-04-17 14:15:05 +10:00
Costin Leau 85912b89fe SQL: Fix LIMIT bug in agg sorting (#41258)
When specifying a limit over an agg sorting, the limit will be pushed
down to the grouping which affects the custom sorting. This commit fixes
that and restricts the limit only to sorting.

Fix #40984

(cherry picked from commit da3726528d9011b05c0677ece6d11558994eccd9)
2019-04-16 22:40:41 +03:00
Marios Trivyzas c4ffd75805 SQL: Translate MIN/MAX on keyword fields as FIRST/LAST (#41247)
Although the translation rule was implemented in the `Optimizer`,
the rule was not added in the list of rules to be executed.

Relates to #41195
Follows #37936


(cherry picked from commit f426a339b77af6008d41cc000c9199fe384e9269)
2019-04-16 20:57:23 +03:00
Hendrik Muhs 02247cc7df [ML-DataFrame] adapt page size on circuit breaker responses (#41149)
handle circuit breaker response and adapt page size to reduce memory pressure, reduce preview buckets to 100, initial page size to 500
2019-04-16 19:49:43 +02:00
Costin Leau 9bf8bd40ae SQL: Tweak pattern matching in SYS TABLES (#41243)
Yet another improvement to SYS TABLES on differentiating between table
types specified as '%' and '' while maintaining legacy support for null

Fix #40775

(cherry picked from commit 6dbca5edd335eb1da8e7825389a15e5fe45397d4)
2019-04-16 19:48:45 +03:00
Gordon Brown ec8709e831
Check allocation rules are cleared after ILM Shrink (#41170)
Adds some checks to make sure that the allocation rules that ILM adds
before a shrink are cleared after the shrink is complete
2019-04-16 09:25:51 -06:00
David Kyle 116167df55 [ML] Write header to autodetect before it is visible to other calls (#41085) 2019-04-16 13:51:29 +01:00
Shaunak Kashyap 750db02b54
Expand beats_system role privileges (#40876) (#41232)
Traditionally we have [recommended](https://www.elastic.co/guide/en/beats/filebeat/current/monitoring.html) that Beats send their monitoring data to the **production** Elasticsearch cluster. Beats do this by calling the `POST _monitoring/bulk` API. When Security is enabled this API call requires the `cluster:admin/xpack/monitoring/bulk` privilege. The built-in `beats_system` role has this privilege.

[Going forward](https://github.com/elastic/beats/pull/9260), Beats will be able to send their monitoring data directly to the **monitoring** Elasticsearch cluster. Beats will do this by calling the regular `POST _bulk` API. When Security is enabled this API call requires the `indices:data/write/bulk` privilege. Further, the call has to be able to create any indices that don't exist.

This PR expands the built-in `beats_system` role's privileges. Specifically, it adds index-level `write` and `create_index` privileges for `.monitoring-beats-*` indices. 

This will allow Beats users to continue using the `beats_system` role for the new direct monitoring route when Security is enabled.
2019-04-15 20:17:05 -07:00
Gordon Brown 7e59794ced
Log every use of ILM Move to Step API (#41171)
Usage of the ILM Move to Step API can result in some very odd
situations, and for diagnosing problems arising from these situations it
would be nice to have a record of when this API was called with what
parameters.

Also, adds a dedicated logger for TransportMoveToStepAction, 
rather than using the (deprecated) inherited one.
2019-04-15 16:20:37 -06:00
Martijn van Groningen f56b2ecb37
Remove xpack dependencies from qa rest modules (#41134) (7.x backport) (#41202)
This commit removes xpack dependencies of many xpack qa modules.
(for some qa modules this will require some more work)

The reason behind this change is that qa rest modules should not depend
on the x-pack plugins, because the plugins are an implementation detail and
the tests should only know about the rest interface and qa cluster that is
being tested.

Also some qa modules rely on xpack plugins and hlrc (which is a valid
dependency for rest qa tests) creates a cyclic dependency and this is
something that we should avoid.  Also Eclipse can't handle gradle cyclic
dependencies (see #41064).

* don't copy xpack-core's plugin property into the test resource of qa
modules. Otherwise installing security manager fails, because it tries
to find the XPackPlugin class.
2019-04-15 19:14:43 +02:00
David Kyle 2b539f8347 [ML DataFrame] Data Frame stop all (#41156)
Wild card support for the data frame stop API
2019-04-15 15:04:28 +01:00
David Roberts 3f00c29adb [ML] Allow xpack.ml.max_machine_memory_percent higher than 100% (#41193)
Values higher than 100% are now allowed to accommodate use
cases where swapping has been determined to be acceptable.
Anomaly detector jobs only use their full model memory
during background persistence, and this is deliberately
staggered, so with large numbers of jobs few will generally
be persisting state at the same time.  Settings higher than
available memory are only recommended for OEM type
situations where a wrapper tightly controls the types of
jobs that can be created, and each job alone is considerably
smaller than what each node can handle.
2019-04-15 14:37:46 +01:00
Ioannis Kakavas fe9442b05b
Add an OpenID Connect authentication realm (#40674) (#41178)
This commit adds an OpenID Connect authentication realm to
elasticsearch. Elasticsearch (with the assistance of kibana or
another web component) acts as an OpenID Connect Relying
Party and supports the Authorization Code Grant and Implicit
flows as described in http://ela.st/oidc-spec. It adds support
for consuming and verifying signed ID Tokens, both RP
initiated and 3rd party initiated Single Sign on and RP
initiated signle logout.
It also adds an OpenID Connect Provider in the idp-fixture to
be used for the associated integration tests.

This is a backport of #40674
2019-04-15 12:41:16 +03:00
Christoph Büscher 2980a6c70f Clarify some ToXContent implementations behaviour (#41000)
This change adds either ToXContentObject or ToXContentFragment to classes
directly implementing ToXContent currently. This helps in reasoning about
whether those implementations output full xcontent object or just fragments.

Relates to #16347
2019-04-15 09:42:08 +02:00
Gordon Brown c8bc4ab003
Improve Watcher test framework resiliency (#40658)
It is possible for the watches tracked by ScheduleTriggerEngineMock to
get out of sync with the Watches in the ScheduleTriggerEngine
production code, which can lead to watches failing to run.

This commit:

1. Changes TimeWarp to try to run the watch on all schedulers, rather than stopping after one which claims to have the watch registered. This reduces the impact of desynchronization between the mocking code and the backing production code.
2. Makes ScheduleTriggerEngineMock respect pauses of execution again. This is necessary to prevent duplicate watch invocations due to the above change.
3. Tweaks how watches are registered in ScheduleTriggerEngineMock to prevent race conditions due to concurrent modification.
4. Tweaks WatcherConcreteIndexTests to use TimeWarp instead of waiting for watches to be triggered, as TimeWarp is more reliable and accomplishes the same goal.
2019-04-12 16:53:50 -06:00
Yogesh Gaikwad 47ba45732d
Find and use non local IPv4 address while testing IP filtering (#40234) (#41141)
For pattern "n:localhost" PatternRule#isLocalhost() matches
any local address, loopback address.
[Note: I think for "localhost" this should not consider IP address
as a match when they are bound to network interfaces. It should just
be loopback address check unless the intent is to match all local addresses.
This class is adopted from Netty3 and I am not sure if this is intended
behavior or maybe I am missing something]

For now I have fixed this assuming the PatternRule#isLocalhost check is
correct by avoiding use of local address to check address denied.

Closes #40194
2019-04-13 04:37:25 +10:00
Lee Hinman 7b190609ab
(7.x) Use environment settings instead of state settings for Watcher config (#41158)
Backport of (#41087)

* Use environment settings instead of state settings for Watcher config

Prior to this we used the settings from cluster state to see whether ILM was
enabled of disabled, however, these settings don't accurately reflect the
`xpack.ilm.enabled` setting in `elasticsearch.yml`.

This commit changes to using the `Environment` settings, which correctly reflect
the ILM enabled setting.

Resolves #41042
2019-04-12 12:15:44 -06:00
Gordon Brown ef310886a7
Add deprecation check for ILM poll interval <1s (#41096)
ILM poll intervals of less than 1 second will not be allowed, so add a
deprecation check for that.

Even though I'm pretty sure zero production clusters will do this, it's
best to be thorough.
2019-04-12 10:43:04 -06:00
Nik Everett c379206c1e
Fix some documentation urls in rest-api-spec (#40618) (#41145)
Fixes some documentation urls in the rest-api-spec. Some of these URLs
pointed to 404s and a few others pointed to deprecated documentation
when we have better documentation now. I'm not consistent about `master`
vs `current` because we're not consistent in other places and I think we
should solve all of those at once with something a little more
automatic.
2019-04-12 10:11:14 -04:00
Hendrik Muhs 3df6798c4c Rollup/DataFrame: disallow partial results (#41114)
disallow partial results in rollup and data frame, after this change the client throws an error directly
replacing the previous runtime exception thrown, allowing better error handling in implementations.
2019-04-12 07:31:22 +02:00
Benjamin Trent 05cf53934a
[ML] checking if p-tasks metadata is null before updating state (#41091) (#41123)
* [ML] checking if p-tasks metadata is null before updating state

* Adding test that validates fix

* removing debug println
2019-04-11 13:54:41 -05:00
Benjamin Trent 9e32e36799
[ML] fixing test related to #40963 (#41074) (#41116) 2019-04-11 11:19:56 -05:00
Armin Braun 233df6b73b
Make Transport Shard Bulk Action Async (#39793) (#41112)
This is a dependency of #39504

Motivation:
By refactoring `TransportShardBulkAction#shardOperationOnPrimary` to async, we enable using `DeterministicTaskQueue` based tests to run indexing operations. This was previously impossible since we were blocking on the `write` thread until the `update` thread finished the mapping update.
With this change, the mapping update will trigger a new task in the `write` queue instead.
This change significantly enhances the amount of coverage we get from `SnapshotResiliencyTests` (and other potential future tests) when it comes to tracking down concurrency issues with distributed state machines.

The logical change is effectively all in `TransportShardBulkAction`, the rest of the changes is then simply mechanically moving the caller code and tests to being async and passing the `ActionListener` down.

Since the move to async would've added more parameters to the `private static` steps in this logic, I decided to inline and dry up (between delete and update) the logic as much as I could instead of passing the listener + wait-consumer down through all of them.
2019-04-11 16:01:52 +02:00
Dimitris Athanasiou 736c7285d4
[ML] Fix scroll size comparison in DatafeedUpdate.isNoop (#41056) (#41079)
Note this does not affect users as the method is only used in tests.
2019-04-11 11:29:51 +03:00
Jason Tedor 24446ceae0
Add packaging to cluster stats response (#41048)
This commit adds a packaging_types field to the cluster stats response
that outlines the build flavors and types present in a cluster.
2019-04-10 13:47:19 -04:00
Hendrik Muhs c37b127a07 fix a timing issue: isFinished is used for a busy loop in testing, (#41055)
test: ensure state is persisted before the isFinished is changed

fixes #41046
2019-04-10 18:47:34 +02:00
Martijn van Groningen 1eff8976a8
Deprecate AbstractHlrc* and AbstractHlrcStreamable* base test classes (#41014)
* moved hlrc parsing tests from xpack to hlrc module and removed dependency on hlrc from xpack core

* deprecated old base test class

* added deprecated jdoc tag

* split test between xpack-core part and hlrc part

* added lang-mustache test dependency, this previously came in via
hlrc dependency.

* added hlrc dependency on a qa module

* duplicated ClusterPrivilegeName class in xpack-core, since x-pack
core no longer has a dependency on hlrc.

* replace ClusterPrivilegeName usages with string literals

* moved tests to dedicated to hlrc packages in order to remove Hlrc part from the name and make sure to use imports instead of full qualified class where possible

* remove ESTestCase. from method invocation and use method directly,
because these tests indirectly extend from ESTestCase
2019-04-10 16:29:17 +02:00
Chris Earle 64a05e522f
Properly handle Monitoring exporters all disabled (#40920) (#41043)
When monitoring exporters are all disabled, which must be done
explicitly, _and_ monitoring collection is enabled, then
any call to `_xpack/monitoring/_bulk` will create a task that
never closes _and_ ES collection will stop happening because
a semaphore is never marked as completed.

This also simplifies the async `ExportBulk` code by removing the
third step (second async step, `close`) entirely because it was
entirely unnecessary by both implementations.
2019-04-10 10:09:28 -04:00
Christoph Büscher d00d3f4afa Mute DataFrameTransformCheckpointTests#testGetBehind 2019-04-10 15:55:09 +02:00
Przemysław Witek f5014ace64
[ML] Add validation that rejects duplicate detectors in PutJobAction (#40967) (#41072)
* [ML] Add validation that rejects duplicate detectors in PutJobAction

Closes #39704

* Add YML integration test for duplicate detectors fix.

* Use "== false" comparison rather than "!" operator.

* Refine error message to sound more natural.

* Put job description in square brackets in the error message.

* Use the new validation in ValidateJobConfigAction.

* Exclude YML tests for new validation from permission tests.
2019-04-10 15:43:35 +02:00
Albert Zaharovits adf3393a4e
Deprecate permission over aliases (#38059) (#41060)
This PR generates deprecation log entries for each Role Descriptor,
used for building a Role, when the Role Descriptor grants more privileges
for an alias compared to an index that the alias points to. This is done in
preparation for the removal of the ability to define privileges over aliases.
There is one log entry for each "role descriptor name"-"alias name" pair.
On such a notice, the administrator is expected to modify the Role Descriptor
definition so that the name pattern for index names does not cover aliases.

Caveats:
* Role Descriptors that are not used in any authorization process,
either because they are not mapped to any user or the user they are mapped to
is not used by clients, are not be checked.
* Role Descriptors are merged when building the effective Role that is used in
the authorization process. Therefore some Role Descriptors can overlap others,
so even if one matches aliases in a deprecated way, and it is reported as such,
it is not at risk from the breaking behavior in the current role mapping configuration
and index-alias configuration. It is still reported because it is a best practice to
change its definition, or remove offending aliases.
2019-04-10 15:02:33 +03:00
Costin Leau 38f471ae1c SQL: Change schema calls to empty set (#41034)
As empty string has a certain meaning, the JDBC driver returns an empty
set instead for better client compatibility.

Fix #41028

(cherry picked from commit 4cbafa585b7a514eb6c156606dd516324cd3980a)
2019-04-10 13:14:12 +03:00
Hendrik Muhs f9018ab11b [ML-DataFrame] create checkpoints on every new run (#40725)
Use the checkpoint service to create a checkpoint on every new run. Expose checkpoints stats on _stats endpoint.
2019-04-10 09:14:11 +02:00
Andrei Stefan 0157ebf947 Increase the interval filtering for CURRENT_DATE/TODAY tests (#40999)
(cherry picked from commit a5911330aecee90da5401e468b9976f66a2556f0)
2019-04-10 08:35:09 +03:00
Jason Tedor bb6f060f74
Add log message to forget follower test
This commit adds a log message to help debug failures in a forget
follower test.
2019-04-09 23:33:29 -04:00
Julie Tibshirani 0d5f86a001 Mute RollupIndexerStateTests#testIndexing.
Tracked in #41046.
2019-04-09 17:17:04 -07:00
Julie Tibshirani 0702c72151 Mute DataFrameGetAndGetStatsIT#testGetPersistedStatsWithoutTask.
Tracked in #40963.
2019-04-09 16:39:16 -07:00
Julie Tibshirani 21c5d7e95f Mute CcrRetentionLeaseIT#testRetentionLeasesAreNotBeingRenewedAfterRecoveryCompletes.
Tracked in #39331.
2019-04-09 16:08:44 -07:00
Julie Tibshirani cbae617898 Mute IndexFollowingIT#testFollowIndex as we await a fix.
Tracked in #41037.
2019-04-09 14:56:37 -07:00
Mark Vieira 1287c7d91f
[Backport] Replace usages RandomizedTestingTask with built-in Gradle Test (#40978) (#40993)
* Replace usages RandomizedTestingTask with built-in Gradle Test (#40978)

This commit replaces the existing RandomizedTestingTask and supporting code with Gradle's built-in JUnit support via the Test task type. Additionally, the previous workaround to disable all tasks named "test" and create new unit testing tasks named "unitTest" has been removed such that the "test" task now runs unit tests as per the normal Gradle Java plugin conventions.

(cherry picked from commit 323f312bbc829a63056a79ebe45adced5099f6e6)

* Fix forking JVM runner

* Don't bump shadow plugin version
2019-04-09 11:52:50 -07:00
Jason Tedor ebba9393c1
Fix unsafe publication of invalid license enforcer (#40985)
The invalid license enforced is exposed to the cluster state update
thread (via the license state listener) before the constructor has
finished. This violates the JLS for safe publication of an object, and
means there is a concurrency bug lurking here. This commit addresses
this by avoiding publication of the invalid license enforcer before the
constructor has returned.
2019-04-09 13:51:37 -04:00