Commit Graph

47866 Commits

Author SHA1 Message Date
Lee Hinman a267df30fa Wait for snapshot completion in SLM snapshot invocation (#47051)
* Wait for snapshot completion in SLM snapshot invocation

This changes the snapshots internally invoked by SLM to wait for
completion. This allows us to capture more snapshotting failure
scenarios.

For example, previously a snapshot would be created and then registered
as a "success", however, the snapshot may have been aborted, or it may
have had a subset of its shards fail. These cases are now handled by
inspecting the response to the `CreateSnapshotRequest` and ensuring that
there are no failures. If any failures are present, the history store
now stores the action as a failure instead of a success.

Relates to #38461 and #43663
2019-09-25 14:25:22 -06:00
Lisa Cawley 287d96d1a1 [DOCS] Fix links to transform pages (#47134) 2019-09-25 13:09:47 -07:00
Mark Vieira d0495b8f93
Track enabled test task candidate class files as task input (#47054) 2019-09-25 11:20:02 -07:00
Gordon Brown a46eef9634
Change SLM stats format (#46991)
Using arrays of objects with embedded IDs is preferred for new APIs over
using entity IDs as JSON keys.  This commit changes the SLM stats API to
use the preferred format.
2019-09-25 11:32:08 -06:00
Colin Goodheart-Smithe 35f056661f
Updates 7.4.0 release notes 2019-09-25 18:10:09 +01:00
James Rodewig 9ddc99de23 [DOCS] Reformat clone index API docs (#46762) 2019-09-25 12:37:53 -04:00
James Rodewig 365aa30b7f
[DOCS] Remove support for `// CONSOLE` magic comment (#46936) (#47110)
#46180 added support for the `[source,console]`
language for snippets which should be tested.

This removes support for the `// CONSOLE` magic comment,
which serve a similar purpose.

Snippets that include the `// CONSOLE` magic comment will return
an exception.
2019-09-25 12:35:48 -04:00
Lisa Cawley d5f396fe55
[DOCS] Adds transforms to Elasticsearch book (#46846) (#47055) 2019-09-25 08:11:37 -07:00
Armin Braun 93fcd23da8
Fail Snapshot on Corrupted Metadata Blob (#47009) (#47096)
We should not be quietly ignoring a corrupted shard-level index-N
blob. Simply creating a new empty shard-level index-N and moving
on means that all snapshots of that shard show `SUCESS` as their
state at the repository root but are in fact broken.
This change at least makes it visible to the user that they can't
snapshot the given shard any more and forces the user to move on
to a new repository since the current one is broken and will not
allow snapshotting the inconsistent shard again.

Also, this change stops the delete action for shards with broken
index-N blobs instead of simply deleting all blobs in the path
containing the broken index-N. This prevents a temporarily
broken/missing index-N blob from corrupting all snapshots of
that shard.
2019-09-25 15:55:33 +02:00
Ioannis Kakavas 58d2bf7e09
Re-enable BWC tests (#47102)
Re-enable BWC tests now that #46534 has been backported to 7.x
2019-09-25 16:05:42 +03:00
István Zoltán Szabó 74fd21f0b0 [DOCS] Reformats ranking evaluation API (#46974)
* [DOCS] Reformats ranking evaluation API.
Co-Authored-By: James Rodewig <james.rodewig@elastic.co>
2019-09-25 15:01:10 +02:00
Nhat Nguyen 22575bd7e6 Remove isRecovering method from Engine (#47039)
We already prevent flushing in Engine if it's recovering. Hence, we can
remove the protection in IndexShard.
2019-09-25 08:58:08 -04:00
Armin Braun c4a166fc9a
Simplify SnapshotResiliencyTests (#46961) (#47108)
Simplify `SnapshotResiliencyTests` to more closely
match the structure of `AbstractCoordinatorTestCase` and allow for
future drying up between the two classes:

* Make the test cluster nodes a nested-class in the test cluster itself
* Remove the needless custom network disruption implementation and
  simply track disconnected node ids like `AbstractCoordinatorTestCase`
  does
2019-09-25 14:53:11 +02:00
István Zoltán Szabó 83365e94ba [DOCS] Reformat suggesters page. (#47010) 2019-09-25 14:42:16 +02:00
Yannick Welsch 9e17b78fee Mute second test in monitoring/bulk/10_basic
Relates #30101
2019-09-25 14:17:01 +02:00
Benjamin Trent 05fb7be571
[7.x] [ML][Inference] Feature pre-processing objects and functions (#46777) (#47040)
* [ML][Inference] Feature pre-processing objects and functions (#46777)

To support inference on pre-trained machine learning models, some basic feature encoding will be necessary. I am using a named object serialization approach so new encodings/pre-processing steps could be added in the future. 

This PR lays down the ground work for 3 basic encodings:

* HotOne
* Target Mean
* Frequency

More feature encodings or pre-processings could be added in the future:

* Handling missing columns
* Standardization
* Label encoding
* etc....

* fixing compilation for namedxcontent tests
2019-09-25 08:16:24 -04:00
Yannick Welsch 81cbd3fba4 Mute ClusterShardLimitIT.testIndexCreationOverLimitFromTemplate
Relates #47107
2019-09-25 14:03:08 +02:00
Yannick Welsch a4cecc54ab Mute monitoring/bulk/20_privileges
Relates #30101
2019-09-25 14:03:08 +02:00
David Turner ac920e8e64 Assert no exceptions during state application (#47090)
Today we log and swallow exceptions during cluster state application, but such
an exception should not occur. This commit adds assertions of this fact, and
updates the Javadocs to explain it.

Relates #47038
2019-09-25 12:32:51 +01:00
Yannick Welsch eb86d71edd Mute MlJobIT.testDeleteJob
Relates #45652
2019-09-25 12:53:09 +02:00
Yannick Welsch 7a5b5af171 Mute MlJobIT.testDeleteJobAsync
Relates #45652
2019-09-25 12:53:05 +02:00
Ioannis Kakavas f785c31531
File based role definition documentation additions (#46304) (#47085)
This commit clarifies and points out that the Role management UI and
the Role management API cannot be used to manage roles that are
defined in roles.yml and that file based role management is
intended to have a small administrative scope and not handle all
possible RBAC use cases.
2019-09-25 13:52:05 +03:00
Ioannis Kakavas 23bceaadf8
Handle RelayState in preparing a SAMLAuthN Request (#46534) (#47092)
This change allows for the caller of the `saml/prepare` API to pass
a `relay_state` parameter that will then be part of the redirect
URL in the response as the `RelayState` query parameter.

The SAML IdP is required to reflect back the value of that relay
state when sending a SAML Response. The caller of the APIs can
then, when receiving the SAML Response, read and consume the value
as it see fit.
2019-09-25 13:23:46 +03:00
Martijn van Groningen eef1ba3fad
Make ingest pipeline resolution logic unit testable (#47026)
Extracted ingest pipeline resolution logic into a static method
and added unit tests for pipeline resolution logic.

Followup from #46847
2019-09-25 11:35:00 +02:00
Paweł Krześniak b234d2cbd7 [Docs] Fix the numbering in the annotated_text example (#47077) 2019-09-25 11:29:01 +02:00
Yogesh Gaikwad 6f453aa6b2
Validate index and cluster privilege names when creating a role (#46361) (#47063)
This commit adds validation so a role cannot be created with
invalid index or cluster privilege name.

Closes #29703
2019-09-25 18:57:11 +10:00
Yannick Welsch 056ac32738 Mute JdbcCsvSpecIT.testAverageWithOneValueAndLimit
Relates to #47080
2019-09-25 10:36:53 +02:00
Daniel Mitterdorfer 48df560593
Emit log message when parent circuit breaker trips (#47000) (#47073)
We emit a debug log message whenever a child circuit breaker trips (in
`ChildMemoryCircuitBreaker#circuitBreak(String, long)`) but we never
emit a log message when the parent circuit breaker trips. As this is
more likely to happen with the real memory circuit breaker it is not
possible to detect this in the logs. With this commit we add a log
message on the same log level (debug) when the parent circuit breaker
trips.
2019-09-25 10:22:46 +02:00
Christoph Büscher 0c187e0a10
Add migration tool checks for `_field_names` disabling (#46972)
This change adds a check to the migration tool that warns about the deprecated
`enabled` setting for the `_field_names` field on 7.x indices and issues a
warning for templates containing this setting, which has been removed
with 8.0.

Relates to #42854, #46681
2019-09-25 10:15:10 +02:00
Yannick Welsch db63e78b68 Mute DebMetadataTests.test05CheckLintian
Relates #46903
2019-09-25 09:29:14 +02:00
Tanguy Leroux b1bf05bb89 Add blob container retries tests for Azure SDK client (#47032)
Similarly to what has been done for S3 and GCS, this commit
adds unit tests that verify the retry logic of the Azure SDK
client implementation when the remote service returns errors.

It only tests the retry logic in case of errors and not in
case of timeouts because Azure client timeout options are
not exposed as settings.
2019-09-25 09:19:48 +02:00
Hendrik Muhs 7377ac4637 [Transform] Replace transforms with transform, index constants (#47023)
- replace "transforms" with "transform" for consistency
 - use constants for internal index naming wherever possible and document required changes
2019-09-25 08:31:43 +02:00
Hendrik Muhs e974f178b5 [Transform] rename data frame transform to transform for hlrc client (#46933)
rename data frame transform to transform for hlrc
2019-09-25 08:31:43 +02:00
James Rodewig 61eef2fd31
[DOCS] Add multi-level nested query example to nested query docs (#46986) 2019-09-25 02:01:37 -04:00
Julie Tibshirani 4faba9cbbf Mention ip fields in the global ordinals docs. (#47045)
Although they do not support eager_global_ordinals, ip fields use global
ordinals for certain aggregations like 'terms'.

This commit also corrects a reference to the sampler aggregation.
2019-09-24 12:39:11 -07:00
Julie Tibshirani 41ee8aa6fc Reject regexp queries on the _index field. (#46945)
We speculatively added support for `regexp` queries on the `_index` field in
#34089 (this functionality was not actually requested by a user). Supporting
regex logic adds complexity to the `_index` field for not much gain, so we
would like to remove it.

From an end-to-end test it turns out this functionality never even worked in
the first place because of an error in how regex flags were interpreted! For
this reason, we can remove support for `regexp` on `_index` without a
deprecation period.

Relates to #46640.
2019-09-24 12:17:00 -07:00
Lisa Cawley 1893f6adc2 [DOCS] Merge monitoring sections in Elasticsearch book (#46885) 2019-09-24 11:39:28 -07:00
Tim Brooks 71ec0707cf
Remove locking around connection attempts (#46845)
Currently in the ConnectionManager we lock around the node id. This is
odd because we key connections by the ephemeral id. Upon further
investigation it appears to me that we do not need the locking. Using
the concurrent map, we can ensure that only one connection attempt
completes. There is a very small chance that a new connection attempt
will proceed right as another connection attempt is completing. However,
since the whole process is asynchronous and event oriented
(lightweight), that does not seem to be an issue.
2019-09-24 11:05:42 -06:00
Benjamin Trent 00c1c0132b
[ML] fix two datafeed flush lockup bugs (#46982) (#47024)
* [ML] fix two flush lockup bugs

* Addressing PR comments

* moving debug logging line so it is only written on success
2019-09-24 13:03:20 -04:00
Tim Brooks 6720c56bdd
Set netty system properties in BuildPlugin (#45881)
Currently in production instances of Elasticsearch we set a couple of
system properties by default. We currently do not apply all of these
system properties in tests. This commit applies these properties in the
tests.
2019-09-24 10:49:36 -06:00
David Roberts ffae769186
[TEST] Rename data-frames to transform in docs test cleanup (#47027)
The renaming of the tests in #46760 caused the
cleanup between tests to be skipped.

Backport of #47016
2019-09-24 17:38:09 +01:00
Tim Brooks f02582de4b
Reduce a bind failure to trace logging (#46891)
Due to recent changes in the nio transport, a failure to bind the server
channel has started to be logged at an error level. This exception leads
to an automatic retry on a different port, so it should only be logged
at a trace level.
2019-09-24 10:32:18 -06:00
David Roberts 22dade8e1b
[DOCS] Add 7.4 breaking changes for transforms and data frame analytics (#46821)
The PRs that made these changes are:

- https://github.com/elastic/elasticsearch/pull/44350
- https://github.com/elastic/elasticsearch/pull/45276
- https://github.com/elastic/elasticsearch/pull/45856

Co-Authored-By: István Zoltán Szabó <istvan.szabo@elastic.co>
Co-Authored-By: Lisa Cawley <lcawley@elastic.co>
2019-09-24 17:20:25 +01:00
Albert Zaharovits 3a82e0f7f4
Do not rewrite aliases on remove-index from aliases requests (#46989) (#47018)
When we rewrite alias requests, after filtering down to only those that
the user is authorized to see, it can be that there are no aliases
remaining in the request. However, core Elasticsearch interprets this as
_all so the user would see more than they are authorized for. To address
this, we previously rewrote all such requests to have aliases `"*"`,
`"-*"`, which would be interpreted when aliases are resolved as
nome. Yet, this is only needed for get aliases requests and we were
applying it to all alias requests, including remove index requests. If
such a request was sent to a coordinating node that is not the master
node, the request would be rewritten to include `"*"` and `"-*"`, and
then the master would authorize the user for these. If the user had
limited permissions, the request would fail, even if they were
authorized on the index that the remove index action was over. This
commit addresses this by rewriting for get aliases and remove
aliases request types but not for the remove index.

Co-authored-by: Albert Zaharovits <albert.zaharovits@elastic.co>
Co-authored-by: Tim Vernum <tim@adjective.org>
2019-09-24 19:07:55 +03:00
Dimitris Athanasiou 64bf1b56fe
[7.x] SQL: Mute pivot testAverageWithOneValueAndOrder and testSumWithoutSubquery (#47030) (#47033)
Relates #47002
2019-09-24 19:04:52 +03:00
Armin Braun 00f2e7f627
Update AWS SDK for repository-s3 plugin to support IAM Roles for Service Accounts (#46969) (#47004)
* Update AWS SDK for repository-s3 and discovery-ec2 plugins
2019-09-24 17:15:11 +02:00
Jack Conradson a1af2fe96a Rename Painless node SSource to SClass (#46984)
Mechanical renaming of SSource node to SClass to better align with the names of what other nodes generate.
2019-09-24 07:35:10 -07:00
David Turner 9135e2f9e3 Improve LeaderCheck rejection messages (#46998)
Today the `LeaderChecker` rejects checks from nodes that are not in the current
cluster with the exception message `leader check from unknown node` which
offers no information about why the node is unknown. In fact the node must have
been in the cluster in the recent past, so it might help guide the user to a
more useful log message if we describe it as a `removed node` instead of an
`unknown node`. This commit changes the exception message like this, and also
tidies up a few other loose ends in the `LeaderChecker`.
2019-09-24 13:41:37 +01:00
David Turner 6943a3101f
Cut PersistedState interface from GatewayMetaState (#46655)
Today `GatewayMetaState` implements `PersistedState` but it's an error to use
it as a `PersistedState` before it's been started, or if the node is
master-ineligible. It also holds some fields that are meaningless on nodes that
do not persist their states. Finally, it takes responsibility for both loading
the original cluster state and some of the high-level logic for writing the
cluster state back to disk.

This commit addresses these concerns by introducing a more specific
`PersistedState` implementation for use on master-eligible nodes which is only
instantiated if and when it's appropriate. It also moves the fields and
high-level persistence logic into a new `IncrementalClusterStateWriter` with a
more appropriate lifecycle.

Follow-up to #46326 and #46532
Relates #47001
2019-09-24 12:31:13 +01:00
Ioannis Kakavas 98e6bb4d01
Workaround JDK-8213202 in SSLClientAuthTests (#46995)
This change works around JDK-8213202, which is a bug related to TLSv1.3
session resumption before JDK 11.0.3 that occurs when there are
multiple concurrent sessions being established. Nodes connecting to
each other will trigger this bug when client authentication is
disabled, which is the case for SSLClientAuthTests.

Backport of #46680
2019-09-24 12:47:56 +03:00