Commit Graph

48574 Commits

Author SHA1 Message Date
Tim Brooks 45e42f4e18
Upgrade to Netty 4.1.43 (#48484)
With this update we can remove the mitigation in our custom allocator
which forces heap buffer allocations.
2019-10-25 10:17:25 -06:00
Jim Ferenczi 7fc413c22c Resolve the role query and the number of docs lazily (#48036)
This commit ensures that the creation of a DocumentSubsetReader does not
eagerly resolve the role query and the number of docs that match.
We want to delay this expensive operation in order to ensure that we really
need this information when we build it. For this reason the role query and the
number of docs are now resolved on demand. This commit also depends on
https://issues.apache.org/jira/browse/LUCENE-9003 that will also compute the global
number of docs lazily.
2019-10-25 18:12:29 +02:00
Tanguy Leroux 2861088a59 Mute RestClientMultipleHostsIntegTests.testCancelAsyncRequests (#48535)
Relates https://github.com/elastic/elasticsearch/issues/45577
2019-10-25 17:42:24 +02:00
James Rodewig e9c8e4f6d1 [DOCS] Fix note format in index suggestion docs (#48536) 2019-10-25 11:31:47 -04:00
Tim Brooks f5f1072824
Multiple remote connection strategy support (#48496)
* Extract remote "sniffing" to connection strategy (#47253)

Currently the connection strategy used by the remote cluster service is
implemented as a multi-step sniffing process in the
RemoteClusterConnection. We intend to introduce a new connection strategy
that will operate in a different manner. This commit extracts the
sniffing logic to a dedicated strategy class. Additionally, it implements
dedicated tests for this class.

Additionally, in previous commits we moved away from a world where the
remote cluster connection was mutable. Instead, when setting updates are
made, the connection is torn down and rebuilt. We still had methods and
tests hanging around for the mutable behavior. This commit removes those.

* Introduce simple remote connection strategy (#47480)

This commit introduces a simple remote connection strategy which will
open remote connections to a configurable list of user supplied
addresses. These addresses can be remote Elasticsearch nodes or
intermediate proxies. We will perform normal clustername and version
validation, but otherwise rely on the remote cluster to route requests
to the appropriate remote node.

* Make remote setting updates support diff strategies (#47891)

Currently the entire remote cluster settings infrastructure is designed
around the sniff strategy. As we introduce an additional conneciton
strategy this infrastructure needs to be modified to support it. This
commit modifies the code so that the strategy implementations will tell
the service if the connection needs to be torn down and rebuilt.

As part of this commit, we will wait 10 seconds for new clusters to
connect when they are added through the "update" settings
infrastructure.

* Make remote setting updates support diff strategies (#47891)

Currently the entire remote cluster settings infrastructure is designed
around the sniff strategy. As we introduce an additional conneciton
strategy this infrastructure needs to be modified to support it. This
commit modifies the code so that the strategy implementations will tell
the service if the connection needs to be torn down and rebuilt.

As part of this commit, we will wait 10 seconds for new clusters to
connect when they are added through the "update" settings
infrastructure.
2019-10-25 09:29:41 -06:00
Luca Cavanna d6d2edf324 Fix .tasks index strict mapping: parent_id should be parent_task_id (#48393)
* Fix .tasks index strict mapping: parent_id should be parent_task_id

The .tasks index has mappings that's strictly defined. `parent_task_id`
was defined as `parent_id` though which would cause an exception in case
a task is persisted that has a parent task id set.

While at it, a couple of compiler warnings were addressed and a test
request builder was removed in favour of using its corresponding request.

* increment version
2019-10-25 17:00:06 +02:00
Luca Cavanna 9c48ed12bc Remove response search phase from ExpandSearchPhase (#48401)
The expand phase is always created providing a function that builds
the next phase to be run, which has a single purpose: sending the
response back. Such small search phase is not necessary and causes some
issues when reporting search progress and counting the search phases
that need to be executed and that are already executed. We can simply
rather send back the response, without creating a specific phase for that.
2019-10-25 17:00:06 +02:00
Christoph Büscher 055a0800eb [Docs] Mention reserved completion suggestion characters (#48445)
We currently don't mention the three reserved characters anywhere. This change
adds a short note mentioning them

Closes #48341
2019-10-25 16:58:23 +02:00
Tanguy Leroux 06d2cc5cef Add missing azure error code (#48520)
In #47176 we changed the internal HTTP server that emulates 
the Azure Storage service so that it includes a response body 
for injected errors. This fixed most of the issues reported in 
#47120 but sadly I missed to map one error to its Azure 
equivalent, and it triggered some CI failures today.

Closes #47120
2019-10-25 16:50:51 +02:00
Andrei Stefan 9a89ed891e Detail the IDEs options for configuring the debug step (#48507)
(cherry picked from commit 451d9cf06f1331ecfb6b4fa4acaed6fb43a1fcbb)
2019-10-25 17:27:48 +03:00
Armin Braun 84a47a9632
Remove Outdated AwaitsFix (#48513) (#48522)
This `AwaitsFix` was accidentally added after the test
was already fixed in #46594 => we can remove it.
2019-10-25 16:08:56 +02:00
Peter Dyson eb44a25899 [DOCS] Reorder bullet items in CCS security docs (#48501)
Adjust the last bullet item to be above the code block for better readability and to avoid it being skimmed over
2019-10-25 09:11:49 -04:00
Armin Braun edab3748e9
Remove Incorrect Assertion from SnapshotsInProgress (#47458) (#48514)
This relates to the effort towards #46250. We added
tracking of the shard generation for successful
snapshots to `8.0`.
This assertion isn't correct though. While an `8.0`
master won't create an entry with sucess state and
a null shard generation it may still (on e.g. master
failover) send a success entry created by a 7.x master
with a `null` generation over the wire.

Closes #47406
2019-10-25 15:03:23 +02:00
Christoph Büscher 3fb3397c12 BlendedTermQuery's equals method should consider boosts (#48193)
This changes the queries equals() method so that the boost factors for each term
are considered for the equality calculation. This means queries are only equal
if both their terms and associated boosts match. The ordering of the terms
doesn't matter as before, which is why we internally need to sort the terms and
boost for comparison on the first equals() call like before. Boosts that are
`null` are considered equal to boosts of 1.0f because topLevelQuery() will only
wrap into BoostQuery if boost is not null and different from 1f.

Closes #48184
2019-10-25 13:35:14 +02:00
Yannick Welsch 486794f24d Show task ID in source of persistent task state update (#48483)
Relates #48395
2019-10-25 10:29:16 +02:00
Russ Cam b24bbd4296 Change policy_id to list type in slm.get_lifecycle (#47766)
This commit changes the REST API spec slm.get_lifecycle's policy_id url part to be of type "list", in line with other REST API specs that accept a comma-separated list of values.

Closes #47765
2019-10-25 09:04:25 +10:00
Tim Brooks c0b545f325
Make BytesReference an interface (#48486)
BytesReference is currently an abstract class which is extended by
various implementations. This makes it very difficult to use the
delegation pattern. The implication of this is that our releasable
BytesReference is a PagedBytesReference type and cannot be used as a
generic releasable bytes reference that delegates to any reference type.
This commit makes BytesReference an interface and introduces an
AbstractBytesReference for common functionality.
2019-10-24 15:39:30 -06:00
Michael Basnight d49958cef3 Remove deprecated test from the HLRC tests (#48424)
The AbstractHlrcWriteableXContentTestCase was replaced by a better test
case a while ago, and this is the last two instances using it. They have
been converted and the test is now deleted.

Ref #39745
2019-10-24 14:02:04 -05:00
Julie Tibshirani b2974e3816 Correct outdated information in _index docs. (#48436)
This PR makes the following updates:
* Update the supported query types to include `prefix` and `wildcard`.
* Specify that queries accept index aliases.
* Clarify that when querying on a remote index name, the separator `:` must be
  present.
2019-10-24 11:02:25 -07:00
Jake Landis a4614daf46
Allow more time for restart tests to reach yellow state. (#48434) (#48480)
The testWatcher method will on occasion timeout waiting for
a yellow cluster state. This change increases the timeout
to 60s.
2019-10-24 12:07:02 -05:00
Michael Basnight c19379ef31 Remove random when using HLRC sync and async calls (#48211)
This commit removes the randomization used by every execute call in the
high level rest tests. Previously every execute call, which can be many
calls per single test, would rely on a random boolean to determine if
they should use the sync or async methods provided to the execute
method. This commit runs the tests twice, using two different clusters,
both of them providing the value one time via a sysprop. This ensures
that the whole suite of tests is run using the sync and async code
paths.

Closes #39667
2019-10-24 09:06:17 -05:00
Martijn van Groningen b034153df7
Change grok watch dog to be Matcher based instead of thread based. (#48346)
There is a watchdog in order to avoid long running (and expensive)
grok expressions. Currently the watchdog is thread based, threads
that run grok expressions are registered and after completion unregister.
If these threads stay registered for too long then the watch dog interrupts
these threads. Joni (the library that powers grok expressions) has a
mechanism that checks whether the current thread is interrupted and
if so abort the pattern matching.

Newer versions have an additional method to abort long running pattern
matching inside joni. Instead of checking the thread's interrupted flag,
joni now also checks a volatile field that can be set via a `Matcher`
instance. This is more efficient method for aborting long running matches.
(joni checks each 30k iterations whether interrupted flag is set vs.
just checking a volatile field)

Recently we upgraded to a recent joni version (#47374), and this PR
is a followup of that PR.

This change should also fix #43673, since it appears when unit tests
are ran the a test runner thread's interrupted flag may already have
been set, due to some thread reuse.
2019-10-24 15:34:01 +02:00
Hendrik Muhs 5ecfcdb162 update warning about index names after transform rename (#48457)
update warning about index names after transform rename
2019-10-24 15:17:20 +02:00
Dimitrios Liappis fc1b4ad23c
Mute testCCRUnfollowDuringSnapshot (#48464)
tracked in #48461
backport of #48462
2019-10-24 15:52:56 +03:00
Przemysław Witek 149537a165
Assert that inference model has been persisted (#48332) (#48453) 2019-10-24 14:18:43 +02:00
Dimitrios Liappis 4d0fb6e551
Mute testBasicTimeBasedRetenion (#48458)
tracked in #48017
backport of #48456
2019-10-24 14:53:12 +03:00
Yannick Welsch acf6d34d69 Always use last properly persisted metadata as previous state (#47779)
On data-only nodes we were not using the last persisted cluster state as base point to compute
what needed storage, but the last applied cluster state (but not necessarily properly persisted)
instead.
2019-10-24 13:30:59 +02:00
Hendrik Muhs ba1c13c47d [Transform] do not fail checkpoint creation due to global checkpoint mismatch (#48423)
Take the max if global checkpoints mismatch instead of throwing an exception. It turned out global
checkpoints can mismatch by design

fixes #48379
2019-10-24 12:22:07 +02:00
Yannick Welsch 65c58ed594
Use _type in op_type REST test (#48418)
Closes #48396
2019-10-24 12:08:21 +02:00
David Turner 50518359fe Fix relocating shards size calculation (#48421)
In #48392 we added a second computation of the sizes of the relocating shards
in `canRemain()` but passed the wrong value for `subtractLeavingShards`. This
fixes that. It also removes some unnecessary logging in a test case added in
the same commit.
2019-10-24 08:58:50 +01:00
Jim Ferenczi dc5c31d67a
Add a deprecation warning regarding allocation awareness in search request (#48351)
This is a follow up of https://github.com/elastic/elasticsearch/issues/43453 where we added
a system property to disallow allocation awareness in search requests. Since search requests
will no longer check the allocation awareness attributes for routing in the next major version,
this change adds a deprecation warning on any setup that uses these attributes.

Relates #43453
2019-10-24 09:25:50 +02:00
Ioannis Kakavas c6b733f1b4
Add populate_user_metadata in OIDC realm (#48357) (#48438)
Make populate_user_metadata configuration parameter
available in the OpenID Connect authentication realm

Resolves: #48217
2019-10-24 09:51:08 +03:00
Martijn van Groningen 05324b7f03
Muted verifying monitoring integration in enrich integration test.
Relates to #48258
2019-10-24 08:39:53 +02:00
Julie Tibshirani 4375316b9d Make sure to list the 7.5 migration docs. 2019-10-23 18:52:22 -07:00
Julie Tibshirani 2664cbd20b
Deprecate the sparse_vector field type. (#48368)
We have not seen much adoption of this experimental field type, and don't see a
clear use case as it's currently designed. This PR deprecates the field type in
7.x. It will be removed from 8.0 in a follow-up PR.
2019-10-23 16:35:03 -07:00
Mayya Sharipova 9e9533f717 Correct syntax from backport
User older format of map

Relates to #48425
2019-10-23 17:19:15 -04:00
Mayya Sharipova 975dbecfa9 Correct rewritting of script_score query (#48425)
Previously there was a bug when an query inside script_score query
was rewritten. If min_score was not set and was equal to null,
we were converting it to float value which resulted to NPE.
This commit corrects this.

Closes #48081
2019-10-23 17:01:51 -04:00
James Rodewig 06dc1fbd96 [DOCS] Reformat ASCII folding token filter docs (#48143) 2019-10-23 15:06:55 -05:00
Igor Motov 8163e0a9e5 Mute XPackRestIT security/authz/14_cat_indices
Mutes "Test empty request while single authorized closed index"

Tracked by #47875
2019-10-23 14:17:44 -04:00
Igor Motov bdbc353dea Geo: improve handling of out of bounds points in linestrings (#47939)
Brings handling of out of bounds points in linestrings in line with
points. Now points with latitude above 90 and below -90 are handled
the same way as for points by adjusting the longitude by moving it by
180 degrees.

Relates to #43916
2019-10-23 14:17:44 -04:00
Jim Ferenczi 41116eb7ea Do not throw errors on unknown types in SearchAfterBuilder (#48147)
* Do not throw errors on unknown types in SearchAfterBuilder

The support for BigInteger and BigDecimal was added for XContent in
https://github.com/elastic/elasticsearch/pull/32888. However the SearchAfterBuilder
xcontent parser doesn't expect them to be present so it throws an AssertionError.
This change fixes this discrepancy by changing the AssertionError into an
IllegalArgumentException that will not cause the node to die when thrown.

Closes #48074
2019-10-23 20:02:14 +02:00
Jim Ferenczi 96556d72cc
Add a known issue to the release notes of 7.4.0 (#48373)
A [bug](https://github.com/elastic/elasticsearch/issues/48358) in 7.4.0 prevents
the activation of the search slow log. This change adds an entry in the release
notes to warn users to not activate it in this version.

Relates #48358
2019-10-23 19:57:37 +02:00
Tom Callahan 892264a97a Add versions 7.4.2 and 6.8.5 2019-10-23 13:32:51 -04:00
David Turner c783a20560 Handle negative free disk space in deciders (#48392)
Today it is possible that the total size of all relocating shards exceeds the
total amount of free disk space. For instance, this may be caused by another
user of the same disk increasing their disk usage, or may be due to how
Elasticsearch double-counts relocations that are nearly complete particularly
if there are many concurrent relocations in progress.

The `DiskThresholdDecider` treats negative free space similarly to zero free
space, but it then fails when rendering the messages that explain its decision.
This commit fixes its handling of negative free space.

Fixes #48380
2019-10-23 18:16:41 +01:00
James Rodewig 640d7416b1
[DOCS] Change prev version to 7.5 in upgrade docs (#48415) 2019-10-23 12:09:26 -05:00
Jake Landis cf175da5a9
Ensure SLM stats does not block an in-place upgrade from 7.4 (… (#48411)
7.5+ for SLM requires [stats] object to exist in the cluster state.
When doing an in-place upgrade from 7.4 to 7.5+ [stats] does not exist
in cluster state, result in an exception on startup [1].

This commit moves the [stats] to be an optional object in the parser
and if not found will default to an empty stats object.

[1] Caused by: java.lang.IllegalArgumentException: Required [stats]
2019-10-23 11:21:39 -05:00
James Rodewig 19afe3f84c [DOCS] Remove duplicate links for ingest processor overview (#48394) 2019-10-23 10:55:49 -05:00
Adrien Grand 81ef72d3ef
Lucene#asSequentialBits gets the leadCost backwards. (#48335) (#48403)
The comment says it needs random-access, but it passes `Long#MAX_VALUE` as the
lead cost, which forces sequential access, it should pass `0` instead. I took
advantage of this fix to improve the logic to leverage an estimation of the
number of times that `Bits#get` gets called to make better decisions.
2019-10-23 17:48:17 +02:00
Przemyslaw Gomulka aaa6209be6
[7.x] [Java.time] Calculate week of a year with ISO rules BACKPORT(#48209) (#48349)
Reverting the change introducing IsoLocal.ROOT and introducing IsoCalendarDataProvider that defaults start of the week to Monday and requires minimum 4 days in first week of a year. This extension is using java SPI mechanism and defaults for Locale.ROOT only.
It require jvm property java.locale.providers to be set with SPI,COMPAT

closes #41670
backport #48209
2019-10-23 17:39:38 +02:00
James Rodewig 9c75f14a9f [DOCS] Reformat classic token filter docs (#48314) 2019-10-23 10:14:25 -05:00