Commit Graph

44300 Commits

Author SHA1 Message Date
Nhat Nguyen f64b20383e
Replace awaitBusy with assertBusy in atLeastDocsIndexed (#38190)
Unlike assertBusy, awaitBusy does not retry if the code-block throws an
AssertionError. A refresh in atLeastDocsIndexed can fail because we call
this method while we are closing some node in FollowerFailOverIT.
2019-02-01 13:31:17 -05:00
Luca Cavanna ee57420de6
Adjust SearchRequest version checks (#38181)
The finalReduce flag is now supported on 6.x too, hence we need to update the version checks in master.
2019-02-01 19:23:13 +01:00
Nhat Nguyen 70235838d1
AwaitsFix testClientSucceedsWithVerificationDisabled (#38213)
Tracked at #38212
2019-02-01 12:50:07 -05:00
Andrey Ershov 04dc41b99e
Zen2ify RareClusterStateIT (#38184)
In Zen 1 there are commit timeout and publish timeout and these
settings could be changed on-the-fly.

In Zen 2, there is only commit timeout and this setting is static.
RareClusterStateIT is actively using these settings and the fact, they
are dynamic.

This commit adds cancelCommitedPublication method to Coordinator to
be used by tests. This method will cancel current committed publication
if there is any.
When there is BlockClusterStateProcessing on the non-master node, the
publication will be accepted and committed, but not yet applied. So we
can use the method above to cancel it.

Also, this commit replaces callback + AtomicReference with ActionFuture,
which makes test code easier to read.
2019-02-01 18:18:11 +01:00
Benjamin Trent 5db305023d
ML: Fix error race condition on stop _all datafeeds and close _all jobs (#38113)
* ML: Ignore when task is not found for _all

* Addressing PR comments

* Update TransportStopDatafeedAction.java
2019-02-01 11:16:35 -06:00
Nhat Nguyen f5f3cb8f4c
AwaitsFix PUT mapping with _doc on an index that has types (#38204)
Tracked at #38202
2019-02-01 12:00:43 -05:00
Shaunak Kashyap cc7c42d7e2
Allow built-in monitoring_user role to call GET _xpack API (#38060)
This PR adds the `monitor/xpack/info` cluster-level privilege to the built-in `monitoring_user` role.

This privilege is required for the Monitoring UI to call the `GET _xpack API` on the Monitoring Cluster. It needs to do this in order to determine the license of the Monitoring Cluster, which further determines whether Cluster Alerts are shown to the user or not.

Resolves #37970.
2019-02-01 08:56:34 -08:00
Nick Knize 603cdf40f1
Update geo_shape docs to include unsupported features (#38138)
There are a two major features that are not yet supported by BKD Backed geo_shape: MultiPoint queries, and CONTAINS relation. It is important we are explicitly clear in the documentation that using the new approach may not work for users that depend on these features. This commit adds an IMPORTANT NOTE section to geo_shape docs that explicitly highlights these missing features and what should be done if they are an absolute necessity.
2019-02-01 10:41:41 -06:00
David Roberts 1fa413a16d
[ML] Remove "8" prefixes from file structure finder timestamp formats (#38016)
In 7.x Java timestamp formats are the default timestamp format and
there is no need to prefix them with "8".  (The "8" prefix was used
in 6.7 to distinguish Java timestamp formats from Joda timestamp
formats.)

This change removes the "8" prefixes from timestamp formats in the
output of the ML file structure finder.
2019-02-01 15:36:04 +00:00
Luca Cavanna 5c58c2508e
Disable bwc tests while backporting #38104 (#38182)
Relates to #38180
2019-02-01 16:34:24 +01:00
Jay Modi 2ca22209cd
Enable TLSv1.3 by default for JDKs with support (#38103)
This commit enables the use of TLSv1.3 with security by enabling us to
properly map `TLSv1.3` in the supported protocols setting to the
algorithm for a SSLContext. Additionally, we also enable TLSv1.3 by
default on JDKs that support it.

An issue was uncovered with the MockWebServer when TLSv1.3 is used that
ultimately winds up in an endless loop when the client does not trust
the server's certificate. Due to this, SSLConfigurationReloaderTests
has been pinned to TLSv1.2.

Closes #32276
2019-02-01 08:34:11 -07:00
Yannick Welsch 025bf28405
Fix _host based require filters (#38173)
Using index.routing.allocation.require._host does not correctly work because the boolean logic in
filter matching is broken (DiscoveryNodeFilters.match(...) will return false) when
opType ==OpType.AND
2019-02-01 16:02:37 +01:00
Tanguy Leroux da6269b456
RestoreService should update primary terms when restoring shards of existing indices (#38177)
When restoring shards of existing indices, the RestoreService also 
restores the values of primary terms stored in the snapshot index 
metadata. The primary terms are not updated and could potentially 
conflict with current index primary terms if the restored primary terms 
are lower than the existing ones.

This situation is likely to happen with replicated closed indices 
(because primary terms are increased when the index is transitioning 
from open to closed state, and the snapshotted primary terms are the
 one at the time the index was opened) (see #38024) and maybe also 
with CCR.

This commit changes the RestoreService so that it updates the primary 
terms using the maximum value between the snapshotted values and 
the existing values.

Related to #33888
2019-02-01 15:59:11 +01:00
Desmond Vehar c1c4abae10 Throw if two inner_hits have the same name (#37645)
This change throws an error if two inner_hits have the same name

Closes #37584
2019-02-01 15:53:50 +01:00
Alexander Reelsen 35ed137684
Ensure joda compatibility in custom date formats (#38171)
If custom date formats are used, there may be combinations that the new
performat DateFormatters.from() method has not covered yet. This adds a
few such corner cases and ensures the tests are correctly commented
out.
2019-02-01 15:42:56 +01:00
Jim Ferenczi 66e4fb4fb6
Do not compute cardinality if the `terms` execution mode does not use `global_ordinals` (#38169)
In #38158 we ensured that global ordinals are not loaded when another execution hint is explicitly set on the source. This change is a follow up that addresses a comment
dd6043c1c0 (r252984782) added after the merge.
2019-02-01 15:32:19 +01:00
Nhat Nguyen 2e475d63f7
Do not set timeout for IndexRequests in GatewayIndexStateIT (#38147)
CI might not be fast enough to publish a dynamic mapping update within 100ms.
2019-02-01 09:30:03 -05:00
Andrey Ershov c1270e97b0
Zen2ify testMasterFailoverDuringIndexingWithMappingChanges (#38178)
In Zen2 cluster bootstrap is required and some parameters are 
called differently in Zen2.
2019-02-01 15:24:08 +01:00
Marios Trivyzas 19dccf8f3e
SQL: [Docs] Add limitation for aggregate functions on scalars (#38186)
Currently aggregate functions can operate only directly on fields.
They cannot be used on top of scalar functions as painless scripting
is currently not supported.
2019-02-01 16:13:51 +02:00
Andrey Ershov bda591453c
Add elasticsearch-node detach-cluster command (#37979)
This commit adds the second part of `elasticsearch-node` tool -
`detach-cluster` command in addition to `unsafe-bootstrap` command.
Also, this commit changes the semantics of `unsafe-bootstrap`, now
`unsafe-bootstrap` changes clusterUUID.
So the algorithm of running `elasticsearch-node` tool is the following:
1) Stop all nodes in the cluster.
2) Pick master-eligible node with the highest (term, version) pair and
run the `unsafe-bootstrap` command on it. If there are no survived
master-eligible nodes - skip this step.
3) Run `detach-cluster` command on the remaining survived nodes.

Detach cluster makes the following changes to the node metadata:
1) Sets clusterUUID committed to false.
2) Sets currentTerm and term to 0. 
3) Removes voting tombstones and sets voting configurations to special
constant MUST_JOIN_ELECTED_MASTER, that prevents initial cluster
bootstrap.

`ElasticsearchNodeCommand` base abstract class is introduced, because
`UnsafeBootstrapMasterCommand` and `DetachClusterCommand` have a lot in
common.
Also, this commit adds "ordinal" parameter to both commands, because it's 
impossible to write IT otherwise.
For MUST_JOIN_ELECTED_MASTER case special handling is introduced in
`ClusterFormationFailureHelper`.
Tests for both commands reside in `ElasticsearchNodeCommandIT` (renamed
from `UnsafeBootstrapMasterIT`).
2019-02-01 14:53:55 +01:00
Alexander Reelsen 979e5576e5
Add tests for fractional epoch parsing (#38162)
Fractional epoch parsing is supported, the tests we used were edge cases
that did not make sense. This adds tests to properly check for this.
2019-02-01 14:48:37 +01:00
Adrien Grand 2229e7231e
Enable bw tests for #37871 and #38032. (#38167)
Mixed-version clusters tests had been disabled initially since they wouldn't
work until the functionality would be backported.
2019-02-01 13:55:51 +01:00
Tanguy Leroux 029e4b6278
Clear send behavior rule in CloseWhileRelocatingShardsIT (#38159)
The current CloseWhileRelocatingShardsIT test adds some "send behavior" 
rule to a target node's mocked transport service in order to detect when shard 
relocating are started. These rules are never cleared and prevent the test to 
complete normally after the rebalance is re-enabled again.

This commit changes the test so that rules are cleared and most verifications 
are done before the rebalance is reenabled again.

Closes #38090
2019-02-01 12:58:46 +01:00
Yannick Welsch ce469cfda5
Fix testCorruptedIndex (#38161)
Folks at the Lucene project do not seem to be interested in classifying corruptions and
distinguishing them from file-system exceptions (see https://issues.apache.org/jira/browse/LUCENE-8525),
so we'll just cop out as well.

Closes #34322
2019-02-01 12:51:38 +01:00
Luca Cavanna e18cac3659
Add finalReduce flag to SearchRequest (#38104)
With #37000 we made sure that fnial reduction is automatically disabled
whenever a localClusterAlias is provided with a SearchRequest.

While working on #37838, we found a scenario where we do need to set a
localClusterAlias yet we would like to perform a final reduction in the
remote cluster: when searching on a single remote cluster.

Relates to #32125

This commit adds support for a separate finalReduce flag to
SearchRequest and makes use of it in TransportSearchAction in case we
are searching against a single remote cluster.

This also makes sure that num_reduce_phases is correct when searching
against a single remote cluster: it makes little sense to return
`num_reduce_phases` set to `2`, which looks especially weird in case
the search was performed against a single remote shard. We should
perform one reduction phase only in this case and `num_reduce_phases`
should reflect that.

* line length
2019-02-01 12:11:42 +01:00
Jim Ferenczi 6fa93ca493
Forbid negative field boosts in analyzed queries (#37930)
This change forbids negative field boost in the `query_string`, `simple_query_string`
and `multi_match` queries.
Negative boosts are not allowed in Lucene 8 (scores must be positive).
The backport of this change to 6x will turn the error into a deprecation warning
in order to raise the awareness of this breaking change in 7.0.

Closes #33309
2019-02-01 11:41:40 +01:00
Jim Ferenczi 57b1d245e8
Remove AtomiFieldData#getLegacyFieldValues (#38087)
This function is unused now that we format the docvalue fields with the default
formatter on the field (#30831)
2019-02-01 11:41:17 +01:00
Andrey Ershov bfd618cf83
Universal cluster bootstrap method for tests with autoMinMasterNodes=false (#38038)
Currently, there are a few tests that use autoMinMasterNodes=false and
hence override addExtraClusterBootstrapSettings, mostly this is 10-30
lines of codes that are copy-pasted from class to class.

This PR introduces `InternalTestCluster.setBootstrapMasterNodeIndex`
which is suitable for all classes and copy-paste could be removed.

Removing code is always a good thing!
2019-02-01 11:34:31 +01:00
Tanguy Leroux 0e6a7c20a1
Fix FullClusterRestartIT.testHistoryUUIDIsAdded (#38098)
This test failed once because the index wasn't fully ready 
(ie, engine opened). This commit changes the test so that 
it waits for the index to be green before checking the history 
UUID.

Closes #34452
2019-02-01 11:16:54 +01:00
Alexander Reelsen 6c5a7387af
Replace joda time in ingest-common module (#38088)
This commit fully replaces any remaining joda time time classes with
java time implementations.

Relates #27330
2019-02-01 10:15:18 +01:00
Alpar Torok d417997aca
Fix eclipse config for ssl-config (#38096) 2019-02-01 10:47:54 +02:00
Jim Ferenczi b7308aa03c
Don't load global ordinals with the `map` execution_hint (#37833)
The terms aggregator loads the global ordinals to retrieve the cardinality of the field to aggregate on. This information is then used to select the strategy to use for the aggregation (breadth_first or depth_first). However this should be avoided if the execution_hint is explicitly set to map since this mode doesn't really need the global ordinals. Since we still need the cardinality of the field this change picks the maximum cardinality in the segments as an estimation of the total cardinality to select the strategy to use (breadth_first or depth_first). This estimation is only used if the execution hint is set to map, otherwise the global ordinals are still used to retrieve the accurate cardinality.

Closes #37705
2019-02-01 09:35:46 +01:00
David Turner 23f00e3676
Relax fault detector in some disruption tests (#38101)
Today we use `AbstractDisruptionTestCase` to test the behaviour of things like
master elections in the presence of cluster disruptions. These tests have
rather enthusiastic fault detection settings, detecting a fault if a single
ping fails, with a one-second timeout. Furthermore there are some tests that
assert the identity of the master remains unchanged during some disruption, and
these assertions fail rather often thanks to the overly sensitive fault
detector.

However in a number of these tests the fault detector need not be this
sensitive. This commit moves some such tests into their own test suite and uses
more sensible fault-detection settings to avoid the kind of master instability
that is causing CI failures.

Closes #37699
2019-02-01 08:10:49 +00:00
Alexander Reelsen c02cd3e2fd
Fix java time epoch date formatters (#37829)
The self written epoch date formatters were not properly able to format
an Instant to a string due to a misconfiguration.

This fix also removes a until now existing runtime behaviour under java
8 regarding the names of the aggregation buckets, which are now the same
as before and have been under java 11.
2019-02-01 09:03:48 +01:00
Yannick Welsch 859e2f5bc8 Adapt timeouts in UpdateMappingIntegrationIT
Relates to #37263 and possibly #36916
2019-02-01 08:58:31 +01:00
Adrien Grand d83c748417
Fix test bug in DynamicMappingsIT. (#37906)
Closes #37898
2019-02-01 08:35:29 +01:00
Przemyslaw Gomulka 2758578570
Trim the JSON source in indexing slow logs (#38081)
The '{' as a first character in log line is causing problems for beats when parsing plaintext logs. This can happen if the submitted document has an additional '\n' at the beginning and we are not reformatting. 
Trimming the source part of a SlogLog solves that and keeps the logs readable.

closes #38080
2019-02-01 08:12:12 +01:00
Tim Vernum 6fcbd07420
Remove heuristics that enable security on trial licenses (#38075)
In 6.3 trial licenses were changed to default to security
disabled, and ee added some heuristics to detect when security should
be automatically be enabled if `xpack.security.enabled` was not set.

This change removes those heuristics, and requires that security be
explicitly enabled (via the `xpack.security.enabled` setting) for
trial licenses.

Relates: #38009
2019-02-01 17:59:13 +11:00
Armin Braun 0a604e3b24
Fix Two Races that Lead to Stuck Snapshots (#37686)
* Fixes two broken spots:
    1. Master failover while deleting a snapshot that has no shards will get stuck if the new master finds the 0-shard snapshot in `INIT` when deleting
    2. Aborted shards that were never seen in `INIT` state by the `SnapshotsShardService` will not be notified as failed, leading to the snapshot staying in `ABORTED` state and never getting deleted with one or more shards stuck in `ABORTED` state
* Tried to make fixes as short as possible so we can backport to `6.x` with the least amount of risk
* Significantly extended test infrastructure to reproduce the above two issues
  * Two new test runs:
      1. Reproducing the effects of node disconnects/restarts in isolation
      2. Reproducing the effects of disconnects/restarts in parallel with shard relocations and deletes
* Relates #32265 
* Closes #32348
2019-02-01 05:45:40 +01:00
Paul Sanwald 0d56955d39
mute test, as this one is failing also per #35450 (#38132) 2019-01-31 22:00:09 -05:00
Nhat Nguyen b8b843476d
Disable dynamic mapping in testSimpleGetFieldMappingsWithDefaults (#38045)
Since #31140 we no longer require acking on the dynamic mapping of index
requests. Thus, a returned mapping from a get mapping request does not
necessarily contain the dynamic updates from the index request. This
commit replaces the dynamic mapping update with a manual put mapping.

Relates #31140
Closes #37928
2019-01-31 21:01:41 -05:00
Nhat Nguyen a8ebe2a217
Fix random params in testSoftDeletesRetentionLock (#38114)
Since #37992 the retainingSequenceNumber is initialized with 0 
while the global checkpoint can be -1.

Relates #37992
2019-01-31 20:50:41 -05:00
Tim Brooks 291c4e7a0c
Fix file reading in ccr restore service (#38117)
Currently we use the raw byte array length when calling the IndexInput
read call to determine how many bytes we want to read. However, due to
how BigArrays works, the array length might be longer than the reference
length. This commit fixes the issue and uses the BytesRef length when
calling read. Additionally, it expands the index follow test to index
many more documents. These documents should potentially lead to large
enough segment files to trigger scenarios where this fix matters.
2019-01-31 18:02:24 -07:00
Lee Hinman c67a9663af
Fix MasterServiceTests.testClusterStateUpdateLogging (#38116)
This changes the test to not use a `CountDownlatch`, instead adding an assertion
for the final logging message and waiting until the `MockAppender` has seen it
before proceeding.

Related to df2c06f6f30f7e23a6863a3f72fc3bdb7648885c
Resolves #23739
2019-01-31 17:13:19 -07:00
Yuri Astrakhan f3cde06a1d
geotile_grid implementation (#37842)
Implements `geotile_grid` aggregation

This patch refactors previous implementation https://github.com/elastic/elasticsearch/pull/30240

This code uses the same base classes as `geohash_grid` agg, but uses a different hashing
algorithm to allow zoom consistency.  Each grid bucket is aligned to Web Mercator tiles.
2019-01-31 19:11:30 -05:00
Nhat Nguyen 6c1e9fad47 Mute testAutoFollowing
Tracked at #37231
2019-01-31 16:57:53 -05:00
Pascal Christoph a3d9ba3f4b Log document id when MapperParsingException occurs (#37800)
Closes #37658
2019-01-31 16:33:13 -05:00
Jake Landis 5b008a34aa
Ingest node - user agent, move device to an object (#38115)
When the ingest node user agent parses the device field, it
will result in a string value. To match the ecs schema
this commit moves the value of the parsed device to an
object with an inner field named 'name'. There are not
any passivity concerns since this modifies an unreleased change.

closes #38094
relates #37329
2019-01-31 13:54:34 -06:00
Julie Tibshirani 91b79ebed4
Update 'removal of types' docs to reflect the new plan. (#38003) 2019-01-31 10:26:24 -08:00
Nhat Nguyen 237fcda2cc
Disable dynamic mapping update in testTransportBulkTasks (#38073)
If a replica does not have a right mapping yet, we will retry the index
request on that replica; then the actual tasks is higher than the
expected tasks. Since #31140 this happens more frequently for we no
longer require acking on the dynamic mapping of index requests.

Relates #31140
Closes #37893
2019-01-31 13:16:52 -05:00