Commit Graph

45522 Commits

Author SHA1 Message Date
Armin Braun 233df6b73b
Make Transport Shard Bulk Action Async (#39793) (#41112)
This is a dependency of #39504

Motivation:
By refactoring `TransportShardBulkAction#shardOperationOnPrimary` to async, we enable using `DeterministicTaskQueue` based tests to run indexing operations. This was previously impossible since we were blocking on the `write` thread until the `update` thread finished the mapping update.
With this change, the mapping update will trigger a new task in the `write` queue instead.
This change significantly enhances the amount of coverage we get from `SnapshotResiliencyTests` (and other potential future tests) when it comes to tracking down concurrency issues with distributed state machines.

The logical change is effectively all in `TransportShardBulkAction`, the rest of the changes is then simply mechanically moving the caller code and tests to being async and passing the `ActionListener` down.

Since the move to async would've added more parameters to the `private static` steps in this logic, I decided to inline and dry up (between delete and update) the logic as much as I could instead of passing the listener + wait-consumer down through all of them.
2019-04-11 16:01:52 +02:00
James Rodewig 93d6766afe [DOCS] Improve docs for 'elasticsearch-keystore add-file' command (#41084) 2019-04-11 08:50:13 -04:00
Armin Braun 9cfe194131
Disable TestingConventions Check in Reindex Module (#41100) (#41110)
* This is currently failing with:

```
Test classes are not included in any enabled task (:modules:reindex:test):
  * org.elasticsearch.client.documentation.ReindexDocumentationIT
  * org.elasticsearch.index.reindex.ManyDocumentsIT
  * org.elasticsearch.index.reindex.ReindexClientYamlTestSuiteIT
  * org.elasticsearch.index.reindex.ReindexWithoutContentIT
  * org.elasticsearch.index.reindex.remote.ReindexFromOldRemoteIT
```

* Same fix as in #38546
2019-04-11 13:47:14 +02:00
Mark Vieira a6dc20a09d Create heap dump on OOME (#41031) 2019-04-11 13:41:35 +03:00
Henning Andersen b967a97f8e
Reindex from remote deprecation warning (#41005)
If a reindex from remote request contains an index name that is URL
escaped, we now issue a warning to be able to not support this in 8.0.
2019-04-11 12:09:53 +02:00
Dimitris Athanasiou 736c7285d4
[ML] Fix scroll size comparison in DatafeedUpdate.isNoop (#41056) (#41079)
Note this does not affect users as the method is only used in tests.
2019-04-11 11:29:51 +03:00
James Rodewig 999462f460 [DOCS] Document limits for JSON objects with `ignore_malformed` mapping setting (#40976) 2019-04-10 14:47:26 -04:00
James Rodewig a0b54a9a92 [DOCS] Restructure `ids` to new query docs format (#41077) 2019-04-10 14:36:50 -04:00
Jason Tedor 24446ceae0
Add packaging to cluster stats response (#41048)
This commit adds a packaging_types field to the cluster stats response
that outlines the build flavors and types present in a cluster.
2019-04-10 13:47:19 -04:00
Ryan Ernst 884c3fd7ca
Use separate tasks per bwc artifact (#40703) (#40826)
This commit changes the bwc builds from a single task for a branch to a
task for each bwc artifact. This reduces the bwc build time when only
needing a specific artifact, for example when running cluster restart
tests on a mac, the windows artifacts or rpm/debs are not needed.
2019-04-10 10:43:01 -07:00
Hendrik Muhs c37b127a07 fix a timing issue: isFinished is used for a busy loop in testing, (#41055)
test: ensure state is persisted before the isFinished is changed

fixes #41046
2019-04-10 18:47:34 +02:00
Zachary Tong e611334b2b Add 7.0.1 version constant 2019-04-10 11:32:53 -04:00
Martijn van Groningen 1eff8976a8
Deprecate AbstractHlrc* and AbstractHlrcStreamable* base test classes (#41014)
* moved hlrc parsing tests from xpack to hlrc module and removed dependency on hlrc from xpack core

* deprecated old base test class

* added deprecated jdoc tag

* split test between xpack-core part and hlrc part

* added lang-mustache test dependency, this previously came in via
hlrc dependency.

* added hlrc dependency on a qa module

* duplicated ClusterPrivilegeName class in xpack-core, since x-pack
core no longer has a dependency on hlrc.

* replace ClusterPrivilegeName usages with string literals

* moved tests to dedicated to hlrc packages in order to remove Hlrc part from the name and make sure to use imports instead of full qualified class where possible

* remove ESTestCase. from method invocation and use method directly,
because these tests indirectly extend from ESTestCase
2019-04-10 16:29:17 +02:00
Chris Earle 64a05e522f
Properly handle Monitoring exporters all disabled (#40920) (#41043)
When monitoring exporters are all disabled, which must be done
explicitly, _and_ monitoring collection is enabled, then
any call to `_xpack/monitoring/_bulk` will create a task that
never closes _and_ ES collection will stop happening because
a semaphore is never marked as completed.

This also simplifies the async `ExportBulk` code by removing the
third step (second async step, `close`) entirely because it was
entirely unnecessary by both implementations.
2019-04-10 10:09:28 -04:00
Christoph Büscher d00d3f4afa Mute DataFrameTransformCheckpointTests#testGetBehind 2019-04-10 15:55:09 +02:00
Zachary Tong 0a85d1fe03 Add missing "long form" 7.0 breaking changes (#41049)
These are breaking changes that were present in the release notes but the PR didn't include any "narrative long form" description for the docs.
2019-04-10 09:44:45 -04:00
Zachary Tong 9aa8ab58ee Add 7.0.0 Release Notes (#41029) 2019-04-10 09:44:45 -04:00
Przemysław Witek f5014ace64
[ML] Add validation that rejects duplicate detectors in PutJobAction (#40967) (#41072)
* [ML] Add validation that rejects duplicate detectors in PutJobAction

Closes #39704

* Add YML integration test for duplicate detectors fix.

* Use "== false" comparison rather than "!" operator.

* Refine error message to sound more natural.

* Put job description in square brackets in the error message.

* Use the new validation in ValidateJobConfigAction.

* Exclude YML tests for new validation from permission tests.
2019-04-10 15:43:35 +02:00
Dimitrios Liappis 799541e068
Mute DateTimeUnitTests.testConversion (#40738)
Due to #39617

Backport of  #40086
2019-04-10 16:37:16 +03:00
Ed Savage 722362e402 Mute MachineLearningIt#testDeleteExpiredData
Tracked in #41070
2019-04-10 13:07:53 +01:00
Albert Zaharovits adf3393a4e
Deprecate permission over aliases (#38059) (#41060)
This PR generates deprecation log entries for each Role Descriptor,
used for building a Role, when the Role Descriptor grants more privileges
for an alias compared to an index that the alias points to. This is done in
preparation for the removal of the ability to define privileges over aliases.
There is one log entry for each "role descriptor name"-"alias name" pair.
On such a notice, the administrator is expected to modify the Role Descriptor
definition so that the name pattern for index names does not cover aliases.

Caveats:
* Role Descriptors that are not used in any authorization process,
either because they are not mapped to any user or the user they are mapped to
is not used by clients, are not be checked.
* Role Descriptors are merged when building the effective Role that is used in
the authorization process. Therefore some Role Descriptors can overlap others,
so even if one matches aliases in a deprecated way, and it is reported as such,
it is not at risk from the breaking behavior in the current role mapping configuration
and index-alias configuration. It is still reported because it is a best practice to
change its definition, or remove offending aliases.
2019-04-10 15:02:33 +03:00
Costin Leau 38f471ae1c SQL: Change schema calls to empty set (#41034)
As empty string has a certain meaning, the JDBC driver returns an empty
set instead for better client compatibility.

Fix #41028

(cherry picked from commit 4cbafa585b7a514eb6c156606dd516324cd3980a)
2019-04-10 13:14:12 +03:00
Jim Ferenczi 4263a28039 Fix rewrite of inner queries in DisMaxQueryBuilder (#40956)
This commit implements missing rewrite for the DisMaxQueryBuilder.

Closes #40953
2019-04-10 11:38:16 +02:00
Adrien Grand ecbb588a14 Fix command line to run doc tests in docs/README. (#40959)
It runs `gradle` instead of `gradlew` and has an extra `\`.
2019-04-10 11:35:19 +02:00
Adrien Grand 683cf56982 Update headline of the "removal of types" doc page to match changes in 7.0. (#40868)
Currently it describes what broke in 6.0.
2019-04-10 11:34:29 +02:00
Hendrik Muhs f9018ab11b [ML-DataFrame] create checkpoints on every new run (#40725)
Use the checkpoint service to create a checkpoint on every new run. Expose checkpoints stats on _stats endpoint.
2019-04-10 09:14:11 +02:00
Martijn van Groningen 46b0fdae33
Add realistic hlrc request serialization test base class and (#40362)
changed hlrc ccr request tests to use AbstractRequestTestCase base class.

This way the request classes are tested in a more realistic setting.
Note this change also adds a test dependency on xpack core module.

Similar to #39844 but then for hlrc request serialization tests.

Removed iterators from hlrc parsing tests.
Use empty xcontent registries.

Relates to #39745
2019-04-10 08:00:01 +02:00
Andrei Stefan 0157ebf947 Increase the interval filtering for CURRENT_DATE/TODAY tests (#40999)
(cherry picked from commit a5911330aecee90da5401e468b9976f66a2556f0)
2019-04-10 08:35:09 +03:00
Jason Tedor bb6f060f74
Add log message to forget follower test
This commit adds a log message to help debug failures in a forget
follower test.
2019-04-09 23:33:29 -04:00
Jason Tedor 3aae98f922
Add debug logging for leases sync on recovery test
This commit adds some debug logging for a retention leases sync on
recovery test.
2019-04-09 22:59:22 -04:00
Julie Tibshirani d38214060e Mute ClusterDisruptionIT#testCannotJoinIfMasterLostDataFolder.
Tracked in #41047.
2019-04-09 17:36:21 -07:00
Jason Tedor c0f715b337
Fix Docker build when sourced from artifacts
This commit fixes an issue when the artifact used to build the Docker
image is sourced from artifacts.elastic.co. In particular, the artifact
was not downloaded to the proper location.
2019-04-09 20:33:15 -04:00
Julie Tibshirani 0d5f86a001 Mute RollupIndexerStateTests#testIndexing.
Tracked in #41046.
2019-04-09 17:17:04 -07:00
Julie Tibshirani 0702c72151 Mute DataFrameGetAndGetStatsIT#testGetPersistedStatsWithoutTask.
Tracked in #40963.
2019-04-09 16:39:16 -07:00
Julie Tibshirani 21c5d7e95f Mute CcrRetentionLeaseIT#testRetentionLeasesAreNotBeingRenewedAfterRecoveryCompletes.
Tracked in #39331.
2019-04-09 16:08:44 -07:00
James Rodewig 2e2e11f7b4 [DOCS] Fix broken links for 7.0 release (#41036)
* [DOCS] Remove references to deprecated 'zen.discovery.minimum_master_nodes' setting
2019-04-09 18:20:08 -04:00
Julie Tibshirani cbae617898 Mute IndexFollowingIT#testFollowIndex as we await a fix.
Tracked in #41037.
2019-04-09 14:56:37 -07:00
Julie Tibshirani a417905098 Mute RareClusterStateIT#testDelayedMappingPropagationOnPrimary as we await a fix.
Tracked in #41030.
2019-04-09 13:41:53 -07:00
Julie Tibshirani a0fc2461d7 Mute DedicatedClusterSnapshotRestoreIT#testSnapshotWithStuckNode as we await a fix. 2019-04-09 12:03:33 -07:00
Mark Vieira 1287c7d91f
[Backport] Replace usages RandomizedTestingTask with built-in Gradle Test (#40978) (#40993)
* Replace usages RandomizedTestingTask with built-in Gradle Test (#40978)

This commit replaces the existing RandomizedTestingTask and supporting code with Gradle's built-in JUnit support via the Test task type. Additionally, the previous workaround to disable all tasks named "test" and create new unit testing tasks named "unitTest" has been removed such that the "test" task now runs unit tests as per the normal Gradle Java plugin conventions.

(cherry picked from commit 323f312bbc829a63056a79ebe45adced5099f6e6)

* Fix forking JVM runner

* Don't bump shadow plugin version
2019-04-09 11:52:50 -07:00
Jason Tedor 321f93c4f9
Wait for all listeners in checkpoint listeners test
It could be that we try to shutdown the executor pool before all the
listeners have been invoked. It can happen that one was not invoked if
it timed out and was in the process of being notified that it timed out
on the executor. If we do this shutdown then, a listener will be met
with rejected execution exception. To address this, we first wait until
all listeners have been notified (or timed out) before proceeding with
shutting down the executor.

Relates #40970
2019-04-09 14:27:09 -04:00
Jason Tedor ebba9393c1
Fix unsafe publication of invalid license enforcer (#40985)
The invalid license enforced is exposed to the cluster state update
thread (via the license state listener) before the constructor has
finished. This violates the JLS for safe publication of an object, and
means there is a concurrency bug lurking here. This commit addresses
this by avoiding publication of the invalid license enforcer before the
constructor has returned.
2019-04-09 13:51:37 -04:00
Armin Braun e71db0531e
Fix Race in AsyncTwoPhaseIndexerTests.testStateMachine (#40947) (#41013)
* The step is incremented by the listner in `org.elasticsearch.xpack.core.indexing.AsyncTwoPhaseIndexerTests.MockIndexer#onFinish` after isFinished is set to true, but the test only waited for `isFinished`,
fixed by calling `isFinished` last
* Also made `step` volatile since we are reading it from different thread from the one incrementing it
* Closes #40946
2019-04-09 19:40:01 +02:00
Ed Savage fdc1bdd4d3 [ML][TEST] Fix randomly failing HLRC test (#40973)
Made changes to ensure that unique IDs are generated for model snapshots
used by the deleteExpiredDataTest test in the MachineLearningIT suite.

Previously a sleep of 1s was performed between jobs under the assumption
that this would be sufficient to guarantee that the timestamps used in
the composition of the snapshot IDs would be different.

The new approach is to wait on the condition that the old and new
timestamps are in fact different (to 1s resolution).
2019-04-09 16:55:35 +01:00
Costin Leau 2ac514b909 SQL: Fix catalog filtering in SYS COLUMNS (#40583)
Properly treat '%' as a wildcard for catalog filtering instead of doing
a straight string match.
Table filtering now considers aliases as well.
Add escaping char for LIKE queries with user defined params
Fix monotony of ORDINAL_POSITION
Add integration test for SYS COLUMNS - currently running only inside
single_node since the cluster name is test dependent.
Add pattern unescaping for index names

Fix #40582

(cherry picked from commit 8e61b77d3f849661b7175544f471119042fe9551)
2019-04-09 18:44:27 +03:00
Alexander Reelsen 3b9ab5da04 Fix order of request body search parameter names in documentation (#40777)
The order was random, which made it super hard to find anything. This
changes the order to be alphabetically.
2019-04-09 16:35:45 +02:00
Martijn van Groningen 040a4961c7
Revert "Revert "Change HLRC CCR response tests to use AbstractResponseTestCase base class. (#40257)"" (#40971)
This reverts commit df91237a94fd3d3ae954eb1845c434dda692d087.
2019-04-09 15:18:33 +02:00
Henning Andersen c5a77e5d8c Node repurpose tool docs (#40525)
Added documentation for node repurpose tool and included documentation on how to repurpose nodes safely. Adjusted order of tools in `elasticsearch-node` tool since the repurpose tool is most likely to be used.

Co-Authored-By: David Turner <david.turner@elastic.co>
2019-04-09 15:07:37 +02:00
Christoph Büscher 40638d7b28 [Docs] Delete explanation for completion suggester default analyzer choice (#36720)
The explanation given in the completion suggester documentation why we use the
"simple" analyzer as the default is no longer valid. Since we still use "simple"
as the default, we should just delete the explanation that doesn't fit anymore.

Closes #36715
2019-04-09 13:50:29 +02:00
David Turner 08ecdfe20e Short-circuit rebalancing when disabled (#40966)
Today if `cluster.routing.rebalance.enable: none` then rebalancing is disabled,
but we still execute `balanceByWeights()` and perform some rather expensive
calculations before discovering that we cannot rebalance any shards. In a large
cluster this can make cluster state updates occur rather slowly. With this
change we check earlier whether rebalancing is globally disabled and, if so,
avoid the rebalancing process entirely.

Relates #40942 which was reverted because of egregiously faulty tests.
2019-04-09 07:59:52 +01:00