This is a dependency of #39504
Motivation:
By refactoring `TransportShardBulkAction#shardOperationOnPrimary` to async, we enable using `DeterministicTaskQueue` based tests to run indexing operations. This was previously impossible since we were blocking on the `write` thread until the `update` thread finished the mapping update.
With this change, the mapping update will trigger a new task in the `write` queue instead.
This change significantly enhances the amount of coverage we get from `SnapshotResiliencyTests` (and other potential future tests) when it comes to tracking down concurrency issues with distributed state machines.
The logical change is effectively all in `TransportShardBulkAction`, the rest of the changes is then simply mechanically moving the caller code and tests to being async and passing the `ActionListener` down.
Since the move to async would've added more parameters to the `private static` steps in this logic, I decided to inline and dry up (between delete and update) the logic as much as I could instead of passing the listener + wait-consumer down through all of them.
* This is currently failing with:
```
Test classes are not included in any enabled task (:modules:reindex:test):
* org.elasticsearch.client.documentation.ReindexDocumentationIT
* org.elasticsearch.index.reindex.ManyDocumentsIT
* org.elasticsearch.index.reindex.ReindexClientYamlTestSuiteIT
* org.elasticsearch.index.reindex.ReindexWithoutContentIT
* org.elasticsearch.index.reindex.remote.ReindexFromOldRemoteIT
```
* Same fix as in #38546
This commit changes the bwc builds from a single task for a branch to a
task for each bwc artifact. This reduces the bwc build time when only
needing a specific artifact, for example when running cluster restart
tests on a mac, the windows artifacts or rpm/debs are not needed.
* moved hlrc parsing tests from xpack to hlrc module and removed dependency on hlrc from xpack core
* deprecated old base test class
* added deprecated jdoc tag
* split test between xpack-core part and hlrc part
* added lang-mustache test dependency, this previously came in via
hlrc dependency.
* added hlrc dependency on a qa module
* duplicated ClusterPrivilegeName class in xpack-core, since x-pack
core no longer has a dependency on hlrc.
* replace ClusterPrivilegeName usages with string literals
* moved tests to dedicated to hlrc packages in order to remove Hlrc part from the name and make sure to use imports instead of full qualified class where possible
* remove ESTestCase. from method invocation and use method directly,
because these tests indirectly extend from ESTestCase
When monitoring exporters are all disabled, which must be done
explicitly, _and_ monitoring collection is enabled, then
any call to `_xpack/monitoring/_bulk` will create a task that
never closes _and_ ES collection will stop happening because
a semaphore is never marked as completed.
This also simplifies the async `ExportBulk` code by removing the
third step (second async step, `close`) entirely because it was
entirely unnecessary by both implementations.
* [ML] Add validation that rejects duplicate detectors in PutJobAction
Closes#39704
* Add YML integration test for duplicate detectors fix.
* Use "== false" comparison rather than "!" operator.
* Refine error message to sound more natural.
* Put job description in square brackets in the error message.
* Use the new validation in ValidateJobConfigAction.
* Exclude YML tests for new validation from permission tests.
This PR generates deprecation log entries for each Role Descriptor,
used for building a Role, when the Role Descriptor grants more privileges
for an alias compared to an index that the alias points to. This is done in
preparation for the removal of the ability to define privileges over aliases.
There is one log entry for each "role descriptor name"-"alias name" pair.
On such a notice, the administrator is expected to modify the Role Descriptor
definition so that the name pattern for index names does not cover aliases.
Caveats:
* Role Descriptors that are not used in any authorization process,
either because they are not mapped to any user or the user they are mapped to
is not used by clients, are not be checked.
* Role Descriptors are merged when building the effective Role that is used in
the authorization process. Therefore some Role Descriptors can overlap others,
so even if one matches aliases in a deprecated way, and it is reported as such,
it is not at risk from the breaking behavior in the current role mapping configuration
and index-alias configuration. It is still reported because it is a best practice to
change its definition, or remove offending aliases.
As empty string has a certain meaning, the JDBC driver returns an empty
set instead for better client compatibility.
Fix#41028
(cherry picked from commit 4cbafa585b7a514eb6c156606dd516324cd3980a)
changed hlrc ccr request tests to use AbstractRequestTestCase base class.
This way the request classes are tested in a more realistic setting.
Note this change also adds a test dependency on xpack core module.
Similar to #39844 but then for hlrc request serialization tests.
Removed iterators from hlrc parsing tests.
Use empty xcontent registries.
Relates to #39745
This commit fixes an issue when the artifact used to build the Docker
image is sourced from artifacts.elastic.co. In particular, the artifact
was not downloaded to the proper location.
* Replace usages RandomizedTestingTask with built-in Gradle Test (#40978)
This commit replaces the existing RandomizedTestingTask and supporting code with Gradle's built-in JUnit support via the Test task type. Additionally, the previous workaround to disable all tasks named "test" and create new unit testing tasks named "unitTest" has been removed such that the "test" task now runs unit tests as per the normal Gradle Java plugin conventions.
(cherry picked from commit 323f312bbc829a63056a79ebe45adced5099f6e6)
* Fix forking JVM runner
* Don't bump shadow plugin version
It could be that we try to shutdown the executor pool before all the
listeners have been invoked. It can happen that one was not invoked if
it timed out and was in the process of being notified that it timed out
on the executor. If we do this shutdown then, a listener will be met
with rejected execution exception. To address this, we first wait until
all listeners have been notified (or timed out) before proceeding with
shutting down the executor.
Relates #40970
The invalid license enforced is exposed to the cluster state update
thread (via the license state listener) before the constructor has
finished. This violates the JLS for safe publication of an object, and
means there is a concurrency bug lurking here. This commit addresses
this by avoiding publication of the invalid license enforcer before the
constructor has returned.
* The step is incremented by the listner in `org.elasticsearch.xpack.core.indexing.AsyncTwoPhaseIndexerTests.MockIndexer#onFinish` after isFinished is set to true, but the test only waited for `isFinished`,
fixed by calling `isFinished` last
* Also made `step` volatile since we are reading it from different thread from the one incrementing it
* Closes#40946
Made changes to ensure that unique IDs are generated for model snapshots
used by the deleteExpiredDataTest test in the MachineLearningIT suite.
Previously a sleep of 1s was performed between jobs under the assumption
that this would be sufficient to guarantee that the timestamps used in
the composition of the snapshot IDs would be different.
The new approach is to wait on the condition that the old and new
timestamps are in fact different (to 1s resolution).
Properly treat '%' as a wildcard for catalog filtering instead of doing
a straight string match.
Table filtering now considers aliases as well.
Add escaping char for LIKE queries with user defined params
Fix monotony of ORDINAL_POSITION
Add integration test for SYS COLUMNS - currently running only inside
single_node since the cluster name is test dependent.
Add pattern unescaping for index names
Fix#40582
(cherry picked from commit 8e61b77d3f849661b7175544f471119042fe9551)
Added documentation for node repurpose tool and included documentation on how to repurpose nodes safely. Adjusted order of tools in `elasticsearch-node` tool since the repurpose tool is most likely to be used.
Co-Authored-By: David Turner <david.turner@elastic.co>
The explanation given in the completion suggester documentation why we use the
"simple" analyzer as the default is no longer valid. Since we still use "simple"
as the default, we should just delete the explanation that doesn't fit anymore.
Closes#36715
Today if `cluster.routing.rebalance.enable: none` then rebalancing is disabled,
but we still execute `balanceByWeights()` and perform some rather expensive
calculations before discovering that we cannot rebalance any shards. In a large
cluster this can make cluster state updates occur rather slowly. With this
change we check earlier whether rebalancing is globally disabled and, if so,
avoid the rebalancing process entirely.
Relates #40942 which was reverted because of egregiously faulty tests.
Armin Braun
James Rodewig
Mark Vieira
Henning Andersen
Dimitris Athanasiou
Jason Tedor
Ryan Ernst
Hendrik Muhs
Zachary Tong
Martijn van Groningen
Chris Earle
Christoph Büscher
Zachary Tong
Przemysław Witek
Dimitrios Liappis
Ed Savage
Albert Zaharovits
Costin Leau
Jim Ferenczi
Adrien Grand
Andrei Stefan
Julie Tibshirani
Ed Savage
Alexander Reelsen
David Turner