This commit moves the gradle wrapper jar file to a hidden directory, so
that it does not clutter the top level names seen when doing an ls in
the project. The actual jar file is never manually edited, and only
changed by running `./gradlew wrapper ...` so it is not important for
this directory to be "visible".
We disable the reindex-from-old tests if we're running on windows or in
a directory that contains a space. This adds a warning to the logs when
we do that so that you can tell that it happened. This will be nice to
have when looking at CI and will be a hint to anyone developing locally.
We still don't have a strong reason for the failures of
testDoNotRenewSyncedFlushWhenAllSealed and
testSyncedFlushSkipOutOfSyncReplicas.
This commit adds debug logging for these two tests.
Since we disable allocation using persistent settings, we should be consistent and remove
the setting from the persistent storage. Otherwise an accidental restart will lead for shards
not being allocated.
Relates to #28757
Since we disable allocation using persistent settings, we should be consistent and remove
the setting from the persistent storage. Otherwise an accidental restart will leed for shards
not being allocated.
Relates to https://github.com/elastic/elasticsearch/pull/28757
Today when an index is created from shrinking or splitting an existing
index, the target index inherits almost none of the source index
settings. This is surprising and a hassle for operators managing such
indices. Given this is the default behavior, we can not simply change
it. Instead, we start by introducing the ability to copy settings. This
flag can be set on the REST API or on the transport layer and it has the
behavior that it copies all settings from the source except non-copyable
settings (a property of a setting introduced in this
change). Additionally, settings on the request will always override.
This change is the first step in our adventure:
- this flag is added here in 7.0.0 and immediately deprecated
- this flag will be backported to 6.4.0 and remain deprecated
- then, we will remove the ability to set this flag to false in 7.0.0
- finally, in 8.0.0 we will remove this flag and the only behavior will
be for settings to be copied
xpack core contains a fork of `Cron` from quartz who's javadoc has a
`<table>` with non-html5 compatible stuff. This html5ifies the table and
switches the `:x-pack:plugin:core` project to building javadoc with
HTML5.
Just like `ElasticsearchException`, the inner most
`XContentParseException` tends to contain the root cause of the
exception and show be show to the user in the `root_cause` field.
The effectively undoes most of the changes that #29373 made to the
`root_cause` for parsing exceptions. The `type` field still changes from
`parse_exception` to `x_content_parse_exception`, but this seems like a
fairly safe change.
`ElasticsearchWrapperException` *looks* tempting to implement this but
the behavior isn't quite right. `ElasticsearchWrapperExceptions` are
entirely unwrapped until the cause no longer
`implements ElasticsearchWrapperException` but `XContentParseException`
should be unwrapped until its cause is no longer an
`XContentParseException` but no further. In other words,
`ElasticsearchWrapperException` are unwrapped one step too far.
Closes#30261
* Adds ClusterState to AsyncActionStep.performAction
This is needed so a new step can be created for the shrink action which
can select a node to allocate to based on the current routing rules and
the node attributes on teh discovery nodes.
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/AsyncActionStep.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/DeleteStep.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/ForceMergeStep.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/RolloverStep.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/ShrinkSetAliasStep.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/ShrinkStep.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/UpdateSettingsStep.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/DeleteStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/ForceMergeStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/RolloverStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/ShrinkSetAliasStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/ShrinkStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/UpdateSettingsStepTests.java
x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/IndexLifecycleRunner.java
x-pack/plugin/index-lifecycle/src/main/java/org/elasticsearch/xpack/inde
xlifecycle/IndexLifecycleService.java
x-pack/plugin/index-lifecycle/src/test/java/org/elasticsearch/xpack/inde
xlifecycle/IndexLifecycleRunnerTests.java
* Adds single node allocation to shrink
This change adds two new steps as the first steps of the shrink action:
1. A `SetSingleNodeAllocateStep` which:
1. Determines which of the active nodes match the existing index
allocation rules
2. Randomly (using Randomness so its deterministic for testing)
picks one of the matching nodes
3. Updates the index settings to add a require allocation rule for
the node that was picked (using the
`index.routing.allocation.require._name` setting)
2. An `AllocationRoutedStep` which ensures that at least one copy of
each shard is allocated according to the new allocation rules
Note that this change also modifies the `AllocationRoutedStep` to add a
boolean field which determines whether the allocation is complete when
at least one copy of each shard matches the allocation rulees or if it
needs to wait for all shard copies to be allocated according to the
rules.
Lastly, a `randomStepKey()` method is added to `AbstractStepTestCase`
for convenience.
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/AllocateAction.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/AllocationRoutedStep.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/SetSingleNodeAllocateStep.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/ShrinkAction.java
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/ShrinkSetAliasStep.java
x-pack/plugin/core/src/test/java/org/elasticsearch/action/admin/indices/
settings/put/UpdateSettingsTestHelper.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/AbstractStepTestCase.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/AllocationRoutedStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/SetSingleNodeAllocateStepTests.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/ShrinkActionTests.java
* Fixes AllocationRoutedStep when `waitForAllShardCopies=false`
This change fixes `AllocationRoutedStep` so that when
`waitForAllShardCopies=false` we wait for any shard copy of each shard
to be allocated according to the allocation rules rather than
specifically the primary of each shard.
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/AllocationRoutedStep.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/AllocationRoutedStepTests.java
* Corrects Licence headers and typo
x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/indexlifec
ycle/SetSingleNodeAllocateStep.java
x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/indexlifec
ycle/SetSingleNodeAllocateStepTests.java
This commit refactors the DataStreamDiagnostics class
achieving the following advantages:
- simpler code; by encapsulating the moving bucket histogram
into its own class
- better performance; by using an array to store the buckets
instead of a map
- explicit handling of gap buckets; in preparation of fixing #30080
Remove double if depending on the Result value. It makes little sense to
pass in a boolean flag based on a Result value that we already have,
if that internally is represented again as a `Result` value.
Also changed the `Result` `lowercase` instance member to be computed
based on `name()` instead of `toString()` which is safer and to use
`Locale.ROOT` instead of `Locale.ENGLISH`
Starting with the refactoring in https://github.com/elastic/elasticsearch/pull/22778 (released in 5.3) we may fail to properly replicate operation when a mapping update on master fails. If a bulk
operations needs a mapping update half way, it will send a request to the master before continuing
to index the operations. If that request times out or isn't acked (i.e., even one node in the cluster
didn't process it within 30s), we end up throwing the exception and aborting the entire bulk. This is
a problem because all operations that were processed so far are not replicated any more to the
replicas. Although these operations were never "acked" to the user (we threw an error) it cause the
local checkpoint on the replicas to lag (on 6.x) and the primary and replica to diverge.
This PR does a couple of things:
1) Most importantly, treat *any* mapping update failure as a document level failure, meaning only
the relevant indexing operation will fail.
2) Removes the mapping update callbacks from `IndexShard.applyIndexOperationOnPrimary` and
similar methods for simpler execution. We don't use exceptions any more when a mapping
update was successful.
I think we need to do more work here (the fact that a single slow node can prevent those mappings
updates from being acked and thus fail operations is bad), but I want to keep this as small as I can
(it is already too big).
[test] add java packaging test project
Adds a project for building and running packaging tests written in java
for portability. The vagrant tasks use jars on the packagingTest
configuration, which are built in the same project. No tests are added
yet.
Corresponding changes are not made to :x-pack:qa:vagrant because the
java packaging tests will all be consolidated into one project.
For #26741
The overall NOTICE file for the ML X-Pack module should
include the notices from the 3rd party C++ components as
well as the 3rd party Java components.
Currently, the only way to get the REST response for the `/_cluster/state`
call to return the `cluster_uuid` is to request the `metadata` metrics,
which is one of the most expensive response structures. However, external
monitoring agents will likely want the `cluster_uuid` to correlate the
response with other API responses whether or not they want cluster
metadata.
`javadoc` will switch from detaulting to html4 to html5 in "a future
release". We should get ahead of it so we're not surprised. Also, HTML5
is the future! Er, the present. Anyway, this follows up from #30220 to
make the Javadoc for two of the four remaining projects HTML5
compatible.
Add yet another warning about data loss to the introductory paragraph about the
unsafe commands. Also move this paragraph next to the details of the unsafe
commands, below the section on the `retry_failed` flag.
Be more specific about how to use the URI parameters and in-body flags.
Clarify statements about when rebalancing takes place (i.e. it respects
settings)
Resolves#16113.
Today when a resize operation is performed, we copy the analysis,
similarity, and sort settings from the source index. It is possible for
the resize request to include additional index settings including
analysis, similarity, and sort settings. We reject sort settings when
validating the request. However, we silently ignore analysis and
similarity settings on the request that are already set on the source
index. Since it is possible to change the analysis and similarity
settings on an existing index, this should be considered a bug and the
sort of leniency that we abhor. This commit addresses this bug by
allowing the request analysis/similarity settings to override the
existing analysis/similarity settings on the target.
Similarly to what has been done in for the repository-s3 plugin,
this commit moves the fixture test into a dedicated
repository-gcs/qa/google-cloud-storage project.
It also exposes some environment variables which allows to
execute the integration tests against the real Google Cloud
Storage service. When the environment variables are not
defined, the integration tests are executed using the fixture
added in #28788. Related to #29349.
The `testDeleteSnapshotWithMissingIndexAndShardMetadata` test uses an
obsolete repository directory structure based on index names instead of
UUIDs. Because it swallows exceptions when deleting test files the test
never failed when the directory structure changed.
This commit fixes the test to use the right directory structure and file
names and to not swallow exceptions anymore.
The REST resize handlers for shrink/split operations are effectively the
same code with a minor difference. This commit collapse these handlers
into a single base class.
This *mostly* silences `javadoc`'s warning about defaulting to
generating html4 files by enabling generating html5 file for the
projects for which that works. It didn't work in a half dozen projects,
about half of which I've fixed in this PR, entirely by replacing
`<tt>thing</tt>` with `{@code thing}`.
There are a few remaining projects that contain javadoc with invalid
html5. I'll fix those projects in a followup.
This is a code-tidying PR, a little side adventure while working on
another change. Previously only shrink request existed but when the
ability to split indices was added, shrink and split were done together
under a single request object: the resize request object. However, the
code inherited the legacy name in the naming of some variables. This
commit cleans this up.
A previous change modified the output of the thread pool info contained
in the nodes info API. This commit adds a note to the migration docs for
this change.
Since #28049, only fully initialized shards are received write requests.
This enhancement allows us to handle all exceptions. In #28571, we
started strictly handling shard-not-available exceptions and tried to
keep the way we report replication errors to users by only reporting if
the error is not shard-not-available exceptions. However, since then we
unintentionally always log warn for all exception. This change restores
to the previous behavior which logs warn only if an exception is not a
shard-not-available exception.
Relates #28049
Relates #28571
We *think* that #28600 is caused by warnings not being collected during
one of the fan out phases of search but we're not 100% sure how this is
happening. This commit drops the number of shards used for the test to 1
so there *isn't* a fan out phase. If this makes the issue go away we'll
have more information.
This folds the `:qa:reindex-from-old` project into the `:modules:reindex`
project. This should speed up the build marginally by removing a single
clsuter start up at the cost of having to wait for old versions of
Elasticsearch to start up when checking reindex's integration tests.
Those don't take that long so this feels worth it.
We had a number of awaitsFix links that weren't updated after the xpack
merge.
Where possible I changed the links to the new locations, but in some
circumstances the original ticket was closed (suggesting the awaitsfix
should be removed) or was otherwise unclear the status.
Lisa Cawley
Ryan Ernst
Nik Everett
Nhat Nguyen
Boaz Leskes
David Turner
Diwas Joshi
Jason Tedor
Colin Goodheart-Smithe
Dimitris Athanasiou
Luca Cavanna
Andy Bristol
Paul Sanwald
David Roberts
Chris Earle
Tanguy Leroux
Julie Tibshirani
Sue Gallagher
Zachary Tong