Restoring a shard from snapshot throws the primary back in time violating assumptions and bringing the validity of global checkpoints in question. To avoid problems, we should make sure that a shard that was restored will never be the source of an ops based recovery to a shard that existed before the restore. To this end we have introduced the notion of `histroy_uuid` in #26577 and required that both source and target will have the same history to allow ops based recoveries. This PR make sure that a shard gets a new uuid after restore.
As suggested by @ywelsch , I derived the creation of a `history_uuid` from the `RecoverySource` of the shard. Store recovery will only generate a uuid if it doesn't already exist (we can make this stricter when we don't need to deal with 5.x indices). Peer recovery follows the same logic (note that this is different than the approach in #26557, I went this way as it means that shards always have a history uuid after being recovered on a 6.x node and will also mean that a rolling restart is enough for old indices to step over to the new seq no model). Local shards and snapshot force the generation of a new translog uuid.
Relates #10708Closes#26544
This commit moves the pre-6.0 node checkpoint constant from
SequenceNumbersService to SequenceNumbers so it can chill with the other
sequence number-related constants.
Relates #26690
Request bodys that only consists of a String value can lead to endless loops in the
parser of several rest requests like e.g. `_count`. Up to 5.2 this seems to have been caught
in the logic guessing the content type of the request, but since then it causes the node to
block. This change introduces checks for receiving a valid xContent object before starting the
parsing in RestActions#parseTopLevelQueryBuilder().
Closes#26083
Adds several small whitelist data structures and a new Whitelist class to separate the idea of loading a whitelist from the actual Painless Definition class. This is the first step of many in allowing users to define custom whitelists per context. Also supports the idea of loading multiple whitelists from different sources for a single context.
* Add a version constant for 5.6.2 so that the 5.6.1 constant
represents the 5.6.1 release and the 5.6.2 constant represents
the unreleased 5.6 branch.
Today we can't validate the array length in `InputStreamStreamInput` since
we can't rely on `InputStream.available` yet in some situations we know
the size of the stream and can apply additional validation.
After recovery completes from a primary, we now update the local
knowledge on the primary of the global checkpoint on the recovery
target. However if this occurs concurrently with a relocation, an
assertion could trip that we are no longer in primary mode. As this
local knowledge should only be tracked when we are in primary mode,
updating this local knowledge should be done under a permit. This commit
causes that to be the case.
Relates #26666
When checking that the global checkpoint on the primary is consistent
with the local checkpoints of the in-sync shards, we have to filter
pre-6.0 nodes from the check or the invariant will trivially trip. This
commit filters these nodes out when checking this invariant.
Relates #26666
This commit adds a skip for the bad request REST test on pre-6.0
nodes. Previously, a request for /_(.*) where $1 is not an existing
endpoint would return a 404. This is because the request would be
treated as a get index request for an index named _$1. However, an index
can never start with "_" so logic was added to detect this and return a
400 instead as this should be treated as a bad request. During the
mixed-cluster BWC tests, a node running pre-6.0 code will still return a
404 though. Therefore, this test needs to skipped in such a
mixed-cluster scenario.
This commit reenables the BWC tests after they were disabled for
backporting the change to track global checkpoints of shard copies on
the primary.
Relates #26666
This commit adds local tracking of the global checkpoints on all shard
copies when a global checkpoint tracker is operating in primary
mode. With this, we relay the global checkpoint on a shard copy back to
the primary shard during replication operations. This serves as another
step towards adding a background sync of the global checkpoint to the
shard copies.
Relates #26666
In this test, 260b is replaced by the regexp \d+b
but the test sometimes produces results like 1.1kb
so this commit adapts the regexp to match values
with decimals
The discovery-file plugin was not config path aware, so it always picked
up the default config path (from Elasticsearch home) rather than a
custom config path. This commit fixes the discovery-file plugin to
respect a custom config path.
Relates #26662
This commit adds validation to the resolving of indexes in the wildcard
expression resolver. It no longer throws a 404 Not Found when resolving
invalid indices. It throws a 400 instead, as it is an invalid
index. This was the behavior of 5.x.
Initialize the default stop-tags in `KuromojiPartOfSpeechFilterFactory` if the
`stoptags` are not given in the config. Also adding a test which checks that
part-of-speech tokens are removed when using the kuromoji_part_of_speech
filter.
This avoids messages with malformed URLs, like
"org.elasticsearch.client.ResponseException: PUT
http://127.0.0.1:9502customer: HTTP/1.1 400 Bad Request".
Relates #26564
The new ops based recovery, introduce as part of #10708, is based on the assumption that all operations below the global checkpoint known to the replica do not need to be synced with the primary. This is based on the guarantee that all ops below it are available on primary and they are equal. Under normal operations this guarantee holds. Sadly, it can be violated when a primary is restored from an old snapshot. At the point the restore primary can miss operations below the replica's global checkpoint, or even worse may have total different operations at the same spot. This PR introduces the notion of a history uuid to be able to capture the difference with the restored primary (in a follow up PR).
The History UUID is generated by a primary when it is first created and is synced to the replicas which are recovered via a file based recovery. The PR adds a requirement to ops based recovery to make sure that the history uuid of the source and the target are equal. Under normal operations, all shard copies will stay with that history uuid for the rest of the index lifetime and thus this is a noop. However, it gives us a place to guarantee we fall back to file base syncing in special events like a restore from snapshot (to be done as a follow up) and when someone calls the truncate translog command which can go wrong when combined with primary recovery (this is done in this PR).
We considered in the past to use the translog uuid for this function (i.e., sync it across copies) and thus avoid adding an extra identifier. This idea was rejected as it removes the ability to verify that a specific translog really belongs to a specific lucene index. We also feel that having a history uuid will serve us well in the future.
Removing several occurrences of this typo in the docs and javadocs, seems to be
a common mistake. Corrections turn up once in a while in PRs, better to correct
some of this in one sweep.
* Fix percolator highlight sub fetch phase to not highlight query twice
The PercolatorHighlightSubFetchPhase does not override hitExecute and since it extends HighlightPhase the search hits
are highlighted twice (by the highlight phase and then by the percolator). This does not alter the results, the second highlighting
just overrides the first one but this slow down the request because it duplicates the work.
This commit refactors the bootstrap checks into a single result object
that encapsulates whether or not the check passed, and a failure message
if the check failed. This simpifies the checks, and enables the messages
to more easily be based on the state used to discern whether or not the
check passed.
Relates #26637
This exposes the node settings and the persistent part of the cluster state to the
bootstrap checks to allow plugins to enforce certain preconditions based on the
recovered state.
After backporting the script_field soft limit to the 6.x branches, this test can
now also run in a mixed cluster.
Relates to #26598
enter the commit message for your changes. Lines starting
This commit pushes the allocation ID down through to the global
checkpoint tracker at construction rather than when activated as a
primary.
Relates #26630
Today we have all non-plugin mappers in core. I'd like to start moving those
that neither map to json datatypes nor are very frequently used like `date` or
`ip` to a module.
This commit creates a new module called `mappers-extra` and moves the
`scaled_float` and `token_count` mappers to it. I'd like to eventually move
`range` fields there but it's more complicated due to their intimate
relationship with range queries.
Relates #10368
Requesting to many script_fields in a search request can be costly
because of script execution. This change introduces a soft limit on the number
of script fields that are allowed per request. The setting can be
changed per index using the index.max_script_fields setting.
Relates to #26390
This PR removes the vInt that precedes every value in order to know how long
they are. Instead the query takes an enum that tells how to compute the length
of values: for fixed-length data (ip addresses, double, float) the length is a
constant while longs and integers use a variable-length representation that
allows the length to be computed from the encoded values.
Also the encoding of ints/longs was made a bit more efficient in order not to
waste 3 bits in the header. As a consequence, values between -8 and 7 can now
be encoded on 1 byte and values between -2048 and 2047 can now be encoded on 2
bytes or less.
Closes#26443
This commit adds a dependency to the install module task on the task
that builds the module. This is needed for standalone integration
tests that require other modules to be installed. Without this, we do
not have a guarantee that the module is bundled.
Boaz Leskes
Martijn van Groningen
Jason Tedor
Armin Braun
Jack Conradson
Tal Levy
Nik Everett
Simon Willnauer
Tanguy Leroux
Peter Dyson
Michael Basnight
Dimitrios Liappis
kel
Claudio Bley
markwalkom
Itamar Syn-Hershko
lcawley
Bernd
Christoph Büscher
David Pilato
Daniel A. Ochoa
Jim Ferenczi
Adrien Grand