This commit marks a failing test as awaits fix. The test is failing due
to a primary shard not knowing its own local checkpoint in the global
checkpoint tracker after recovery. If such a shard becomes primary after
promotion, and is then subsequently relocated, it can lead to a
violation of an assertion that when the primary context is transferred
the knowledge of all in-sync local checkpoints is consistent with the
global checkpoint on the relocation target.
Relates #25415
This commit removes the default path settings for data and logs. With
this change, we now ship the packages with these settings set in the
elasticsearch.yml configuration file rather than going through the
default.path.data and default.path.logs dance that we went through in
the past.
Relates #25408
Today we load plugins reflectively, looking for constructors that
conform to specific signatures. This commit tightens the reflective
operations here, not allowing plugins to have ambiguous constructors.
Relates #25405
This commit removes path.conf as a valid setting and replaces it with a
command-line flag for specifying a non-default path for configuration.
Relates #25392
This commit removes an abstraction that was introduced when introducing
the primary context. As this abstraction is used in exactly one place,
we simply make that abstraction local to its usage so that we do not
accumulate yet another general abstraction with exactly one usage.
Relates #25402
This commit updates some assertions in the primary context sealing test
after the restriction on updating allocation IDs from master and
updating global checkpoint on replica while sealed were removed.
* Introduce primary context
The target of a primary relocation is not aware of the state of the
replication group. In particular, it is not tracking in-sync and
initializing shards and their checkpoints. This means that after the
target shard is started, its knowledge of the replication group could
differ from that of the relocation source. In particular, this differing
view can lead to it computing a global checkpoint that moves backwards
after it becomes aware of the state of the entire replication
group. This commit addresses this issue by transferring a primary
context during relocation handoff.
* Fix test
* Add assertion messages
* Javadocs
* Barrier between marking a shard in sync and relocating
* Fix misplaced call
* Paranoia
* Better latch countdown
* Catch any exception
* Fix comment
* Fix wait for cluster state relocation test
* Update knowledge via upate local checkpoint API
* toString
* Visibility
* Refactor permit
* Push down
* Imports
* Docs
* Fix compilation
* Remove assertion
* Fix compilation
* Remove context wrapper
* Move PrimaryContext to new package
* Piping for cluster state version
This commit adds piping for the cluster state version to the global
checkpoint tracker. We do not use it yet.
* Remove unused import
* Implement versioning in tracker
* Fix test
* Unneeded public
* Imports
* Promote on our own
* Add tests
* Import
* Newline
* Update comment
* Serialization
* Assertion message
* Update stale comment
* Remove newline
* Less verbose
* Remove redundant assertion
* Tracking -> in-sync
* Assertions
* Just say no
Friends do not let friends block the cluster state update thread on
network operations.
* Extra newline
* Add allocation ID to assertion
* Rename method
* Another rename
* Introduce sealing
* Sealing tests
* One more assertion
* Fix imports
* Safer sealing
* Remove check
* Remove another sealed check
You can continue a test started in a previous snippet by marking the
next snippet with `// TEST[continued]`. The trouble is, if you mark the
first snippet in a file with `// TEST[continued]` you'd get difficult
to predict behavior because you'd continue the test started in another
file. This will usually create a test that fails the build but it isn't
easy to track down what you did wrong. This commit catches this
scenario up front and fails the build with a useful error message.
Similarly, if you put `// TEST[continued]` directly after a
`// TESTSETUP` section then the docs tests will fail to run but the
error message did not point you to the `// TEST[continued]` snippet.
This commit catches this scenario up front as well and fails the build
with a useful error message.
This removes the remaining usage of `mapping.single_type` from the parent join
module and moves it's bwc test to the mixed cluster tests
Relates to #24961
Relates to #20257
The following token filters were moved: stemmer, stemmer_override, kstem, dictionary_decompounder, hyphenation_decompounder, reverse, elision and truncate.
Relates to #23658
While real secure settings (ie an ES keystore) cannot be merged
together, mocked secure settings can and need to be sometimes merged.
This commit adds a merge method to allow tests to merge together
multiple instances of secure settings.
This change removes the remaining explicitly specified `index.mapper.single_type`
settings from tests in order to allow the removal of the setting.
This is the already approved part of #25375 broken out to simplfiy reviews on
When Log4j 2 was introduced, we removed support for the system property
es.logger.prefix. Yet, some code was left behind. This commit removes
that dead code.
Relates #25377
* Remove the setting from the yml tests and replace with tests using
`join` field. We can't use the setting in yml tests without lots of
backflips but we have `ReindexParentChildTests` for the coverage.
There weren't tests for `join` field with reindex before this. Adding
these tests discovered #25363.
* Remove the setting from `ReindexParentChildTests` and replace with
`index.version.created=V_5_6_0`. This test can be entirely removed
when legacy parent/child support is dropped from core.
* Port the yml tests that set _parent into integ tests so they
can set the index created version. These tests can be removed
when we drop support for _parent in core.
* Port a delete-by-query test for filtering based on type to an
`ESIntegTestCase` so it can use `index.version.created=5.6.0` to
setup documents of multiple types. This whole feature can be dropped
when we no longer support multiple types per index.
Relates to #24961
Added unit test coverage for GlobalOrdinalsSignificantTermsAggregator, GlobalOrdinalsSignificantTermsAggregator.WithHash, SignificantLongTermsAggregator and SignificantStringTermsAggregator.
Removed integration test.
Relates #22278
This change cleans up remaining tests to not use index.mapping.single_type=false
but instead where applicable use a single type or markt the index as created
with a pre 6.x version.
Yet, there is still on leftover in the client tests that needs special attention.
See `org.elasticsearch.client.SearchIT`
Relates to #24961
OldIndexBackwardsCompatibilityIT#testOldClusterStates tested whether global and index metadata could be read from data directory,
this can also be tested in full cluster qa test that checks cluster state via api.
Relates to #24939
When JAVA_HOME is not set we try to detect the location of Java. If its
location contains a space, due to a lack of quoting we will be
unsuccessful in invoking Java. This commit adds the necessary quoting to
handle this case.
Relates #23822
Tests were randomly assigning `targetField` to an existing field that was an array,
causing path resolution issues. This PR fixes those tests
Closes#25346 & #25348
`InternalEngineTests.testConcurrentWritesAndCommits` can be very heavy on disks
if threads are slow and the main thread keeps on pulling commit points holding on
to many many segments. This commit adds some quadratic backoff to not pile up too many
commits and to make sure indexing threads can make progress. This also now doesn't do
busy waiting but waits on a latch with a timeout.
Closes#25110
Custom whitelists in Painless will need to allow classes to be augmented beyond the currently hard-coded Augmentation class tied to Painless directly. This change allows any class to specify an augmentation on a Painless struct using an appropriate static method. Changes to loading the whitelist have also been created to allow for this specification of a different class for augmentation.
In #24379 we added ability to upgrade templates on full cluster startup. This PR invokes the same update procedure also when a new node first joins the cluster allowing to update templates on a rolling cluster restart as well.
Closes#24680
When shrinking an index we initialize its max unsafe auto ID timestamp
to the maximum of the max unsafe auto ID timestamps on the source
shards.
Relates #25356
#25147 added the translog deletion policy but didn't enable it by default. This PR enables a default retention of 512MB (same maximum size of the current translog) and an age of 12 hours (i.e., after 12 hours all translog files will be deleted). This increases to chance to have an ops based recovery, even if the primary flushed or the replica was offline for a few hours.
In order to see which parts of the translog are committed into lucene the translog stats are extended to include information about uncommitted operations.
Views now include all translog ops and guarantee, as before, that those will not go away. Snapshotting a view allows to filter out generations that are not relevant based on a specific sequence number.
Relates to #10708
This change cleans up core tests to not use `index.mapping.single_type=false`
but instead where applicable use a single type or markt the index as created
with a pre 6.x version.
Relates to #24961
Due to limitations with CreateProcessW on Windows (ultimately used by
ProcessBuilder) with respect to maximum path lengths, we need to get the
short path name for any native controllers before trying to start them
in case the absolute path exceeds the maximum path length. This commit
uses JNA to invoke the necessary Windows API for this to start the
native controller using the short path.
To be precise about the limitation here, the MSDN docs for
CreateProcessW say for the command line parameter:
>The command line to be executed. The maximum length of this string is
>32,768 characters, including the Unicode terminating null character. If
>lpApplicationName is NULL, the module name portionof lpCommandLine is
>limited to MAX_PATH characters.
This is exactly how the Windows implementation of Process in the JDK
invokes CreateProcessW: with the executable name (lpApplicationName) set
to NULL.
Relates #25344
Most notable changes:
- better update concurrency: LUCENE-7868
- TopDocs.totalHits is now a long: LUCENE-7872
- QueryBuilder does not remove the boolean query around multi-term synonyms:
LUCENE-7878
- removal of Fields: LUCENE-7500
For the `TopDocs.totalHits` change, this PR relies on the fact that the encoding
of vInts and vLongs are compatible: you can write and read with any of them as
long as the value can be represented by a positive int.
The `document_type` parameter is no longer required to be specified,
because by default from 6.0 only a single type is allowed. (`index.mapping.single_type` defaults to `true`)
Martijn van Groningen
Daniel Mitterdorfer
Jason Tedor
Deb Adair
Ryan Ernst
Alexander Kazakov
Nik Everett
Simon Willnauer
Igor Motov
Christoph Büscher
markharwood
Boaz Leskes
Tanguy Leroux
dkimdon
Tal Levy
Jack Conradson
Yannick Welsch
Adrien Grand