This adds a force-merge step to the searchable snapshot action, enabled by default,
but parameterizable using the `force_merge-index" optional boolean.
eg.
```
PUT _ilm/policy/my_policy
{
"policy": {
"phases": {
"cold": {
"actions": {
"searchable_snapshot" : {
"snapshot_repository" : "backing_repo",
"force_merge_index": true
}
}
}
}
}
}
```
(cherry picked from commit d0a17b2d35f1b083b574246bdbf3e1929471a4a9)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
"Transitive" is technically ok here but it's an overloaded word and it's
not immediately clear which meaning is intended so this log message
always makes me do a double-take. I think both "transient" and
"transitory" are clearer, with "transient" being the usual choice.
This commit makes IpFieldMapper extend ParametrizedFieldMapper. It also
updates the IpFieldMapper docs to add the ignore_malformed parameter,
which was not previously documented.
This suite is still occasionally failing with a timeout on macOS.
Suggest further increasing this timeout until this suite is broken up.
Relates #58071
Sometimes this test would refresh the disk stats so quickly that it hit
the refresh rate limiter even though it was almost completely disabled.
This commit allows the rate limiter to be completely disabled.
Closes#60587
Closes#60853. After upgrading to CentOS 8, the behaviour of chroot has
subtly changed. Now we have to explicitly set the GID in order to get
the previous behaviour of creating files with GID 0.
The Elasticsearch `run` task in Gradle is backed by testclusters
spinning up a node and waiting indefinitely on output from that node.
However, this thread currently waits for an interrupt to exit. If
Elasticsearch fails to start, we hang indefinitely. This commit adds a
check after each poll of the process stdout to see if it is still alive,
and fails the build if it has died.
closes#60761
Several /proc files are expected to contain a single line. We assert on
this in tests, but the contents of the file are lost and the assertion
therefore lacks important information to debug why the file appeared to
have multiple lines. This commit dumps the contents of the file on
assertion failure.
relates #59284
Though bats tests were recently removed, there remains a few unnecessary
setups needed for those tests in our vagrant files, as well as CI setup.
Additionally, we no longer rely on the vagrant images setting a
JAVA_HOME, instead relying on DistroTestPlugin to pull the appropriate
jdk when testing with no-jdk distributions.
The ReloadSecureSettingsIT uses latches to ensure coordination across
requests to the underlying in memory cluster. However, in the case of an
expected failure, if the assertion fails, the latch will never be
counted down, and will cause the test to hang indefinitely. This commit
ensures the latch is always counted down with a try/finally.
relates #51546
Collapse search queries that sort by a field can throw
an ArrayStoreException due to a bug in the [sort optimization](https://github.com/elastic/elasticsearch/pull/51852)
introduced in 7.7.0. Search collapsing were not supposed to
be eligible for this sort optimization so this change explicitly
filters them from this new feature.
Changes:
* Moves `Retrieve selected fields` to its own page and adds a title abbreviation.
* Adds existing script and stored fields content to `Retrieve selected fields`
* Adds a xref for `Retrieve selected fields` to `Search your data`
* Adds related redirects and updates existing xrefs
Uses `my-data-stream` in place of `logs` for data stream examples.
This provides a more intuitive experience for users that copy/paste
their own values into snippets.
* [ML] have DELETE analytics ignore stats failures and clean up unused stats (#60776)
When deleting an analytics configuration, the request MIGHT fail if
the .ml-stats index does not exist or is in strange state (shards unallocated).
Instead of making the request fail, we should log that we were unable to delete the stats docs and then
have them cleaned up in the 'delete_expire_data' janitorial process
* Simplify jdk downloads via artifact transforms
This reworks the jdk download plugin to use artifact transforms.
That allows us to simplify the jdk download plugin by a lot. The
benefits of using artifact transforms are:
- they transform an artifact to an upacked folder on the fly as part
of the dependenc resolution allowing us to remove all the custom created unpack tasks
and configurations
- Artifact transforms support gradle build cache. Requesting a jdk folder on a clean machine
will likely be resolved from the build cache
- The manual mingling about not extracting jdks multiple times by all jdks channeling through
root project configurations can be removed as they support up-to-date checking and build cache
which will ensure these archives are only resolved once per build at max.
Also the test coverage has been ported to Spock that supports data driven testing. This porting
includes an introduction to a wiremock fixture that can be used later on for mocking repository
urls in other integration tests
* Simplify artifact transform registration
* Change jdk finalization and repository setup
Jdk finalization is now done via `configuration.defaultDependencies`.
This has two benefits:
- we only configure and finalize the JDKs we actually use during the build
- afterEvaluate hooks are problematic in combination with task avoidance api.
Relying on task avoidance api in Gradle moves more configuration logic into the materialization
of tasks which is, with task avoidance api, is done ideally during the task graph calculation.
Anything created (e.g. jdks) created late in this task graph calculation would never be finalized
via afterEvaluate hooks as this has been fired before already with the jdk not there yet.
Furthermore we now only configure repositories in the projects where the jdk is declared
(aka the jdk-download plugin is applied) and only if the jdk is actually requested during the build.
* Fix javadoc
* Fix jdk download repo content filtering
* Minor cleanup
* Apply review feedback and cleanup
Backport of #60746
to wait for at least a single shard to be allocated for a backing index of a data stream,
so that total store size is larger than zero (which is what the tests expects).
Closes#60461
remove test, scripts are excluded in the change collector, the test is a leftover from a previous
solution of #57332, which has been discarded
relates #60724fixes#60794
When an exception is thrown during test inference we are
not including the cause message in our logging. This commit
addresses this issue.
Backport of #60749
It appears the odd permission problems of NOTICE and the lintian
overrides file have disappeared, probably through further build cleanup.
This commit re-enables the lintian tests.
closes#58730
Currently the transport replication action does not propagate the force
execution parameter when acquiring the indexing permit. The logic to
acquire the index permit supports force execution, so this parameter
should be propagate. Fixes#60359.
This commit removes the last of the bats tests, converting the rpm/deb
upgrade tests to java. It adds a new pattern of tasks, similar in nature
but separate from the existing distro tests, named `distroUpgradeTest`.
For each index compatible version, a `distroUpgradeTest.VERSION` task
exxists. Each distribution then has a task, named
`distroUpgradeTest.VERSION.DISTRO`.
One thing to note is these new tests do not cover no-jdk versions of
the rpm/deb packages, since the distribution/bwc project does not
currently build those.
closes#59145closes#46005
I was unable to reproduce this locally on either 7.6 (first introduced) and 7.x. This is already not muted
on master and doesn't appear to have failures. There were some API changes at the time that could
have affected this test, and I'm wondering with backports if this is now stable again. If this has more
failures, I will continue to investigate further.
Relates to #51939
Add VPC endpoint as the recommended way of connecting to s3 in private subnets
Backport of #60654
Co-authored-by: Bill Mitchell <vocatan@users.noreply.github.com>
Co-authored-by: David Turner <david.turner@elastic.co>
This pull request adds recovery state tracking for Searchable Snapshots.
In order to track recoveries for searchable snapshot backed indices, this pull
request adds a new type of RecoveryState.
This newRecoveryState instance is able to deal with the
small differences that arise during Searchable snapshots recoveries.
Those differences can be summarized as follows:
- The Directory implementation that's provided by SearchableSnapshots mark the
snapshot files as reused during recovery. In order to keep track of the
recovery process as the cache is pre-warmed, those files shouldn't be marked
as reused.
- Once the shard is created, the cache starts its pre-warming phase, meaning that
we should keep track of those downloads during that process and tie the recovery
to this pre-warming phase. The shard is considered recovered once this pre-warming
phase has finished.
Backport of #60505
Andrei Dan
Rene Groeschke
Martijn van Groningen
David Kyle
David Turner
Alan Woodward
Rory Hunter
Mark Vieira
Ryan Ernst
Jim Ferenczi
James Rodewig
Mark Tozzi
Armin Braun
Benjamin Trent
Rory Hunter
Steven Goossens
Hendrik Muhs
Dimitris Athanasiou
Russ Cam
Tim Brooks
Ryan Ernst
Jack Conradson
Lisa Cawley
Francisco Fernández Castaño