This commit adds an API to read translog snapshot from Lucene,
then cut-over from the existing translog to the new API in CCR.
Relates #30086
Relates #29530
* master:
Mute ML upgrade test (#30458)
Stop forking javac (#30462)
Client: Deprecate many argument performRequest (#30315)
Docs: Use task_id in examples of tasks (#30436)
Security: Rename IndexLifecycleManager to SecurityIndexManager (#30442)
[Docs] Fix typo in cardinality-aggregation.asciidoc (#30434)
Avoid NPE in `more_like_this` when field has zero tokens (#30365)
Build: Switch to building javadoc with html5 (#30440)
Add a quick tour of the project to CONTRIBUTING (#30187)
Reindex: Use request flavored methods (#30317)
Silence SplitIndexIT.testSplitIndexPrimaryTerm test failure. (#30432)
Auto-expand replicas when adding or removing nodes (#30423)
Docs: fix changelog merge
Fix line length violation in cache tests
Add stricter geohash parsing (#30376)
Add failing test for core cache deadlock
[DOCS] convert forcemerge snippet
Update forcemerge.asciidoc (#30113)
Added zentity to the list of API extension plugins (#29143)
Fix the search request default operation behavior doc (#29302) (#29405)
Previously only index and delete operations are indexed into Lucene,
therefore every segment should have both _id and _version terms as these
operations contain both terms. However, this is no longer guaranteed
after noop is also indexed into Lucene. A segment which contains only
no-ops does not have neither _id or _version because a no-op does not
contain these terms.
This change adds a dummy version to no-ops and makes _id terms optional
in PerThreadIDVersionAndSeqNoLookup.
Relates #30226
GetSettingsResponseTests fails on the ccr branch because it excludes the
root-level settings while we introduced "index.soft_deletes" - a
root-level setting in this branch.
This commit moves "index.soft_deletes" to the grouped-level settings.
Moreover, as we are going to introduce more settings for soft_deletes, I
think it's better to organize all these settings into a single group.
Fixes and edge case when using `more_like_this` where TermVectorsWriter
could throw an NPE when a field produced zero tokens after analysis. This
changes the implementation to use an empty list of tokens in this case.
Closes#30148
* elastic-master:
Watcher: Mark watcher as started only after loading watches (#30403)
Pass the task to broadcast actions (#29672)
Disable REST default settings testing until #29229 is back-ported
Correct wording in log message (#30336)
Do not fail snapshot when deleting a missing snapshotted file (#30332)
AwaitsFix testCreateShrinkIndexToN
DOCS: Correct mapping tags in put-template api
DOCS: Fix broken link in the put index template api
Add put index template api to high level rest client (#30400)
Relax testAckedIndexing to allow document updating
[Docs] Add snippets for POS stop tags default value
Move respect accept header on no handler to 6.3.1
Respect accept header on no handler (#30383)
[Test] Add analysis-nori plugin to the vagrant tests
[Docs] Fix bad link
[Docs] Fix end of section in the korean plugin docs
Expose the Lucene Korean analyzer module in a plugin (#30397)
Docs: remove transport_client from CCS role example (#30263)
[Rollup] Validate timezone in range queries (#30338)
Use readFully() to read bytes from CipherInputStream (#28515)
Fix docs Recently merged #29229 had a doc bug that broke the doc build. This commit fixes.
Test: remove cluster permission from CCS user (#30262)
Add Get Settings API support to java high-level rest client (#29229)
Watcher: Remove unneeded index deletion in tests
Auto-expands replicas in the same cluster state update (instead of a follow-up reroute) where nodes are added or removed.
Closes#1873, fixing an issue where nodes drop their copy of auto-expanded data when coming up, only to sync it again later.
Adds verification that geohashes are not empty and contain only
valid characters. It fixes the issue when en empty geohash is
treated as [-180, -90] and geohashes with non-geohash character
are getting resolved into invalid coordinates.
Closes#23579
When deleting or creating a snapshot for a given shard, elasticsearch
usually starts by listing all the existing snapshotted files in the repository.
Then it computes a diff and deletes the snapshotted files that are not
needed anymore. During this deletion, an exception is thrown if the file
to be deleted does not exist anymore.
This behavior is challenging with cloud based repository implementations
like S3 where a file that has been deleted can still appear in the bucket for
few seconds/minutes (because the deletion can take some time to be fully
replicated on S3). If the deleted file appears in the listing of files, then the
following deletion will fail with a NoSuchFileException and the snapshot
will be partially created/deleted.
This pull request makes the deletion of these files a bit less strict, ie not
failing if the file we want to delete does not exist anymore. It introduces a
new BlobContainer.deleteIgnoringIfNotExists() method that can be used
at some specific places where not failing when deleting a file is
considered harmless.
Closes#28322
The test indexes new documents and is thus correct in testing that the response result
is `CREATED`. Sadly we can't guarantee exactly once delivery just yet.
Relates #9967Closes#21658
This commit introduces a soft-deletes retention merge policy based on
the global checkpoint. Some notes on this simple retention policy:
- This policy keeps all operations whose seq# is greater than the
persisted global checkpoint and configurable extra operations prior to
the global checkpoint. This is good enough for querying history changes.
- This policy is not watertight for peer-recovery. We send the
safe-commit in peer-recovery, thus we need to also send all operations
after the local checkpoint of that commit. This is analog to the min
translog generation for recovery.
- This policy is too simple to support rollback.
Relates #29530
Today when processing a request for a URL path for which we can not find
a handler we send back a plain-text response. Yet, we have the accept
header in our hand and can respect the accepted media type of the
request. This commit addresses this.
Changes how data is read from CipherInputStream
Instead of using `read()` and checking that the bytes read are what we
expect, use `readFully()` which will read exactly the number of bytes
while keep reading until the end of the stream or throw an
`EOFException` if not all bytes can be read.
This approach keeps the simplicity of using CipherInputStream while
working as expected with both JCE and BCFIPS Security Providers
This PR adds support for the Get Settings API to the java high-level rest client.
Furthermore, logic related to the retrieval of default settings has been moved from the rest layer into the transport layer and now default settings may be retrieved consistency via both the rest API and the transport API.
* master:
Set the new lucene version for 6.4.0
[ML][TEST] Clean up jobs in ModelPlotIT
Upgrade to 7.4.0-snapshot-1ed95c097b (#30357)
Watcher: Ensure trigger service pauses execution (#30363)
[DOCS] Added coming qualifiers in changelog
[DOCS] Commented out empty sections in the changelog to fix the doc build. (#30372)
Security: reduce garbage during index resolution (#30180)
Make RepositoriesMetaData contents unmodifiable (#30361)
Change quad tree max levels to 29. Closes#21191 (#29663)
Test: use trial license in qa tests with security
[ML] Add integration test for model plots (#30359)
SQL: Fix bug caused by empty composites (#30343)
[ML] Account for gaps in data counts after job is reopened (#30294)
InternalEngineTests.testConcurrentOutOfOrderDocsOnReplica should use two documents (#30121)
Change signature of Get Repositories Response (#30333)
Tests: Use different watch ids per test in smoke test (#30331)
[Docs] Add term query with normalizer example
Adds Eclipse config for xpack licence headers (#30299)
Watcher: Make start/stop cycle more predictable and synchronous (#30118)
[test] add debug logging for packaging test
[DOCS] Removed X-Pack Breaking Changes
[DOCS] Fixes link to TLS LDAP info
Update versions for start_trial after backport (#30218)
Packaging: Set elasticsearch user to have non-existent homedir (#29007)
[DOCS] Fixes broken links to bootstrap user (#30349)
Fix NPE when CumulativeSum agg encounters null/empty bucket (#29641)
Make licensing FIPS-140 compliant (#30251)
[DOCS] Reorganizes authentication details in Stack Overview (#30280)
Network: Remove http.enabled setting (#29601)
Fix merging logic of Suggester Options (#29514)
[DOCS] Adds LDAP realm configuration details (#30214)
[DOCS] Adds native realm configuration details (#30215)
ReplicationTracker.markAllocationIdAsInSync may hang if allocation is cancelled (#30316)
[DOCS] Enables edit links for X-Pack pages (#30278)
Packaging: Unmark systemd service file as a config file (#29004)
SQL: Reduce number of ranges generated for comparisons (#30267)
Tests: Simplify VersionUtils released version splitting (#30322)
Cancelling a peer recovery on the source can leak a primary permit (#30318)
Added changelog entry for deb prerelease version change (#30184)
Convert server javadoc to html5 (#30279)
Create default ES_TMPDIR on Windows (#30325)
[Docs] Clarify `fuzzy_like_this` redirect (#30183)
Post backport of #29658.
Fix docs of the `_ignored` meta field.
Remove MapperService#types(). (#29617)
Remove useless version checks in REST tests. (#30165)
Add a new `_ignored` meta field. (#29658)
Move repository-azure fixture test to QA project (#30253)
# Conflicts:
# buildSrc/version.properties
# server/src/test/java/org/elasticsearch/index/engine/InternalEngineTests.java
Upgrade to lucene-7.4.0-snapshot-1ed95c097b
This version contains:
* An Analyzer for Korean
* An IntervalQuery and IntervalsSource that retrieve minimum intervals of positional queries.
* A new API to retrieve matches (offsets and positions) of a query for a single document.
* Support for soft deletes in the index writer.
* A fixed shingle filter that handles index time synonyms.
* Support for emoji sequence in ICUTokenizer (with an upgrade to icu 61.1)
We were recently looking at bugs that can only occur if two different documents were indexed concurrently. For example, what happens if the local checkpoint advances above the sequence number of a document that's being indexed. That can only happen if another concurrent operation caused the checkpoint to advance. It has to be another document to allow concurrency as we acquire a per uid lock.While our investigation proved that the suspected bug doesn't exists, we still discovered our unit testing coverage is not good enough to cover this case.
This PR extend the test concurrent out of order replica processing to use two documents in its history.
The Get Repositories response object held a list of RepositoryMetaData
entries. This object does not have the from/toXContent methods that are
needed to expose this to the high level REST client. The
RepositoriesMetaData, however, does, and it also contains a list of
RepositoryMetaData objects within it. So rather than duplicate this
logic or move it (RepositoriesMetaData is a fragment object used by
cluster state), the object holding state in the Response was changed to
use the RepositoriesMetaData instead. This also cleans up the read/write
methods in the response, as they can now use the same read/write in
RepositoriesMetaData, which also were not present in the singular class.
Fix NPE when CumulativeSum agg encounters null/empty bucket
If the cusum agg encounters a null value, it's because the value is
missing (like the first value from a derivative agg), the path is
not valid, or the bucket in the path was empty.
Previously cusum would just explode on the null, but this changes it
so we only increment the sum if the value is non-null and finite.
This is safe because even if the cusum encounters all null or empty
buckets, the cumulative sum is still zero (like how the sum agg returns
zero even if all the docs were missing values)
I went ahead and tweaked AggregatorTestCase to allow testing pipelines,
so that I could delete the IT test and reimplement it as AggTests.
Closes#27544
This commit removes the http.enabled setting. While all real nodes (started with bin/elasticsearch) will always have an http binding, there are many tests that rely on the quickness of not actually needing to bind to 2 ports. For this case, the MockHttpTransport.TestPlugin provides a dummy http transport implementation which is used by default in ESIntegTestCase.
closes#12792
Suggester Options have a collate match field that is returned when the prune
option is set to true. These values should be merged together in the query
reduce phase, otherwise good suggestions that result in rare hits in shards with
results that do not arrive first may be incorrectly marked as not matching the
collate query.
At the end of recovery, we mark the recovering shard as "in sync" on the primary. From this point on
the primary will treat any replication failure on it as critical and will reach out to the master to fail the
shard. To do so, we wait for the local checkpoint of the recovered shard to be above the global
checkpoint (in order to maintain global checkpoint invariant).
If the master decides to cancel the allocation of the recovering shard while we wait, the method can
currently hang and fail to return. It will also ignore the interrupts that are triggered by the cancelled
recovery due to the primary closing.
Note that this is crucial as this method is called while holding a primary permit. Since the method
never comes back, the permit is never released. The unreleased permit will then block any primary
relocation *and* while the primary is trying to relocate all indexing will be blocked for 30m as it
waits to acquire the missing permit.
The code in `SourceRecoveryHandler` runs under a `CancellableThreads` instance in order to allow long running operations to be interrupted when the recovery is cancelled. Sadly if this happens at just the wrong moment while acquiring a permit from the primary, that primary can be leaked and never be freed.
Note that this is slightly better than it sounds - we only cancel recoveries on the source side if the primary shard itself is closed.
Relates to https://github.com/elastic/elasticsearch/pull/30316
This commit adds a tombstone document into Lucene for every No-op.
With this change, Lucene index is expected to have a complete history
of operations like Translog. In fact, this guarantee is subjected to the
soft-deletes retention merge policy.
Relates #29530
This adds a new `_ignored` meta field which indexes and stores fields that have
been ignored at index time because of the `ignore_malformed` option. It makes
malformed documents easier to identify by using `exists` or `term(s)` queries
on the `_ignored` field.
Closes#29494
* WIP commit to try calling rewrite on coordinating node during TransportSearchAction
* Use re-written query instead of using the original query
* fix incorrect/unused imports and wildcarding
* add error handling for cases where an exception is thrown
* correct exception handling such that integration tests pass successfully
* fix additional case covered by IndicesOptionsIntegrationIT.
* add integration test case that verifies queries are now valid
* add optional value for index
* address review comments: catch superclass of XContentParseException
fixes#29483
The variadic constructor was only used in a few places and the
RepositoriesMetaData class is backed by a List anyway, so just using a
List will make it simpler to instantiate it.
We still don't have a strong reason for the failures of
testDoNotRenewSyncedFlushWhenAllSealed and
testSyncedFlushSkipOutOfSyncReplicas.
This commit adds debug logging for these two tests.
Today when an index is created from shrinking or splitting an existing
index, the target index inherits almost none of the source index
settings. This is surprising and a hassle for operators managing such
indices. Given this is the default behavior, we can not simply change
it. Instead, we start by introducing the ability to copy settings. This
flag can be set on the REST API or on the transport layer and it has the
behavior that it copies all settings from the source except non-copyable
settings (a property of a setting introduced in this
change). Additionally, settings on the request will always override.
This change is the first step in our adventure:
- this flag is added here in 7.0.0 and immediately deprecated
- this flag will be backported to 6.4.0 and remain deprecated
- then, we will remove the ability to set this flag to false in 7.0.0
- finally, in 8.0.0 we will remove this flag and the only behavior will
be for settings to be copied
Just like `ElasticsearchException`, the inner most
`XContentParseException` tends to contain the root cause of the
exception and show be show to the user in the `root_cause` field.
The effectively undoes most of the changes that #29373 made to the
`root_cause` for parsing exceptions. The `type` field still changes from
`parse_exception` to `x_content_parse_exception`, but this seems like a
fairly safe change.
`ElasticsearchWrapperException` *looks* tempting to implement this but
the behavior isn't quite right. `ElasticsearchWrapperExceptions` are
entirely unwrapped until the cause no longer
`implements ElasticsearchWrapperException` but `XContentParseException`
should be unwrapped until its cause is no longer an
`XContentParseException` but no further. In other words,
`ElasticsearchWrapperException` are unwrapped one step too far.
Closes#30261
Remove double if depending on the Result value. It makes little sense to
pass in a boolean flag based on a Result value that we already have,
if that internally is represented again as a `Result` value.
Also changed the `Result` `lowercase` instance member to be computed
based on `name()` instead of `toString()` which is safer and to use
`Locale.ROOT` instead of `Locale.ENGLISH`
Starting with the refactoring in https://github.com/elastic/elasticsearch/pull/22778 (released in 5.3) we may fail to properly replicate operation when a mapping update on master fails. If a bulk
operations needs a mapping update half way, it will send a request to the master before continuing
to index the operations. If that request times out or isn't acked (i.e., even one node in the cluster
didn't process it within 30s), we end up throwing the exception and aborting the entire bulk. This is
a problem because all operations that were processed so far are not replicated any more to the
replicas. Although these operations were never "acked" to the user (we threw an error) it cause the
local checkpoint on the replicas to lag (on 6.x) and the primary and replica to diverge.
This PR does a couple of things:
1) Most importantly, treat *any* mapping update failure as a document level failure, meaning only
the relevant indexing operation will fail.
2) Removes the mapping update callbacks from `IndexShard.applyIndexOperationOnPrimary` and
similar methods for simpler execution. We don't use exceptions any more when a mapping
update was successful.
I think we need to do more work here (the fact that a single slow node can prevent those mappings
updates from being acked and thus fail operations is bad), but I want to keep this as small as I can
(it is already too big).
Currently, the only way to get the REST response for the `/_cluster/state`
call to return the `cluster_uuid` is to request the `metadata` metrics,
which is one of the most expensive response structures. However, external
monitoring agents will likely want the `cluster_uuid` to correlate the
response with other API responses whether or not they want cluster
metadata.
Today when a resize operation is performed, we copy the analysis,
similarity, and sort settings from the source index. It is possible for
the resize request to include additional index settings including
analysis, similarity, and sort settings. We reject sort settings when
validating the request. However, we silently ignore analysis and
similarity settings on the request that are already set on the source
index. Since it is possible to change the analysis and similarity
settings on an existing index, this should be considered a bug and the
sort of leniency that we abhor. This commit addresses this bug by
allowing the request analysis/similarity settings to override the
existing analysis/similarity settings on the target.
The `testDeleteSnapshotWithMissingIndexAndShardMetadata` test uses an
obsolete repository directory structure based on index names instead of
UUIDs. Because it swallows exceptions when deleting test files the test
never failed when the directory structure changed.
This commit fixes the test to use the right directory structure and file
names and to not swallow exceptions anymore.
Martijn van Groningen
Nhat Nguyen
aditya-agrawal
Jack Conradson
Yannick Welsch
Jason Tedor
Igor Motov
Stéphane Campinas
Tanguy Leroux
Boaz Leskes
Ioannis Kakavas
tomcallahan
Jim Ferenczi
Michael Basnight
Zachary Tong
Ryan Ernst
James Baiera
Adrien Grand
Paul Sanwald
Diwas Joshi
Nik Everett
Luca Cavanna
Chris Earle