* Change fastForwardProcessedSeqNo method in LocalCheckpointTracker to persisted checkpoint.
This change inverts fastForwardProcessedSeqNo to fastForwardPersistedSeqNo for use in
Segment Replication. This is so that a Segrep Engine can match the logic of InternalEngine
where the seqNo is incremented with each operation, but only persisted in the tracker on a flush.
With Segment Replication we bump the processed number with each operation received index/delete/noOp, and
invoke this method when we receive a new set of segments to bump the persisted seqNo.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Extract Translog specific engine methods into an abstract class.
This change extracts translog specific methods to an abstract engine class so that other engine
implementations can reuse translog logic.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Add a separate Engine implementation for replicas with segment replication enabled.
This change adds a new engine intended to be used on replicas with segment replication enabled.
This engine does not wire up an IndexWriter, but still writes all operations to a translog.
The engine uses a new ReaderManager that refreshes from an externally provided SegmentInfos.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Fix spotless checks.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Fix :server:compileInternalClusterTestJava compilation.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Fix failing test naming convention check.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* PR feedback.
- Removed isReadOnlyReplica from overloaded constructor and added feature flag checks.
- Updated log msg in NRTReplicationReaderManager
- cleaned up store ref counting in NRTReplicationEngine.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Fix spotless check.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Remove TranslogAwareEngine and build translog in NRTReplicationEngine.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Fix formatting
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Add missing translog methods to NRTEngine.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Remove persistent seqNo check from fastForwardProcessedSeqNo.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* PR feedback.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Add test specific to translog trimming.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Javadoc check.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Add failEngine calls to translog methods in NRTReplicationEngine.
Roll xlog generation on replica when a new commit point is received.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Intial PR adding classes and tests related to checkpoint publishing
Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>
* Putting a Draft PR with all changes in classes. Testing is still not included in this commit.
Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>
* Wiring up index shard to new engine, spotless apply and removing unnecessary tests and logs
Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>
* Adding Unit test for checkpointRefreshListener
Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>
* Applying spotless check
Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>
* Fixing import statements *
Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>
* removing unused constructor in index shard
Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>
* Addressing comments from last commit
Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>
* Adding package-info.java files for two new packages
Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>
* Adding test for null checkpoint publisher and addreesing PR comments
Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>
* Add docs for indexshardtests and remove shard.refresh
Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>
* Use set to make shell scripts more strict
Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>
* Change -o pipefail to -e
Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>
* Set scripts to standard rule
Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>
Refactors XContentType.fromMediaTypeOrFormat to fromMediaType so Accept headers
and Content-Type headers can be parsed separately. This helps in reusing the
same parse logic in for REST Versioning API support.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* Replace internal usages of 'master' term in 'client' directory
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Add a uni test for NodeSelector to test deprecated master role
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Replace internal usages of 'master' terminology in server/src/main directory
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Restore rename DISCOVERED_MASTER in ClusterHealthResponse
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Rename two methods in unit tests
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Replace master word in ClusterState
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Replace master word in LeaderChecker JoinHelper JoinTaskExecutor
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Replace master word in more classes
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Replace master word in more classes
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Replace master word in more classes
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Replace master word in more classes
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Replace master word in more classes
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Replace master word in more classes
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Replace master word in DiscoveryNodes classes
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Replace master word in more classes
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Correct mistakes
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Adjust format by spotlessApply task
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Change MASTER__NODE_BOOTSTRAPPED_MSG in test
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Fix SnapshotDisruptionIT by renaming to cluster-manager
Signed-off-by: Tianli Feng <ftianli@amazon.com>
Adds a new multi_term aggregation. The current implementation focuses
on adding new type aggregates. Performance (latency) is suboptimal in this
iteration, mainly because of brute force encoding/decoding a list of values
into bucket keys. A performance improvement change will be made as a
follow on.
Signed-off-by: Peng Huo <penghuo@gmail.com>
This reverts commit 6a2a33d187.
During the process of deprecating REST API request parameter master_timeout and adding alternative parameter cluster_manager_timeout, I made High-Level-Rest-Client tests allow deprecation warning temporarily, by changing the argument of `setStrictDeprecationMode()` to false when building `RestClient` for tests, in the above commit / PR https://github.com/opensearch-project/OpenSearch/pull/2702,
This PR sets the High-Level-Rest-Client tests back to treating warning header as a failure.
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Upgrade to Apache Commons Logging 1.2
Signed-off-by: Yoann Rodière <yoann@hibernate.org>
* Clarify that Apache HTTP/commons-* dependencies are not just for tests
Signed-off-by: Yoann Rodière <yoann@hibernate.org>
Temporarily build rest client with setStrictDeprecationMode(false) to allow
deprecation warning in HLRC tests while master_timeout parameters is
being refactored.
Signed-off-by: Tianli Feng <ftianli@amazon.com>
Add a new node role `cluster_manager`, as the replacement for `master` role, that used in setting `node.roles: [ master ]`.
They have got the same functionality, but can NOT be assigned to `node.roles` together.
* Add `CLUSTER_MANAGER_ROLE` in `DiscoveryNodeRole` class, and deprecate `MASTER_ROLE`
* Remove `MASTER_ROLE` from the `DiscoveryNodeRole.BUILT_IN_ROLES`, and temporarily load the role by the existing method `DiscoveryNode.setAdditionalRoles`
* Add a method `validateRole()` in `DiscoveryNodeRole` class, it's used to validate a specific role is compatible with the other roles to be assigned to a node together.
* Add deprecation message when assigning `master` role in setting `node.roles`
* Replace most `MASTER_ROLE` with `CLUSTER_MANAGER_ROLE` in current unit and integration tests
* Add new unit and integration tests to validate `CLUSTER_MANAGER_ROLE` and `MASTER_ROLE` can be treated as the same.
More explanation:
* New node will have "cluster_manager", "data", and "ingest" roles by default. Which means the default value of setting "node.roles" will be `["cluster_manager","data","ingest"]`, instead of `["master","data","ingest"]`
* "cluster_manager" role will be treated as "master" role in the OpenSearch node of previous versions.
* "cluster_manager” role and "master" role have got the same abbreviation name: "m"
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Add a new setting `cluster.no_cluster_manager_block`, aims to replace the existing `cluster.no_master_block`
* Set the value of the old setting `cluster.no_master_block` as the fallback to the new setting
* Deprecate the existing `setting cluster.no_master_block`
* Add unit tests
Signed-off-by: Tianli Feng <ftianli@amazon.com>
Removes the type variable as input to CreateIndexRequest.mapping(Object...)
along with the CreateIndexRequestBuilder helper class. This also refactors the
method name to setMapping for consistency with other methods (e.g.,
setSettings).
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Continues removal of types from PutIndexTemplateRequest and
PutIndexTemplateRequestBuilder.mapping. Delegated mapping method in
PutIndexTemplateRequestBuilder is refactored to setMapping for consistency with
similar methods (e.g., setSettings, setAliases).
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
First pass to remove types from CreateIndexRequest and CreateIndexRequestBuilder
mapping method. This method is overloaded several times so the most widely used
methods in the RequestBuilder are refactored from mapping to setMapping to avoid
confusion, conflicts, and to be consistent with other method names (e.g.,
setSettings, setCause, setAlias).
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* [Upgrade] Lucene 9.1.0-snapshot-ea989fe8f30
Upgrades from Lucene 9.0.0 to 9.1.0-snapshot-ea989fe8f30 in preparation for
9.1.0 GA.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* Add spanishplural token filter
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* fix KNOWN_TOKENIZERS
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
This commit upgrades the core codebase from Lucene 8.10.1 to
lucene 9.0.0. It includes all necessary refactoring of features and
API changes when upgrading to a new major Lucene release.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Co-authored-by: Andriy Redko <drreta@gmail.com>
Andriy Redko
dependabot[bot]
Marc Handalian
Rishikesh Pasham
Poojita Raj
Nick Knize
Suraj Singh
Tianli Feng
Owais Kazi
Daniel Doubrovkine (dB.)
Peng Huo
Tushar Kharbanda
Vacha Shah
Yoann Rodière
Wenjun Ruan
Lukáš Vlček