* Add RemoteDirectory interface to copy segment files to/from remote store
Signed-off-by: Sachin Kale <kalsac@amazon.com>
Co-authored-by: Sachin Kale <kalsac@amazon.com>
* Add index level setting for remote store
Signed-off-by: Sachin Kale <kalsac@amazon.com>
Co-authored-by: Sachin Kale <kalsac@amazon.com>
* Add RemoteDirectoryFactory and use RemoteDirectory instance in RefreshListener
Co-authored-by: Sachin Kale <kalsac@amazon.com>
Signed-off-by: Sachin Kale <kalsac@amazon.com>
* Upload segment to remote store post refresh
Signed-off-by: Sachin Kale <kalsac@amazon.com>
Co-authored-by: Sachin Kale <kalsac@amazon.com>
Upgrades to latest snapshot of lucene 9.3; including reducing maxFullFlushMergeWaitMillis
in LuceneTest.testWrapLiveDocsNotExposeAbortedDocuments to 0 ms to ensure aborted
docs are not merged away in the test with the new mergeOnRefresh default policy.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Fix the support of RestClient Node Sniffer for OpenSearch 2.x, and update unit tests for OpenSearch.
The current code contains the logic to be compatible with Elasticsearch 2.x version, which is conflict with OpenSearch 2.x, so removed that part of legacy code.
* Update the script create_test_nodes_info.bash to dump the response of Nodes Info API GET _nodes/http for OpenSearch 1.0 and 2.0 version, which used for unit test.
* Remove the support of Elasticsearch version 2.x for the Sniffer
* Update unit test to validate the Sniffer compatible with OpenSearch 1.x and 2.x
* Update the API response parser to meet the array notation (in ES 6.1 and above) for the node attributes setting. It will result the value of `node.attr` setting will not be parsed as array in the Sniffer, when using the Sniffer on cluster in Elasticsearch 6.0 and above.
* Replace "master" node role with "cluster_manager" in unit test
Signed-off-by: Tianli Feng <ftianli@amazon.com>
Passing bash with flags to the first argument of /usr/bin/env requires
its own flag to interpret it correctly. Rather than use `env -S` to
split the argument, have the script `set -e` to enable the same behavior
explicitly in preinst and postinst scripts.
Also set `-o pipefail` for consistency.
Closes: #3492
Signed-off-by: Cole White <cwhite@wikimedia.org>
This change expands on the existing SegmentReplicationSource interface and its corresponding Factory class by introducing an implementation where the replication source is a primary shard (PrimaryShardReplicationSource). These code paths execute on the target. The primary shard implementation creates the requests to be send to the source/primary shard.
Correspondingly, this change also defines two request classes for the GET_CHECKPOINT_INFO and GET_SEGMENT_FILES requests as well as an abstract superclass.
A CopyState class has been introduced that captures point-in-time, file-level details from an IndexShard. This implementation mirrors Lucene's NRT CopyState implementation.
Finally, a service class has been introduce for segment replication that runs on the source side (SegmentReplicationSourceService) which handles these two types of incoming requests. This includes private handler classes that house the logic to respond to these requests, with some functionality stubbed for now. The service class also uses a simple map to cache CopyState objects that would be needed by replication targets.
Unit tests have been added/updated for all new functionality.
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Support use of IRSA for repository-s3 plugin credentials
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
* Address code review comments
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
* Address code review comments
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
Filter out invalid URI and HTTP method of a error message, which shown when there is no handler found for a REST request sent by user, so that HTML special characters <>&"' will not shown in the error message.
The error message is return as mine-type `application/json`, which can't contain active (script) content, so it's not a vulnerability. Besides, no browsers are going to render as html when the mine-type is that.
While the common security scanners will raise a false-positive alarm for having HTML tags in the response without escaping the HTML special characters, so the solution only aims to satisfy the code security scanners.
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Add SegmentReplicationTargetService to orchestrate replication events.
This change introduces boilerplate classes for Segment Replication and a target service
to orchestrate replication events.
It also includes two refactors of peer recovery components for reuse.
1. Rename RecoveryFileChunkRequest to FileChunkRequest and extract code to handle throttling into
ReplicationTarget.
2. Extracts a component to execute retryable requests over the transport layer.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Code cleanup.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Make SegmentReplicationTargetService component final so that it can not
be extended by plugins.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Set term vector flags to false for ._index_prefix field (#1901).
Signed-off-by: Vesa Pehkonen <vesa.pehkonen@intel.com>
* Replaced the FieldType copy ctor with ctor for the prefix field and replaced
setting the field type parameters with setIndexOptions(). (#1901)
Signed-off-by: Vesa Pehkonen <vesa.pehkonen@intel.com>
* Added tests for term vectors. (#1901)
Signed-off-by: Vesa Pehkonen <vesa.pehkonen@intel.com>
* Fixed code formatting error.
Signed-off-by: Vesa Pehkonen <vesa.pehkonen@intel.com>
Co-authored-by: sdp <sdp@9049fa06826d.jf.intel.com>
* Replace the non-inclusive terminology "master" with "cluster manager" in code comments, internal variable/method/class names, in `server/src/test` directory.
* Backwards compatibility is not impacted.
* Add a new unit test `testDeprecatedMasterNodeFilter()` to validate using `master:true` or `master:false` can filter the node in [Cluster Stats](https://opensearch.org/docs/latest/opensearch/rest-api/cluster-stats/) API, after the `master` role is deprecated in PR https://github.com/opensearch-project/OpenSearch/pull/2424
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Change fastForwardProcessedSeqNo method in LocalCheckpointTracker to persisted checkpoint.
This change inverts fastForwardProcessedSeqNo to fastForwardPersistedSeqNo for use in
Segment Replication. This is so that a Segrep Engine can match the logic of InternalEngine
where the seqNo is incremented with each operation, but only persisted in the tracker on a flush.
With Segment Replication we bump the processed number with each operation received index/delete/noOp, and
invoke this method when we receive a new set of segments to bump the persisted seqNo.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Extract Translog specific engine methods into an abstract class.
This change extracts translog specific methods to an abstract engine class so that other engine
implementations can reuse translog logic.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Add a separate Engine implementation for replicas with segment replication enabled.
This change adds a new engine intended to be used on replicas with segment replication enabled.
This engine does not wire up an IndexWriter, but still writes all operations to a translog.
The engine uses a new ReaderManager that refreshes from an externally provided SegmentInfos.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Fix spotless checks.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Fix :server:compileInternalClusterTestJava compilation.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Fix failing test naming convention check.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* PR feedback.
- Removed isReadOnlyReplica from overloaded constructor and added feature flag checks.
- Updated log msg in NRTReplicationReaderManager
- cleaned up store ref counting in NRTReplicationEngine.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Fix spotless check.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Remove TranslogAwareEngine and build translog in NRTReplicationEngine.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Fix formatting
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Add missing translog methods to NRTEngine.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Remove persistent seqNo check from fastForwardProcessedSeqNo.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* PR feedback.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Add test specific to translog trimming.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Javadoc check.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Add failEngine calls to translog methods in NRTReplicationEngine.
Roll xlog generation on replica when a new commit point is received.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Intial PR adding classes and tests related to checkpoint publishing
Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>
* Putting a Draft PR with all changes in classes. Testing is still not included in this commit.
Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>
* Wiring up index shard to new engine, spotless apply and removing unnecessary tests and logs
Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>
* Adding Unit test for checkpointRefreshListener
Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>
* Applying spotless check
Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>
* Fixing import statements *
Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>
* removing unused constructor in index shard
Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>
* Addressing comments from last commit
Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>
* Adding package-info.java files for two new packages
Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>
* Adding test for null checkpoint publisher and addreesing PR comments
Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>
* Add docs for indexshardtests and remove shard.refresh
Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>
* [Type removal] Remove type handling in bulk request parser
Signed-off-by: Suraj Singh <surajrider@gmail.com>
* [Type removal] Remove testTypesStillParsedForBulkMonitoring as it is no longer present in codebase
Signed-off-by: Suraj Singh <surajrider@gmail.com>
This change removes some old commits from the 2.0.0-rc1 release notes. These commits were already released as part of 1.x releases.
Add back some missing type removal commits to the 2.0.0 release notes
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
Anshu Agarwal
dependabot[bot]
Rishab Nahata
Andriy Redko
Sachin Kale
Nick Knize
Xue Zhou
opensearch-trigger-bot[bot]
Tianli Feng
Cole White
Suraj Singh
Kartik Ganesh
Marc Handalian
vpehkone
Ankit Jain
Rishikesh Pasham
Poojita Raj
Rabi Panda