* Add RemoteDirectory interface to copy segment files to/from remote store
Signed-off-by: Sachin Kale <kalsac@amazon.com>
Co-authored-by: Sachin Kale <kalsac@amazon.com>
* Add index level setting for remote store
Signed-off-by: Sachin Kale <kalsac@amazon.com>
Co-authored-by: Sachin Kale <kalsac@amazon.com>
* Add RemoteDirectoryFactory and use RemoteDirectory instance in RefreshListener
Co-authored-by: Sachin Kale <kalsac@amazon.com>
Signed-off-by: Sachin Kale <kalsac@amazon.com>
* Upload segment to remote store post refresh
Signed-off-by: Sachin Kale <kalsac@amazon.com>
Co-authored-by: Sachin Kale <kalsac@amazon.com>
Upgrades to latest snapshot of lucene 9.3; including reducing maxFullFlushMergeWaitMillis
in LuceneTest.testWrapLiveDocsNotExposeAbortedDocuments to 0 ms to ensure aborted
docs are not merged away in the test with the new mergeOnRefresh default policy.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
This change expands on the existing SegmentReplicationSource interface and its corresponding Factory class by introducing an implementation where the replication source is a primary shard (PrimaryShardReplicationSource). These code paths execute on the target. The primary shard implementation creates the requests to be send to the source/primary shard.
Correspondingly, this change also defines two request classes for the GET_CHECKPOINT_INFO and GET_SEGMENT_FILES requests as well as an abstract superclass.
A CopyState class has been introduced that captures point-in-time, file-level details from an IndexShard. This implementation mirrors Lucene's NRT CopyState implementation.
Finally, a service class has been introduce for segment replication that runs on the source side (SegmentReplicationSourceService) which handles these two types of incoming requests. This includes private handler classes that house the logic to respond to these requests, with some functionality stubbed for now. The service class also uses a simple map to cache CopyState objects that would be needed by replication targets.
Unit tests have been added/updated for all new functionality.
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
Filter out invalid URI and HTTP method of a error message, which shown when there is no handler found for a REST request sent by user, so that HTML special characters <>&"' will not shown in the error message.
The error message is return as mine-type `application/json`, which can't contain active (script) content, so it's not a vulnerability. Besides, no browsers are going to render as html when the mine-type is that.
While the common security scanners will raise a false-positive alarm for having HTML tags in the response without escaping the HTML special characters, so the solution only aims to satisfy the code security scanners.
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Add SegmentReplicationTargetService to orchestrate replication events.
This change introduces boilerplate classes for Segment Replication and a target service
to orchestrate replication events.
It also includes two refactors of peer recovery components for reuse.
1. Rename RecoveryFileChunkRequest to FileChunkRequest and extract code to handle throttling into
ReplicationTarget.
2. Extracts a component to execute retryable requests over the transport layer.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Code cleanup.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Make SegmentReplicationTargetService component final so that it can not
be extended by plugins.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Replace the non-inclusive terminology "master" with "cluster manager" in code comments, internal variable/method/class names, in `server/src/test` directory.
* Backwards compatibility is not impacted.
* Add a new unit test `testDeprecatedMasterNodeFilter()` to validate using `master:true` or `master:false` can filter the node in [Cluster Stats](https://opensearch.org/docs/latest/opensearch/rest-api/cluster-stats/) API, after the `master` role is deprecated in PR https://github.com/opensearch-project/OpenSearch/pull/2424
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Change fastForwardProcessedSeqNo method in LocalCheckpointTracker to persisted checkpoint.
This change inverts fastForwardProcessedSeqNo to fastForwardPersistedSeqNo for use in
Segment Replication. This is so that a Segrep Engine can match the logic of InternalEngine
where the seqNo is incremented with each operation, but only persisted in the tracker on a flush.
With Segment Replication we bump the processed number with each operation received index/delete/noOp, and
invoke this method when we receive a new set of segments to bump the persisted seqNo.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Extract Translog specific engine methods into an abstract class.
This change extracts translog specific methods to an abstract engine class so that other engine
implementations can reuse translog logic.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Add a separate Engine implementation for replicas with segment replication enabled.
This change adds a new engine intended to be used on replicas with segment replication enabled.
This engine does not wire up an IndexWriter, but still writes all operations to a translog.
The engine uses a new ReaderManager that refreshes from an externally provided SegmentInfos.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Fix spotless checks.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Fix :server:compileInternalClusterTestJava compilation.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Fix failing test naming convention check.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* PR feedback.
- Removed isReadOnlyReplica from overloaded constructor and added feature flag checks.
- Updated log msg in NRTReplicationReaderManager
- cleaned up store ref counting in NRTReplicationEngine.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Fix spotless check.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Remove TranslogAwareEngine and build translog in NRTReplicationEngine.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Fix formatting
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Add missing translog methods to NRTEngine.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Remove persistent seqNo check from fastForwardProcessedSeqNo.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* PR feedback.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Add test specific to translog trimming.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Javadoc check.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Add failEngine calls to translog methods in NRTReplicationEngine.
Roll xlog generation on replica when a new commit point is received.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Intial PR adding classes and tests related to checkpoint publishing
Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>
* Putting a Draft PR with all changes in classes. Testing is still not included in this commit.
Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>
* Wiring up index shard to new engine, spotless apply and removing unnecessary tests and logs
Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>
* Adding Unit test for checkpointRefreshListener
Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>
* Applying spotless check
Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>
* Fixing import statements *
Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>
* removing unused constructor in index shard
Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>
* Addressing comments from last commit
Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>
* Adding package-info.java files for two new packages
Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>
* Adding test for null checkpoint publisher and addreesing PR comments
Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>
* Add docs for indexshardtests and remove shard.refresh
Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>
* [Type removal] Remove type handling in bulk request parser
Signed-off-by: Suraj Singh <surajrider@gmail.com>
* [Type removal] Remove testTypesStillParsedForBulkMonitoring as it is no longer present in codebase
Signed-off-by: Suraj Singh <surajrider@gmail.com>
Adds the remaining javadocs to internal classes and reenables the missingJavadoc
gradle task on the server module. From here forward if class level javadocs are
missing in the server module, gradle check will fail!
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* Bootstrap should implement a denylist of Java versions (ranges)
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
* Addressing code review comments
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
Adds class level javadocs to org.opensearch.search, and
org.opensearch.search.rescore, searchAfter, slice, sort, and suggest
subpackages.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Adds class level javadocs to org.opensearch.dfs, fetch, internal, lookup,
profile, and query packages.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Adds class level javadocs to org.opensearch.search.aggregations, builder, and
collapse packages.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* Moving RecoveryState.Index to a top-level class and renaming
This class is a building block of replication and will be re-used between peer recovery and segment replication. Thus, the inner class has been extracted to a top-level class and moved to the replication.common package. It has been renamed to ReplicationLuceneIndex to better reflect what it represents. It has two dependent inner classes from RecoveryState that have also been moved along with it - these remain inner classes since they are not currently used anywhere else. The RecoveryFilesDetails class has been renamed to FilesDetails and the FileDetail class has been renamed to FileMetadata.
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Incorporate PR comments
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Revert Project_Default.xml
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Update REST Action test to no longer mock a final class
Instead, the test now populates dummy data.
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Removing mocking of ReplicationLuceneIndex in RecoverySourceHandlerTests
The class has been marked final, so it can no longer be mocked. Instead, the test class sets up the lucene index class by adding the smae file metadata that is set up for the store.
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Configure @opensearch.internal as custom Javadoc tag
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Revert "Configure @opensearch.internal as custom Javadoc tag"
This reverts commit 2077d76e421fc2ceea7c1c9714b6a1906a555a31.
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Bump HdrHistogram from 2.1.9 to 2.1.12 in /server
Bumps [HdrHistogram](https://github.com/HdrHistogram/HdrHistogram) from 2.1.9 to 2.1.12.
- [Release notes](https://github.com/HdrHistogram/HdrHistogram/releases)
- [Commits](https://github.com/HdrHistogram/HdrHistogram/compare/HdrHistogram-2.1.9...HdrHistogram-2.1.12)
---
updated-dependencies:
- dependency-name: org.hdrhistogram:HdrHistogram
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* Updating SHAs
Signed-off-by: dependabot[bot] <support@github.com>
* Update test to match new behavior of HdrHistogram
There was a [change in behavior in HdrHistogram][1] that causes the
tests to fail with the new version. The fix here is to add another value
in the test data set for the failing test, which has the result of
making both versions of HdrHistogram agree on the percentile values.
[1]: 5c7226c4ff
Signed-off-by: Andrew Ross <andrross@amazon.com>
* Remove unnecessary exclusion on third party audit
Signed-off-by: Andrew Ross <andrross@amazon.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
Co-authored-by: Andrew Ross <andrross@amazon.com>
Removes last trace of TypeFieldMapper since types have been removed and
NestedPathFieldMapper is now available.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
opensearch-trigger-bot[bot]
Anshu Agarwal
Rishab Nahata
Andriy Redko
Sachin Kale
Nick Knize
Kartik Ganesh
Tianli Feng
Marc Handalian
Ankit Jain
Rishikesh Pasham
Suraj Singh
Poojita Raj
dependabot[bot]