* Add ShardIndexingPressure framework level construct and related Stats artefacts.
* Test and code refactoring for shard indexing pressure.
* Moved the average calculation logic to common memory manager util.
* Add wrapper for releasable in ShardIndexingPressure operations.
Signed-off-by: Saurabh Singh <sisurab@amazon.com>
It introduces a Memory Manager for Shard Indexing Pressure. It is responsible for increasing and decreasing the allocated shard limit based on incoming requests, and validate the current values against the thresholds.
Signed-off-by: Saurabh Singh <sisurab@amazon.com>
This commit adds test coverage for translog pruning setting to
MetadataCreateIndexServiceTests
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* Rename translog pruing setting to CCR specific setting
Signed-off-by: Sai Kumar <karanas@amazon.com>
* Rename to index.plugins.replication.translog.retention_lease.pruning.enabled as
index settings needs "index." as prefix
Signed-off-by: Sai Kumar <karanas@amazon.com>
* Add deprecations to retention pruning controls
This commit adds deprecation flags to all added settings, variables, and methods
specific to ccr's retention lease pruning mechanism.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* Addressed CR comments
Signed-off-by: Sai Kumar <karanas@amazon.com>
* fix javadoc deprecation
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* fix deprecation tag in TranslogDeletionPolicy
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* Addressed test issue under translog tests
Signed-off-by: Sai Kumar <karanas@amazon.com>
Co-authored-by: Nicholas Walter Knize <nknize@apache.org>
The changes ensure that in the event of a partial zone failure, the surviving nodes in the minority zone don't get overloaded with shards, this is governed by a skewness limit.
Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>
This will cause the leader stuck on IO during publication to step down and eventually trigger a leader election.
Issue Description
---
The publication of cluster state is time bound to 30s by a cluster.publish.timeout settings. If this time is reached before the new cluster state is committed, then the cluster state change is rejected and the leader considers itself to have failed. It stands down and starts trying to elect a new master.
There is a bug in leader that when it tries to publish the new cluster state it first tries acquire a lock to flush the new state under a mutex to disk. The same lock is used to cancel the publication on timeout. Below is the state of the timeout scheduler meant to cancel the publication. So essentially if the flushing of cluster state is stuck on IO, so will the cancellation of the publication since both of them share the same mutex. So leader will not step down and effectively block the cluster from making progress.
Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>
* Drop mocksocket in favour of custom security manager checks (tests only)
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
* Slightly relaxed host checks to allow all local addresses
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
* Refactor the logic to control the format for code coverage report and rename the system property
* Remove outdated code of giving JaCoCo files permission when Java security manager enabled
Signed-off-by: Tianli Feng <ftianli@amazon.com>
The public key has changed since the initial release. This commit fixes the
public key and uses the .sig files that are published to the artifacts site.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Most PRs do not add/update links, however sites go down often. This change makes sure that we catch any broken link in the repository and fix it, but at the same time we do not want to block PRs because of some unrelated broken links.
This PR updates the workflow to run everyday at midnight UTC.
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
* Changes to support retrieval of operations from translog based on specified range
Signed-off-by: Sai Kumar <karanas@amazon.com>
* Addressed CR comments
Signed-off-by: Sai Kumar <karanas@amazon.com>
* Added testcases for internal engine
Signed-off-by: Sai Kumar <karanas@amazon.com>
* Support for translog pruning based on retention leases
Signed-off-by: Sai Kumar <karanas@amazon.com>
* Addressed CR Comments
Signed-off-by: Sai Kumar <karanas@amazon.com>
* Addressed test case issue
Signed-off-by: Sai Kumar <karanas@amazon.com>
* Explicitly point out the JDK 8 requirement is for runtime, but not for compiling.
* Clarify the JAVAx_HOME env variables are for the "backwards compatibility test".
* Add explanation on how the backwards compatibility tests get the OpenSearch distributions for a specific version.
Signed-off-by: Tianli Feng <ftianli@amazon.com>
The version framework only added support for OpenSearch 1.x bwc with legacy
clusters. This commit adds support for v2.0 which will be the last version with
bwc support for legacy clusters (v7.10)
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* Drop mocksocket & securemock dependencies from sniffer and rest client (not needed)
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
* Removing .gitignore
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
Rabi Panda
Saurabh Singh
Daniel Doubrovkine (dB.)
Shivansh Arora
Nick Knize
Sai
Xue Zhou
Tianli Feng
Andriy Redko
CEHENKLE
Bukhtawar Khan
Abbas Hussain
Sven R
Vacha
Anas Alkouz
Andriy Redko
Minal Shah