When creating the second instance of an InternalEngine using the same
translog config of the default InternalEngine instance, the second
instance will attempt to delete all the existing translog files. I found
a deterministic test failure when running with the seed
`E3E6AAD95ABD299B`.
As opposed to creating a second engine instance with a different
translog location, just close the first one before creating the second.
Signed-off-by: Andrew Ross <andrross@amazon.com>
This commit adds a method that can be used to provide a custom TranslogDeletionPolicy
from within plugins that implement the EnginePlugin interface. This enables plugins to
provide a custom deletion policy with the current limitation that only one plugin can
override the policy. An exception will be thrown if more than one plugin overrides the
policy.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
This commit adds an explicit version check for shard indexing pressure
serialization. This is required to not mandate test have the cluster service
initialized while asserting node attributes.
Signed-off-by: Saurabh Singh <sisurab@amazon.com>
Co-authored-by: Saurabh Singh <sisurab@amazon.com>
This commit adds an extension point to EngineConfig through EnginePlugin using
a new EngineConfigFactory mechanism. EnginePlugin provides interface methods to
override configurations in EngineConfig. The EngineConfigFactory produces a new
instance of the EngineConfig using these overrides. Defaults are used absent
overridden configurations.
This serves as a mechanism to override Engine configurations (e.g., CodecService,
TranslogConfig) enabling Plugins to have higher fidelity for changing Engine
behavior without having to override the entire Engine (which is only permitted for
a single plugin).
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* Modernize and consolidate JDKs usage across all stages of the build. Update JDK-14 requirement, switch to JDK-17 instead
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
* Updating phrasing based on review feedback
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
* Fixed runtime Java version usage, it has to be respected when RUNTIME_JAVA_HOME == JAVA_HOME
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
* Addressing review comments
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
Fixes flakiness for test testReplicaThreadedThroughputDegradationAndRejection.
Reduced the number of threads concurrently executing from the initial range between (100-120) to a new range between (80-100), as the previous range was breaking the node limits set as 10kb, for every execution where number of threads were greater than 110.
Signed-off-by: Saurabh Singh <sisurab@amazon.com>
* Allow building on FreeBSD
With this set of change, we are able to successfuly run:
```
./gradlew publishToMavenLocal -Dbuild.snapshot=false
```
This step is used in the OpenSearch repository context when building
plugins in the current state of the CI.
While here, reorder OS conditions alphabetically.
Before building, the openjdk14 package was installed and the environment
was adjusted to use it:
```
sudo pkg install openjdk14
export JAVA_HOME=/usr/local/openjdk14/
export PATH=$JAVA_HOME/bin:$PATH
```
Signed-off-by: Romain Tartière <romain@blogreen.org>
* Unbreak CI with FreeBSD support
Signed-off-by: dblock <dblock@dblock.org>
Co-authored-by: dblock <dblock@dblock.org>
* Modernize and consolidate JDKs usage across all stages of the build. Use JDK-17 as bundled JDK distribution to run tests
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
* Using -Djava.security.egd=file:/dev/urandom explicitly for cli tests
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
Shard level indexing pressure improves the current Indexing Pressure framework which performs memory accounting at node level and rejects the requests. This takes a step further to have rejections based on the memory accounting at shard level along with other key performance factors like throughput and last successful requests.
**Key features**
- Granular tracking of indexing tasks performance, at every shard level, for each node role i.e. coordinator, primary and replica.
- Smarter rejections by discarding the requests intended only for problematic index or shard, while still allowing others to continue (fairness in rejection).
- Rejections thresholds governed by combination of configurable parameters (such as memory limits on node) and dynamic parameters (such as latency increase, throughput degradation).
- Node level and shard level indexing pressure statistics exposed through stats api.
- Integration of Indexing pressure stats with Plugins for for metric visibility and auto-tuning in future.
- Control knobs to tune to the key performance thresholds which control rejections, to address any specific requirement or issues.
- Control knobs to run the feature in shadow-mode or enforced-mode. In shadow-mode only internal rejection breakdown metrics will be published while no actual rejections will be performed.
The changes were divided into small manageable chunks as part of the following PRs against a feature branch.
- Add Shard Indexing Pressure Settings. #716
- Add Shard Indexing Pressure Tracker. #717
- Refactor IndexingPressure to allow extension. #718
- Add Shard Indexing Pressure Store #838
- Add Shard Indexing Pressure Memory Manager #945
- Add ShardIndexingPressure framework level construct and Stats #1015
- Add Indexing Pressure Service which acts as orchestrator for IP #1084
- Add plumbing logic for IndexingPressureService in Transport Actions. #1113
- Add shard indexing pressure metric/stats via rest end point. #1171
- Add shard indexing pressure integration tests. #1198
Signed-off-by: Saurabh Singh <sisurab@amazon.com>
Co-authored-by: Saurabh Singh <sisurab@amazon.com>
Co-authored-by: Rabi Panda <adnapibar@gmail.com>
* Support for Heap after GC stats
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
* Addressing code review comments
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
* Using the right version 2.0.0 (instead of 1.2.0) for the change
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
This commit adds test coverage for translog pruning setting to
MetadataCreateIndexServiceTests
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* Rename translog pruing setting to CCR specific setting
Signed-off-by: Sai Kumar <karanas@amazon.com>
* Rename to index.plugins.replication.translog.retention_lease.pruning.enabled as
index settings needs "index." as prefix
Signed-off-by: Sai Kumar <karanas@amazon.com>
* Add deprecations to retention pruning controls
This commit adds deprecation flags to all added settings, variables, and methods
specific to ccr's retention lease pruning mechanism.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* Addressed CR comments
Signed-off-by: Sai Kumar <karanas@amazon.com>
* fix javadoc deprecation
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* fix deprecation tag in TranslogDeletionPolicy
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* Addressed test issue under translog tests
Signed-off-by: Sai Kumar <karanas@amazon.com>
Co-authored-by: Nicholas Walter Knize <nknize@apache.org>
The changes ensure that in the event of a partial zone failure, the surviving nodes in the minority zone don't get overloaded with shards, this is governed by a skewness limit.
Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>
This will cause the leader stuck on IO during publication to step down and eventually trigger a leader election.
Issue Description
---
The publication of cluster state is time bound to 30s by a cluster.publish.timeout settings. If this time is reached before the new cluster state is committed, then the cluster state change is rejected and the leader considers itself to have failed. It stands down and starts trying to elect a new master.
There is a bug in leader that when it tries to publish the new cluster state it first tries acquire a lock to flush the new state under a mutex to disk. The same lock is used to cancel the publication on timeout. Below is the state of the timeout scheduler meant to cancel the publication. So essentially if the flushing of cluster state is stuck on IO, so will the cancellation of the publication since both of them share the same mutex. So leader will not step down and effectively block the cluster from making progress.
Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>
* Drop mocksocket in favour of custom security manager checks (tests only)
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
* Slightly relaxed host checks to allow all local addresses
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
Nick Knize
Andrew Ross
Owais Kazi
Vacha
Kyle J. Davis
Andriy Redko
Daniel Doubrovkine (dB.)
Rabi Panda
CEHENKLE
Saurabh Singh
Xue Zhou
Sergey Nuyanzin
Romain Tartière
Rishikesh Pasham
Shivansh Arora
Sai
Tianli Feng
Bukhtawar Khan