* Remove v6.5.x constants
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
* Address code review comments.
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
* Add back version assumption for testRemoteClusterSettingsUpgraded
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
* Remove unused import after rebase.
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
This commit removes the `LegacyVersion.v6.4.x` constants and their usages, since we no longer need to support those in OpenSearch 2.0 onwards.
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
I previously added a helper that started a MockLogAppender to ensure it
was never added to a Logger before it was started. I subsequently found
the opposite case in RolloverIT.java where the appender was stopped
before it was closed, therefore creating a race where a concurrently
running test in the same JVM could cause a logging failure. This seems
like a really easy mistake to make when writing a test or introduce when
refactoring a test. I've made a change to use try-with-resources to
ensure that proper setup and teardown is done. This should make it much
harder to introduce this particular test bug in the future.
Unfortunately, it did involve touching a lot of files. The changes here
are purely structural to leverage try-with-resources; no testing logic
has been changed.
Signed-off-by: Andrew Ross <andrross@amazon.com>
* Fixing allocation filters to persist existing state on settings update
Signed-off-by: Ankit Jain <jain.ankitk@gmail.com>
* Adding test for filter settings update
Signed-off-by: Ankit Jain <jain.ankitk@gmail.com>
* Adding more tests and review comments
Signed-off-by: Ankit Jain <jain.ankitk@gmail.com>
* Adding assertion and unit test for operation type mismatch
Signed-off-by: Ankit Jain <jain.ankitk@gmail.com>
* Updating test names
Signed-off-by: Ankit Jain <jain.ankitk@gmail.com>
This commit removes LegacyESVersion.V_6_3_x constants including all
pre-release versions and bug fixes.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* Bridging the gap in network overhead measurement in the profiler
Signed-off-by: Poojita Raj <poojiraj@amazon.com>
* add tests + remove total n/w time field
Signed-off-by: Poojita Raj <poojiraj@amazon.com>
* refactor network time into a class
Signed-off-by: Poojita Raj <poojiraj@amazon.com>
* gradle test fix
Signed-off-by: Poojita Raj <poojiraj@amazon.com>
This commit removes the following deprecations and invalid tests:
* LegacyUpdateMappingIntegrationIT
* ShapeBuilder new z axis checks
* MatchPhrase new zeroTermsQuery checks
* TypeQuery utf8
* CompositeValuesSource format checks
* LegacyESVersionTests bumped to 6.8.15
* LegacyDynamicMapping, LegacySimilarity, LegacyMapperService tests
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
This commit removes unnecessary serialization of unused flavor variable in build
metadata from V_1_3_0+
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
This commit removes legacy version checks in DynamicTemplate parsing that are no
longer valid in OpenSearch 2.0.0.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
This commit removes deprecated analyzer instantiation that is no longer
permitted in OpenSearch 2.0.0.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* Changes made by adding the Task indicator and the IllegalArgs.
Signed-off-by: meghasaik <kavmegha@amazon.com>
* Changes made by changing the String message for uncancellable task.
Signed-off-by: meghasaik <kavmegha@amazon.com>
* Increase the max waiting time before checking the FS health after restoring the FS status in the test
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Remove another duplicate assetion statement
Signed-off-by: Tianli Feng <ftianli@amazon.com>
This commit removes LegacyESVersion.V_6_2_x constants including all
pre-release versions and bug fixes.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
This commit removes LegacyESVersion.V_6_1_x constants including all
pre-release versions and bug fixes.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
This PR removes LegacyESVersion.V_6_0_* constants including all pre-release
versions and bug fixes.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
In preparation for removing all LegacyESVersion support by 3.0; this commit
largely refactors the LegacyESVersion test logic from the OpenSearch Version
test logic into an independent test class. This PR also updates Version.fromString
to ensure a proper legacy version is returned when major is > 3 (to support
legacy yaml test and build scripts).
Note that bwc w/ legacy versions are still supported so some cross compatibility
testing is retained in the Version test class.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Zen1 discovery was deprecated in Legacy 7.x for eventual removal. OpenSearch 1.x
carries this deprecation. This commit completely removes all support for Zen1
discovery in favor of Zen2.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Lucene 9 removes support for SimpleFS File System format. This PR completely
removes SimpleFS support which was deprecated in a previous PR.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* Signed-off-by: Megha Sai Kavikondala <kavmegha@amazon.com>
Changes related to Informative error messages.
* Signed-off-by: Megha Sai Kavikondala <kavmegha@amazon.com>
Spotless changes
* Signed-off-by: Megha Sai Kavikondala <kavmegha@amazon.com>
naming and message changes.
Signed-off-by: Megha Sai Kavikondala <kavmegha@amazon.com>
* Signed-off-by: Megha Sai Kavikondala <kavmegha@amazon.com>
Deleting the log message.
I observed a test failure with the message
'Attempted to append to non-started appender mock' from an assertion in
`OpenSearchTestCase::after`. I believe this indicates that a
MockLogAppender (which is named "mock") was added as an appender to the
static logging context and some other test in the same JVM happened to
cause a logging statement to hit that appender and cause an error, which
then caused an unrelated test to fail (because they share static state
with the logger). Almost all usages of MockLogAppender start it
immediately after creation. I found a few that did not and fixed those.
I also made a static helper in MockLogAppender to start it upon
creation.
Signed-off-by: Andrew Ross <andrross@amazon.com>
* Added logic to allow {dot} files on startup
Signed-off-by: Ryan Bogan <rbogan@amazon.com>
* Ensures that only plugin directories are returned by findPluginDirs()
Signed-off-by: Ryan Bogan <rbogan@amazon.com>
* Prevents . files from being returned as plugins
Signed-off-by: Ryan Bogan <rbogan@amazon.com>
As part of the commit 2ebd0e04, we added a new method to the EnginePlugin to provide a
custom TranslogDeletionPolicy. This commit makes minTranslogGenRequired method
abstract in this class for implementation by child classes. The default implementation
is provided by DefaultTranslogDeletionPolicy.
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
The settings and the corresponding logic for translog pruning by retention lease which were added as part of #1100 have been deprecated. This commit removes those deprecated code in favor of an extension point for providing a custom TranslogDeletionPolicy.
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
When creating the second instance of an InternalEngine using the same
translog config of the default InternalEngine instance, the second
instance will attempt to delete all the existing translog files. I found
a deterministic test failure when running with the seed
`E3E6AAD95ABD299B`.
As opposed to creating a second engine instance with a different
translog location, just close the first one before creating the second.
Signed-off-by: Andrew Ross <andrross@amazon.com>
This commit adds a method that can be used to provide a custom TranslogDeletionPolicy
from within plugins that implement the EnginePlugin interface. This enables plugins to
provide a custom deletion policy with the current limitation that only one plugin can
override the policy. An exception will be thrown if more than one plugin overrides the
policy.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
This commit adds an explicit version check for shard indexing pressure
serialization. This is required to not mandate test have the cluster service
initialized while asserting node attributes.
Signed-off-by: Saurabh Singh <sisurab@amazon.com>
Co-authored-by: Saurabh Singh <sisurab@amazon.com>
This commit adds an extension point to EngineConfig through EnginePlugin using
a new EngineConfigFactory mechanism. EnginePlugin provides interface methods to
override configurations in EngineConfig. The EngineConfigFactory produces a new
instance of the EngineConfig using these overrides. Defaults are used absent
overridden configurations.
This serves as a mechanism to override Engine configurations (e.g., CodecService,
TranslogConfig) enabling Plugins to have higher fidelity for changing Engine
behavior without having to override the entire Engine (which is only permitted for
a single plugin).
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Fixes flakiness for test testReplicaThreadedThroughputDegradationAndRejection.
Reduced the number of threads concurrently executing from the initial range between (100-120) to a new range between (80-100), as the previous range was breaking the node limits set as 10kb, for every execution where number of threads were greater than 110.
Signed-off-by: Saurabh Singh <sisurab@amazon.com>
* Modernize and consolidate JDKs usage across all stages of the build. Use JDK-17 as bundled JDK distribution to run tests
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
* Using -Djava.security.egd=file:/dev/urandom explicitly for cli tests
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
Shard level indexing pressure improves the current Indexing Pressure framework which performs memory accounting at node level and rejects the requests. This takes a step further to have rejections based on the memory accounting at shard level along with other key performance factors like throughput and last successful requests.
**Key features**
- Granular tracking of indexing tasks performance, at every shard level, for each node role i.e. coordinator, primary and replica.
- Smarter rejections by discarding the requests intended only for problematic index or shard, while still allowing others to continue (fairness in rejection).
- Rejections thresholds governed by combination of configurable parameters (such as memory limits on node) and dynamic parameters (such as latency increase, throughput degradation).
- Node level and shard level indexing pressure statistics exposed through stats api.
- Integration of Indexing pressure stats with Plugins for for metric visibility and auto-tuning in future.
- Control knobs to tune to the key performance thresholds which control rejections, to address any specific requirement or issues.
- Control knobs to run the feature in shadow-mode or enforced-mode. In shadow-mode only internal rejection breakdown metrics will be published while no actual rejections will be performed.
The changes were divided into small manageable chunks as part of the following PRs against a feature branch.
- Add Shard Indexing Pressure Settings. #716
- Add Shard Indexing Pressure Tracker. #717
- Refactor IndexingPressure to allow extension. #718
- Add Shard Indexing Pressure Store #838
- Add Shard Indexing Pressure Memory Manager #945
- Add ShardIndexingPressure framework level construct and Stats #1015
- Add Indexing Pressure Service which acts as orchestrator for IP #1084
- Add plumbing logic for IndexingPressureService in Transport Actions. #1113
- Add shard indexing pressure metric/stats via rest end point. #1171
- Add shard indexing pressure integration tests. #1198
Signed-off-by: Saurabh Singh <sisurab@amazon.com>
Co-authored-by: Saurabh Singh <sisurab@amazon.com>
Co-authored-by: Rabi Panda <adnapibar@gmail.com>
* Support for Heap after GC stats
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
* Addressing code review comments
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
* Using the right version 2.0.0 (instead of 1.2.0) for the change
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
This commit adds test coverage for translog pruning setting to
MetadataCreateIndexServiceTests
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* Rename translog pruing setting to CCR specific setting
Signed-off-by: Sai Kumar <karanas@amazon.com>
* Rename to index.plugins.replication.translog.retention_lease.pruning.enabled as
index settings needs "index." as prefix
Signed-off-by: Sai Kumar <karanas@amazon.com>
* Add deprecations to retention pruning controls
This commit adds deprecation flags to all added settings, variables, and methods
specific to ccr's retention lease pruning mechanism.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* Addressed CR comments
Signed-off-by: Sai Kumar <karanas@amazon.com>
* fix javadoc deprecation
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* fix deprecation tag in TranslogDeletionPolicy
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* Addressed test issue under translog tests
Signed-off-by: Sai Kumar <karanas@amazon.com>
Co-authored-by: Nicholas Walter Knize <nknize@apache.org>
The changes ensure that in the event of a partial zone failure, the surviving nodes in the minority zone don't get overloaded with shards, this is governed by a skewness limit.
Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>
This will cause the leader stuck on IO during publication to step down and eventually trigger a leader election.
Issue Description
---
The publication of cluster state is time bound to 30s by a cluster.publish.timeout settings. If this time is reached before the new cluster state is committed, then the cluster state change is rejected and the leader considers itself to have failed. It stands down and starts trying to elect a new master.
There is a bug in leader that when it tries to publish the new cluster state it first tries acquire a lock to flush the new state under a mutex to disk. The same lock is used to cancel the publication on timeout. Below is the state of the timeout scheduler meant to cancel the publication. So essentially if the flushing of cluster state is stuck on IO, so will the cancellation of the publication since both of them share the same mutex. So leader will not step down and effectively block the cluster from making progress.
Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>
* Drop mocksocket in favour of custom security manager checks (tests only)
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
* Slightly relaxed host checks to allow all local addresses
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
* Changes to support retrieval of operations from translog based on specified range
Signed-off-by: Sai Kumar <karanas@amazon.com>
* Addressed CR comments
Signed-off-by: Sai Kumar <karanas@amazon.com>
* Added testcases for internal engine
Signed-off-by: Sai Kumar <karanas@amazon.com>
* Support for translog pruning based on retention leases
Signed-off-by: Sai Kumar <karanas@amazon.com>
* Addressed CR Comments
Signed-off-by: Sai Kumar <karanas@amazon.com>
* Addressed test case issue
Signed-off-by: Sai Kumar <karanas@amazon.com>
The version framework only added support for OpenSearch 1.x bwc with legacy
clusters. This commit adds support for v2.0 which will be the last version with
bwc support for legacy clusters (v7.10)
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
This commit stages the branch to the next 1.0.1 patch release. BWC testing needs
this even if the next revision is never actually released.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
PluginInfo should use .onOrAfter(Version.V_1_1_0) instead of
.after(Version.V_1_0_0) for the new custom folder name for plugin feature.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Lucene 9 removes support for SimpleFS File System format. This commit deprecates
the SimpleFS format in favor of NIOFS.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
In some cases as one shared with issue #1099, the maxConcurrentSearchRequests was chosen as 0 which
will compute the final value during execution of the request based on processor counts. When this
computed value is less than number of search request in msearch request, it will execute all the
requests in multiple iterations causing the failure since test will only wait for one such
iteration. Hence setting the maxConcurrentSearchRequests explicitly to number of search requests
being added in the test to ensure correct behavior
Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com>
* Part 1: Support for cancel_after_timeinterval parameter in search and msearch request
This commit introduces the new request level parameter to configure the timeout interval after which
a search request will be cancelled. For msearch request the parameter is supported both at parent
request and at sub child search requests. If it is provided at parent level and child search request
doesn't have it then the parent level value is set at such child request. The parent level msearch
is not used to cancel the parent request as it may be tricky to come up with correct value in cases
when child search request can have different runtimes
TEST: Added test for ser/de with new parameter
Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com>
* Part 2: Support for cancel_after_timeinterval parameter in search and msearch request
This commit adds the handling of the new request level parameter and schedule cancellation task. It
also adds a cluster setting to set a global cancellation timeout for search request which will be
used in absence of request level timeout.
TEST: Added new tests in SearchCancellationIT
Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com>
* Address Review feedback for Part 1
Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com>
* Address review feedback for Part 2
Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com>
* Update CancellableTask to remove the cancelOnTimeout boolean flag
Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com>
* Replace search.cancellation.timeout cluster setting with search.enforce_server.timeout.cancellation to control if cluster level cancel_after_time_interval should take precedence over request level cancel_after_time_interval value
Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com>
* Removing the search.enforce_server.timeout.cancellation cluster setting and just keeping search.cancel_after_time_interval setting with request level parameter taking the precedence.
Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com>
Co-authored-by: Sorabh Hamirwasia <hsorabh@amazon.com>
_cat/master is a fundamental API to know the master instance in the cluster. Given RestClusterState is exempted from tripping already, doesn't make sense for RestMasterAction to trip
Signed-off-by: Bukhtawar Khan bukhtawa@amazon.com
This change adds the initial version of a new CLI tool `opensearch-upgrade` as part of the OpenSearch distribution. This tool is meant for assisting during an upgrade from an existing Elasticsearch v7.10.2/v6.8.0 node to OpenSearch. It automates the process of importing existing configurations and installing of core plugins.
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
* Lower build requirement from Java 14+ to Java 11+
Avoid use of -Werror -Xlint:all, which may change significantly across
java releases (new warnings could be added). Instead, just list the
warnings individually.
Workaround JDK 11 compiler bug (JDK-8209058) that only impacts test fixture
code in the build itself.
Signed-off-by: Robert Muir <rmuir@apache.org>
* Disable warning around -source 7 -release 7 for java version checker
The java version checker triggers some default warnings because it
targets java7:
```
> Task :distribution:tools:java-version-checker:compileJava FAILED
warning: [options] source value 7 is obsolete and will be removed in a future release
warning: [options] target value 7 is obsolete and will be removed in a future release
warning: [options] To suppress warnings about obsolete options, use -Xlint:-options.
error: warnings found and -Werror specified
```
Suppress this warning explicitly for this module.
Signed-off-by: Robert Muir <rmuir@apache.org>
* more java14 -> java11 cleanup
Signed-off-by: Robert Muir <rmuir@apache.org>
Co-authored-by: Robert Muir <rmuir@apache.org>
Rabi Panda
Andriy Redko
sharp-pixel
Tianli Feng
Suraj Singh
Andrew Ross
Ankit Jain
Nick Knize
Daniel Doubrovkine (dB.)
Poojita Raj
Megha Sai Kavikondala
Vacha
Sarat Vemulapalli
Tianli Feng
Xue Zhou
Ryan Bogan
Saurabh Singh
Owais Kazi
Shivansh Arora
Sai
Bukhtawar Khan
Abbas Hussain
Nick Knize
Sorabh
Sorabh
frotsch
Vlad Rozov
afazel
Chang Liu
Rabi Panda
shwetathareja