* Use set to make shell scripts more strict
Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>
* Change -o pipefail to -e
Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>
* Set scripts to standard rule
Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>
* Replace 'master' terminology with 'cluster manager' in 'qa' directory
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Replace master_node with cluster_manager_node in cluster.state API test
Signed-off-by: Tianli Feng <ftianli@amazon.com>
- Deprecate the request parameter `master_timeout` that used in Index APIs which have got the parameter.
(except index template APIs, which is addressed in PR https://github.com/opensearch-project/OpenSearch/pull/2678)
- Add alternative new request parameter `cluster_manager_timeout`.
- Add unit tests.
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* [Upgrade] Lucene 9.1.0-snapshot-ea989fe8f30
Upgrades from Lucene 9.0.0 to 9.1.0-snapshot-ea989fe8f30 in preparation for
9.1.0 GA.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* Add spanishplural token filter
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* fix KNOWN_TOKENIZERS
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Removes multi-type support from IndexTemplateMetadata so that instead of holding
a map of multiple types to mappings, it only returns a single mapping for a
single type. Also removes type from documentMapper() method to avoid any
accidental NullPointerExceptions in the internal mapping retrieval.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* [Remove] TrimUnsafeCommit logic for legacy 6.x indexes
Multiple txlog commits was introduced in legacy 7.x. Legacy 6.x indexes could
therefore not have a safe commit. Since OpenSearch 2.0 is no longer compatible
with legacy 6.x indexes, the logic to trim these unsafe commits is safely
removed.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* fix assertion typo
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* rebase and incorporate pr feedback
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Removes include_type_name from the high level reset client along with relevant
deprecated methods in IndicesClient. All tests are updated to remove the
parameter from the rest requests along with various toXContent methods that are
no longer required.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
This test is slated for removal as it only applies to ancient indexes (Legacy
6.x). Muting test so bwc tests are consistent and no longer angry.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Removes type support from DocWrite Request and Response, all derived classes,
and all places used.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* [Remove] types from SearchHit and Explain API
Removes type support from SearchHit and Explain API.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* fix failing tests
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* move TYPES_DEPRECATION_MESSAGE from RestExplainAction to FullClusterRestartIT
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* fix FullClusterRestartIT
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* fix failing tests
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* fix testSoftDelete
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* Remove type param from yml and integ test files
Signed-off-by: Suraj Singh <surajrider@gmail.com>
* Remove type mapping specific update API
Signed-off-by: Suraj Singh <surajrider@gmail.com>
Removes type support from the get and mget transport and rest actions along
with removal from ShardGetService.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* Bump com.diffplug.spotless from 5.6.1 to 6.2.0
Bumps com.diffplug.spotless from 5.6.1 to 6.2.0.
---
updated-dependencies:
- dependency-name: com.diffplug.spotless
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* spotlessApply
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Nicholas Walter Knize <nknize@apache.org>
* Remove doc type specific indexing APIs and relevant changes
Signed-off-by: Suraj Singh <surajrider@gmail.com>
* Remove type param from yml and integ test files
Signed-off-by: Suraj Singh <surajrider@gmail.com>
Remove the deprecated sync flush API which was replaced by sequence
number and retention lease mechanisms and no longer used in 2.0.0.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Soft Deletes have been enabled by default since Legacy version 7.0 and made
mandatory in Version 2.0.0. This commit refactors the InternalEngine to always
use soft-deletes. It is a follow on to making soft deletes mandatory in 2.0.0.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* Remove v6.5.x constants
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
* Address code review comments.
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
* Add back version assumption for testRemoteClusterSettingsUpgraded
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
* Remove unused import after rebase.
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
This commit removes the `LegacyVersion.v6.4.x` constants and their usages, since we no longer need to support those in OpenSearch 2.0 onwards.
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
This commit removes LegacyESVersion.V_6_1_x constants including all
pre-release versions and bug fixes.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
This PR removes LegacyESVersion.V_6_0_* constants including all pre-release
versions and bug fixes.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* Made changes.
Signed-off-by: Megha Sai Kavikondala <kavmegha@amazon.com>
* Signed-off-by: Megha Sai Kavikondala <kavmegha@amazon.com>
Changes made by deleting the TestSettingsIT file and adding new lines in FullClusterRestartSettingsUpgradeIT.java
* Added logic to allow {dot} files on startup
Signed-off-by: Ryan Bogan <rbogan@amazon.com>
* Ensures that only plugin directories are returned by findPluginDirs()
Signed-off-by: Ryan Bogan <rbogan@amazon.com>
* Prevents . files from being returned as plugins
Signed-off-by: Ryan Bogan <rbogan@amazon.com>
* Modernize and consolidate JDKs usage across all stages of the build. Use JDK-17 as bundled JDK distribution to run tests
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
* Using -Djava.security.egd=file:/dev/urandom explicitly for cli tests
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
Shard level indexing pressure improves the current Indexing Pressure framework which performs memory accounting at node level and rejects the requests. This takes a step further to have rejections based on the memory accounting at shard level along with other key performance factors like throughput and last successful requests.
**Key features**
- Granular tracking of indexing tasks performance, at every shard level, for each node role i.e. coordinator, primary and replica.
- Smarter rejections by discarding the requests intended only for problematic index or shard, while still allowing others to continue (fairness in rejection).
- Rejections thresholds governed by combination of configurable parameters (such as memory limits on node) and dynamic parameters (such as latency increase, throughput degradation).
- Node level and shard level indexing pressure statistics exposed through stats api.
- Integration of Indexing pressure stats with Plugins for for metric visibility and auto-tuning in future.
- Control knobs to tune to the key performance thresholds which control rejections, to address any specific requirement or issues.
- Control knobs to run the feature in shadow-mode or enforced-mode. In shadow-mode only internal rejection breakdown metrics will be published while no actual rejections will be performed.
The changes were divided into small manageable chunks as part of the following PRs against a feature branch.
- Add Shard Indexing Pressure Settings. #716
- Add Shard Indexing Pressure Tracker. #717
- Refactor IndexingPressure to allow extension. #718
- Add Shard Indexing Pressure Store #838
- Add Shard Indexing Pressure Memory Manager #945
- Add ShardIndexingPressure framework level construct and Stats #1015
- Add Indexing Pressure Service which acts as orchestrator for IP #1084
- Add plumbing logic for IndexingPressureService in Transport Actions. #1113
- Add shard indexing pressure metric/stats via rest end point. #1171
- Add shard indexing pressure integration tests. #1198
Signed-off-by: Saurabh Singh <sisurab@amazon.com>
Co-authored-by: Saurabh Singh <sisurab@amazon.com>
Co-authored-by: Rabi Panda <adnapibar@gmail.com>
Owais Kazi
Tianli Feng
dependabot[bot]
Wenjun Ruan
Andriy Redko
Nick Knize
Suraj Singh
Venkata Jyothsna Donapati
Rabi Panda
Vacha
Ryan Bogan
Megha Sai Kavikondala
Himanshu Setia
Saurabh Singh