- Deprecate the request parameter `master_timeout` that used in Index APIs which have got the parameter.
(except index template APIs, which is addressed in PR https://github.com/opensearch-project/OpenSearch/pull/2678)
- Add alternative new request parameter `cluster_manager_timeout`.
- Add unit tests.
Signed-off-by: Tianli Feng <ftianli@amazon.com>
- Deprecate the request parameter `master_timeout` that used in Cluster APIs which have got the parameter.
- Add alternative new request parameter `cluster_manager_timeout`.
- Add unit tests.
Signed-off-by: Tianli Feng <ftianli@amazon.com>
Changes version comparison logic to only translate major version when
comparing with legacy 7x versions. This is needed beginning in 2.0 so
that when running 2.0+ versions in bwc mode for 3.0+ upgrades, node
versions no longer have to translate major version or spoof to legacy
versions.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Apply the change of CAT Nodes API in PR #2435 to other applicable CAT APIs.
- Deprecate the request parameter `master_timeout` that used in many CAT APIs.
- Add alternative new request parameter `cluster_manager_timeout`.
- Add unit tests.
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Move parseDeprecatedMasterTimeoutParameter method into BaseRestHandler class to reduce duplication
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Add more comments to unit test
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Make log message key different
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Prohibit using 'master_timeout' and 'cluster_manager_timeout' parameter together
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Add separate unit tests for BaseRestHandler.parseDeprecatedMasterTimeoutParameter()
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Restore unit test for cat allocation api
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Adjust format by spotlessApply task
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Fix testBothParamsNotValid() by adding warning assertion
Signed-off-by: Tianli Feng <ftianli@amazon.com>
Enables merge on refresh and merge on commit in Opensearch by
way of two new index options:
index.merge_on_flush.max_full_flush_merge_wait_time and
index.merge_on_flush.enabled. Default merge_on_flush is disabled and
wait time is 10s.
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
Change InboundDecoder ensureVersionCompatibility check for onOrAfter
V_2_0_0 instead of explicit version check. This way bug fix and minor
versions will correctly handshake in a mixed 1.x Cluster.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Adds support for using geo_bounding_box queries on geo_shape field types by
lifting the geo_point restriction in the QueryBuilder. Bounding Box query
integration tests are abstracted to test both geo_point and geo_shape types.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Adds support for using the geo_distance query on geo_shape field types by
lifting the geo_point restriction in the QueryBuilder. Distance query
integration tests are abstracted to test both geo_point and geo_shape field
types.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* Add `cluster_manager_node` into `ClusterState Metric`, as an alternative to `master_node`. So that the request parameter "metric" in `Cluster reroute` and `Cluster state` API accept the new value `cluster_manager_node`
* Deprecate the enum value of `Metric: MASTER_NODE("master_node")`
* Add YAML REST tests for the new parameter value for "cluster state" and "cluster reroute" API
Signed-off-by: Tianli Feng <ftianli@amazon.com>
Add a new node role `cluster_manager`, as the replacement for `master` role, that used in setting `node.roles: [ master ]`.
They have got the same functionality, but can NOT be assigned to `node.roles` together.
* Add `CLUSTER_MANAGER_ROLE` in `DiscoveryNodeRole` class, and deprecate `MASTER_ROLE`
* Remove `MASTER_ROLE` from the `DiscoveryNodeRole.BUILT_IN_ROLES`, and temporarily load the role by the existing method `DiscoveryNode.setAdditionalRoles`
* Add a method `validateRole()` in `DiscoveryNodeRole` class, it's used to validate a specific role is compatible with the other roles to be assigned to a node together.
* Add deprecation message when assigning `master` role in setting `node.roles`
* Replace most `MASTER_ROLE` with `CLUSTER_MANAGER_ROLE` in current unit and integration tests
* Add new unit and integration tests to validate `CLUSTER_MANAGER_ROLE` and `MASTER_ROLE` can be treated as the same.
More explanation:
* New node will have "cluster_manager", "data", and "ingest" roles by default. Which means the default value of setting "node.roles" will be `["cluster_manager","data","ingest"]`, instead of `["master","data","ingest"]`
* "cluster_manager" role will be treated as "master" role in the OpenSearch node of previous versions.
* "cluster_manager” role and "master" role have got the same abbreviation name: "m"
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Replace 'master' with 'cluster_manager' in the table header of [GET Cat Nodes](https://opensearch.org/docs/latest/opensearch/rest-api/cat/cat-nodes/) API, to promote inclusive language.
* Add `cm` as the alias for the table header `cluster_manager`
* Add `master` as the alias for the table header `cluster_manager`, for keeping compatibility when using `GET _cat/nodes?v&h=master` to show the specific column only.
* Add YAML rest test
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Replace 'discovered_master' with 'discovered_cluster_manager' in the table header of `GET Cat Health` API, to promote inclusive language.
* Add `dcm` as the alias for the table header `discovered_cluster_manager`
* Add `discovered_master` as the alias for the table header `discovered_cluster_manager`, for keeping compatibility when using `GET _cat/nodes?v&h=discovered_master` to show the specific column only.
* Add YAML rest test
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Add a new field discovered_cluster_manager to the response of `GET _cluster/health` API, which has got the same value with the existing discovered_master field, aims to replace the non-inclusive term "master" in the API response field.
* Add yaml rest test to validate `discovered_cluster_manager` and `discovered_master` field have got the same value.
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* `master_timeout` is being used in multiple APIs, such as cat allocation and cat nodes APIs. The PR take CAT Nodes API as a sample.
* Add a new request parameter `cluster_manager_timeout`, as the alternative for the existing `master_timeout`, in CAT Nodes API
* Throw an exception when both master_timeout and cluster_manager_timeout are assigned but with different values, such as `GET _cat/nodes?v&master_timeout=1s&cluster_manager_timeout=2s`. This is addressed by the temporary method `validateParamValuesAreEqual()` added in `RestRequest` class.
* Add deprecation log when using request parameter `master_timeout`
* Deprecate request parameter `master_timeout` of in rest api spec
* Add unit tests
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Add a new path `/_cat/cluster_manager` for the REST request handler `RestMasterAction`
* Deprecate the existing path `/_cat/master`
* Change the name of the `RestMasterAction` handler from `cat_master_action` to `cat_cluster_manager_action`
* Change the response of `GET _cat` to only show the new path `/_cat/cluster_manager`
* Rename the JSON rest-api-spec file `cat.master.json` to `cat.cluster_manager.json` and update the paths.
* Add YAML REST test. By default, it will run test for both endpoints `GET _cat/cluster_manager` and `GET _cat/master`
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Add a new setting `cluster.no_cluster_manager_block`, aims to replace the existing `cluster.no_master_block`
* Set the value of the old setting `cluster.no_master_block` as the fallback to the new setting
* Deprecate the existing `setting cluster.no_master_block`
* Add unit tests
Signed-off-by: Tianli Feng <ftianli@amazon.com>
Removes the type variable as input to CreateIndexRequest.mapping(Object...)
along with the CreateIndexRequestBuilder helper class. This also refactors the
method name to setMapping for consistency with other methods (e.g.,
setSettings).
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Continues removal of types from PutIndexTemplateRequest and
PutIndexTemplateRequestBuilder.mapping. Delegated mapping method in
PutIndexTemplateRequestBuilder is refactored to setMapping for consistency with
similar methods (e.g., setSettings, setAliases).
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
First pass to remove types from CreateIndexRequest and CreateIndexRequestBuilder
mapping method. This method is overloaded several times so the most widely used
methods in the RequestBuilder are refactored from mapping to setMapping to avoid
confusion, conflicts, and to be consistent with other method names (e.g.,
setSettings, setCause, setAlias).
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* [Upgrade] Lucene 9.1.0-snapshot-ea989fe8f30
Upgrades from Lucene 9.0.0 to 9.1.0-snapshot-ea989fe8f30 in preparation for
9.1.0 GA.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* Add spanishplural token filter
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* fix KNOWN_TOKENIZERS
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Unmutes IndexPrimaryRelocationIT.testPrimaryRelocationWhileIndexing which was
fixed by LuceneChangesSnapshot using accurate ops history.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Adds default implementation for getEngineFactory in EnginePlugin. The
default just returns Optional.empty(), allowing plugin developers to
implement this plugin without implementing this method.
Signed-off-by: John Mazanec <jmazane@amazon.com>
This commit upgrades the core codebase from Lucene 8.10.1 to
lucene 9.0.0. It includes all necessary refactoring of features and
API changes when upgrading to a new major Lucene release.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Co-authored-by: Andriy Redko <drreta@gmail.com>
Improves the LuceneChangesSnapshot to get an accurate count of recovery
operations using sort by sequence number optimization.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Removes the obsolete types based .addMapping method from CreateIndexRequest and
CreateIndexRequestBuilder. Tests are refactored to use the new source only
setMapping method.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Removes types from the TaskResults internal index along with the getMappings
method from IndexMetadata. This is needed to further remove types from
CreateIndexRequest.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* [Remove] Type from TermsLookUp
Signed-off-by: Suraj Singh <surajrider@gmail.com>
* Fix unit test failure
Signed-off-by: Suraj Singh <surajrider@gmail.com>
* Discrepancy in result from _validate/query API and actual query validity
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
* Moved the validate() check later into the flow to allow range validation to trigger first
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
Removes types from UID class along with cleaning up all obsolete MapperService
dependendices. This includes removing UID from the Translog Delete operation
which is no longer needed due to type dependency removal.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Removes multi-type support from IndexTemplateMetadata so that instead of holding
a map of multiple types to mappings, it only returns a single mapping for a
single type. Also removes type from documentMapper() method to avoid any
accidental NullPointerExceptions in the internal mapping retrieval.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
This inner class is no longer required because its functionality has been moved to the generic GatedCloseable class.
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Reducing duplication in plugins around ref-counted releasable classes
Both AmazonEc2Reference and AmazonS3Reference duplicate the same logic - a subclass of AbstractRefCounted that also implements Releasable. This change centralizes this paradigm into a AbstractRefCountedReleasable class and supports both clients via generics. It also updates all fetching implementations to use the get() method instead of client()
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Introduce Reference classes for the Closeable and AutoCloseable interfaces
These classes allow you to wrap a reference instance with an onClose runnable that is executed when close() is invoked. Two separate classes are needed because the close() signatures for the two interfaces are different. This change takes the first step to have implementing classes extend from these generic superclasses, before attempting to remove the subclasses entirely. The get() method is also replaced throughout the code base.
Note that there is also a separate Releasable interface that has a similar access pattern, but is implemented separately. This is used in AbstractRefCountedReleasable introduced in a prior commit
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* More improvements and refactoring
* Functionality around one-way gating is now moved to a dedicated class - OneWayGate. This replaces duplicate functionality throughout the code.
* The two *Reference classes have been renamed to Gated* since that better represents their functionality
* The AbstractRefCountedReleasable has been improved to no longer be abstract by accepting the shutdown hook. This removes the need for the inner class in ReleasableBytesReference, and further simplifies the plugin subclasses (these could probably be removed entirely).
* Finally, unit tests have been added for some classes
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Added tests for GatedCloseable
Also updated the license information in GatedAutoCloseableTests
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Fixing license information in new files
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Added unit tests for RefCountedReleasable
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* [Remove] TrimUnsafeCommit logic for legacy 6.x indexes
Multiple txlog commits was introduced in legacy 7.x. Legacy 6.x indexes could
therefore not have a safe commit. Since OpenSearch 2.0 is no longer compatible
with legacy 6.x indexes, the logic to trim these unsafe commits is safely
removed.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* fix assertion typo
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* rebase and incorporate pr feedback
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* MapperService has to be passed in as null for EnginePlugins CodecService constructor
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
* Addressing code review comments
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
* Delayed CodecService instantiation up to the shard initialization
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
* Added logger (associated with shard) to CodecServiceConfig
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
* Refactored the EngineConfigFactory / IndexShard instantiation of the CodecService
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
Removes include_type_name from the high level reset client along with relevant
deprecated methods in IndicesClient. All tests are updated to remove the
parameter from the rest requests along with various toXContent methods that are
no longer required.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* Replace the exclusionary word whitelist with allowlist, and blacklist with denylist, in code commet and internal variable/method/class/package name.
Signed-off-by: Andreas <apre@gmx.at>
* Add '_name' field support to score functions and provide it back in explanation response
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
* Address code review comments
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
Removes the deprecated types based get, getMapping, getAsync, and
getMappingAsync methods from IndicesClient. It also removes extra nesting of
mappings belong the deprecated type named object and removes the types based
methods from the affected request classes.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* Reintroduce negative epoch_millis #1991
Fixes a regression introduced with Elasticsearch 7 regarding the date
field type that removed support for negative timestamps with sub-second
granularity.
Thanks to Ryan Kophs (https://github.com/rkophs) for allowing me to use
his previous work.
Signed-off-by: Breno Faria <breno.faria@intrafind.de>
* applying spotless fix
Signed-off-by: Breno Faria <breno.faria@intrafind.de>
* more conservative implementation of isSupportedBy
Signed-off-by: Breno Faria <breno.faria@intrafind.de>
* adding braces to control flow statement
Signed-off-by: Breno Faria <breno.faria@intrafind.de>
* spotless fix...
Signed-off-by: Breno Faria <breno.faria@intrafind.de>
Co-authored-by: Breno Faria <breno.faria@intrafind.de>
Removes the type parameter from Client.prepare(Index,Delete,Update) and
everywhere it's used throughout the codebase except for prepareIndex(index,
type, id) which is removed in a follow up.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Removes type support from DocWrite Request and Response, all derived classes,
and all places used.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* [Remove] types from SearchHit and Explain API
Removes type support from SearchHit and Explain API.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* fix failing tests
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* move TYPES_DEPRECATION_MESSAGE from RestExplainAction to FullClusterRestartIT
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* fix FullClusterRestartIT
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* fix failing tests
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* fix testSoftDelete
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
With soft deletes no longer optional, peer recovery is switched to always use the
lucene index instead of replaying operations from the translog.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* Remove type param from yml and integ test files
Signed-off-by: Suraj Singh <surajrider@gmail.com>
* Remove type mapping specific update API
Signed-off-by: Suraj Singh <surajrider@gmail.com>
Removes type support from the get and mget transport and rest actions along
with removal from ShardGetService.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Default mappings were deprecated in Legacy version 6x and cannot be added to
indexes created in legacy 7+ or any version of OpenSearch. All support for
default mappings are removed for OpenSearch 2+.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
With types deprecation the type support is removed from internal search API
(SearchRequest and QueryShardContext).
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* Support unordered non-overlapping intervals
This commit exposes Intervals.unorderedNoOverlaps (LUCENE-8828).
Signed-off-by: Matt Weber <matt@mattweber.org>
* Replace ordered and overlap flags with enum.
- add IntervalMode enum
- deprecate ordered
- bwc between IntervalMode and ordered flag
- update tests
- update rest tests
Signed-off-by: Matt Weber <matt@mattweber.org>
* fix annotated text plugin test
Signed-off-by: Matt Weber <matt@mattweber.org>
* bwc to 2x only
Signed-off-by: Matt Weber <matt@mattweber.org>
* BWC rest tests for ordered flag
Signed-off-by: Matt Weber <matt@mattweber.org>
* Bump com.diffplug.spotless from 5.6.1 to 6.2.0
Bumps com.diffplug.spotless from 5.6.1 to 6.2.0.
---
updated-dependencies:
- dependency-name: com.diffplug.spotless
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* spotlessApply
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Nicholas Walter Knize <nknize@apache.org>
* Remove doc type specific indexing APIs and relevant changes
Signed-off-by: Suraj Singh <surajrider@gmail.com>
* Remove type param from yml and integ test files
Signed-off-by: Suraj Singh <surajrider@gmail.com>
Tianli Feng
Wenjun Ruan
Nick Knize
Suraj Singh
Mohit Godwani
Sruti Parthiban
Andriy Redko
Owais Kazi
Viggo
John Mazanec
Kartik
Kartik
Daniel Doubrovkine (dB.)
Subhobrata Dey
aponb
Breno Faria
Matt Weber
dependabot[bot]
Ankit Jain
Peng Huo
Andrey Pleskach