Adds javadocs to classes in the org.opensearch.monitor, persistence, plugins,
repository, script, threadpool, usage, and watcher packages.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Adds javadocs to classes in the org.opensearch.discovery, env, gateway, http,
ingest, lucene, and node packages.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
The types exist transport action can be removed now that the TransportClient has
been removed and types support has been removed.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
In preparation for re-enabling the missingJavadoc gradle task this change adds
in the missing package-info.java files to the server folder. For now general
javadocs are added to these files with the intent to clean up with better
descriptions over time.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Refactors XContentType.fromMediaTypeOrFormat to fromMediaType so Accept headers
and Content-Type headers can be parsed separately. This helps in reusing the
same parse logic in for REST Versioning API support.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* Replace internal usages of 'master' terminology in server/src/main directory
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Restore rename DISCOVERED_MASTER in ClusterHealthResponse
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Rename two methods in unit tests
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Replace master word in ClusterState
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Replace master word in LeaderChecker JoinHelper JoinTaskExecutor
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Replace master word in more classes
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Replace master word in more classes
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Replace master word in more classes
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Replace master word in more classes
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Replace master word in more classes
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Replace master word in more classes
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Replace master word in DiscoveryNodes classes
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Replace master word in more classes
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Correct mistakes
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Adjust format by spotlessApply task
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Change MASTER__NODE_BOOTSTRAPPED_MSG in test
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Fix SnapshotDisruptionIT by renaming to cluster-manager
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* [Remove] Type from nested fields using new metadata field mapper
types support is removed yet nested documents use the _type field to store the
path for nested documents. A new _nested_path metadata field mapper is added to
take the place of the _type field in order to remove the type dependency in
nested documents. BWC is handled in the new field mapper to ensure compatibility
with older versions.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* pr fixes
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* add test to merge same mapping with empty index settings
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* This change formalizes the notion of feature flags, and adds a "replication type" setting that will differentiate between document and segment replication, gated by a feature flag.
Since seg-rep is currently an incomplete implementation, the feature flag ensures that the setting is not visible to users without explicitly setting a system property. We can then continue to merge seg-rep related changes from the feature branch to `main` safely hidden behind the feature flag gate.
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Update security policy for testing feature flags
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Co-authored-by: Nicholas Walter Knize <nknize@apache.org>
A few places still referenced legacy ESTestCase name. This refactors those
instances to OpenSearchTestCase.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
OpenSearch 2.0.0 no longer needs HLRC compatibility with legacy clients. This
commit removes all logic to spoof the version as a legacy cluster.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Adds a new multi_term aggregation. The current implementation focuses
on adding new type aggregates. Performance (latency) is suboptimal in this
iteration, mainly because of brute force encoding/decoding a list of values
into bucket keys. A performance improvement change will be made as a
follow on.
Signed-off-by: Peng Huo <penghuo@gmail.com>
* Refactoring GatedAutoCloseable to AutoCloseableRefCounted
This is a part of the process of merging our feature branch - feature/segment-replication - back into main by re-PRing our changes from the feature branch.
GatedAutoCloseable currently wraps a subclass of RefCounted. Segment replication adds another subclass, but this also wraps RefCounted. Both subclasses have the same shutdown hook - decRef. This change makes the superclass less generic to increase code convergence.
The breakdown of the plan to merge segment-replication to main is detailed in #2355
Segment replication design proposal - #2229
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Minor refactoring in RecoveryState
This change makes two minor updates to RecoveryState -
1. The readRecoveryState API is removed because it can be replaced by an invocation of the constructor
2. The class members of the Timer inner class are changed to private, and accesses are only through the public APIs
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Update RecoveryTargetTests to test Timer subclasses deterministically
This change removes the use of RandomBoolean in testing the Timer classes and creates a dedicated unit test for each. The common test logic is shared via a private method.
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Move the RecoveryState.Timer class to a top-level class
This will eventually be reused across both replication use-cases - peer recovery and segment replication.
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Further update of timer tests in RecoveryTargetTests
Removes a non-deterministic code path around stopping the timer, and avoids assertThat (deprecated)
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Rename to ReplicationTimer
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Remove RecoveryTargetTests assert on a running timer
Trying to serialize and deserialize a running Timer instance, and then checking for equality leads to flaky test failures when the ser/deser takes time.
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
AllFieldMapper was deprecated in legacy 6x. The remaining references are
removed, along with the field mapper and corresponding tests.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* Adding a null pointer check to fix index_prefix query
Signed-off-by: Vacha Shah <vachshah@amazon.com>
* Adding test
Signed-off-by: Vacha Shah <vachshah@amazon.com>
- Deprecate the request parameter `master_timeout` that used in Ingest APIs and Script APIs which have got the parameter.
- Add alternative new request parameter `cluster_manager_timeout`.
- Add unit tests.
Signed-off-by: Tianli Feng <ftianli@amazon.com>
This change refactors an earlier change to impose a reg-ex size limit on the include/exclude string. Instead of accepting an IndexSettings instance, the class now accepts a integer limit value. This is necessary because the IncludeExclude class is used outside the core codebase, whose use-cases may be unaware of indices and their settings. To ensure that a limit is always imposed, a default limit is defined in the class.
(cherry picked from commit ba1966853e728b153e42be59ba449420e79b09ee)
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
Co-authored-by: Kartik Ganesh <gkart@amazon.com>
* Deprecate the request parameter `master_timeout` that used in Snapshot APIs which have got the parameter.
* Add alternative new request parameter `cluster_manager_timeout`.
* Add unit tests.
Signed-off-by: Tianli Feng <ftianli@amazon.com>
- Deprecate the request parameter `master_timeout` that used in Index Template APIs which have got the parameter.
(The other Index APIs are addressed in PR https://github.com/opensearch-project/OpenSearch/pull/2660)
- Add alternative new request parameter `cluster_manager_timeout`.
- Add unit tests.
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Updates to the large string reg-ex check
Removed the null-case for IndexSettings since this only occurs in tests. The tests now use a dummy Index Setting.
This change also fixes a bug with the base case handling of max regex length in the check.
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Bugfix to guard against stack overflow errors caused by very large reg-ex input
This change fixes a code path that did not properly impose the index-level max_regex_length limit. Therefore, it was possibly to provide ar arbitrarily large string as the include/exclude reg-ex value under search aggregations. This exposed the underlying node to crashes from a StackOverflowError, due to how the Lucene RegExp class processes strings using stack frames.
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Adding integration tests for large string RegEx
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Spotless
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Fix issue that deprecated setting 'cluster.initial_master_nodes' is not identified during node bootstrap
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Restore a variable name
Signed-off-by: Tianli Feng <ftianli@amazon.com>
- Deprecate the request parameter `master_timeout` that used in Index APIs which have got the parameter.
(except index template APIs, which is addressed in PR https://github.com/opensearch-project/OpenSearch/pull/2678)
- Add alternative new request parameter `cluster_manager_timeout`.
- Add unit tests.
Signed-off-by: Tianli Feng <ftianli@amazon.com>
- Deprecate the request parameter `master_timeout` that used in Cluster APIs which have got the parameter.
- Add alternative new request parameter `cluster_manager_timeout`.
- Add unit tests.
Signed-off-by: Tianli Feng <ftianli@amazon.com>
Changes version comparison logic to only translate major version when
comparing with legacy 7x versions. This is needed beginning in 2.0 so
that when running 2.0+ versions in bwc mode for 3.0+ upgrades, node
versions no longer have to translate major version or spoof to legacy
versions.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Apply the change of CAT Nodes API in PR #2435 to other applicable CAT APIs.
- Deprecate the request parameter `master_timeout` that used in many CAT APIs.
- Add alternative new request parameter `cluster_manager_timeout`.
- Add unit tests.
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Move parseDeprecatedMasterTimeoutParameter method into BaseRestHandler class to reduce duplication
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Add more comments to unit test
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Make log message key different
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Prohibit using 'master_timeout' and 'cluster_manager_timeout' parameter together
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Add separate unit tests for BaseRestHandler.parseDeprecatedMasterTimeoutParameter()
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Restore unit test for cat allocation api
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Adjust format by spotlessApply task
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Fix testBothParamsNotValid() by adding warning assertion
Signed-off-by: Tianli Feng <ftianli@amazon.com>
Enables merge on refresh and merge on commit in Opensearch by
way of two new index options:
index.merge_on_flush.max_full_flush_merge_wait_time and
index.merge_on_flush.enabled. Default merge_on_flush is disabled and
wait time is 10s.
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
Change InboundDecoder ensureVersionCompatibility check for onOrAfter
V_2_0_0 instead of explicit version check. This way bug fix and minor
versions will correctly handshake in a mixed 1.x Cluster.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Adds support for using geo_bounding_box queries on geo_shape field types by
lifting the geo_point restriction in the QueryBuilder. Bounding Box query
integration tests are abstracted to test both geo_point and geo_shape types.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Adds support for using the geo_distance query on geo_shape field types by
lifting the geo_point restriction in the QueryBuilder. Distance query
integration tests are abstracted to test both geo_point and geo_shape field
types.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* Add `cluster_manager_node` into `ClusterState Metric`, as an alternative to `master_node`. So that the request parameter "metric" in `Cluster reroute` and `Cluster state` API accept the new value `cluster_manager_node`
* Deprecate the enum value of `Metric: MASTER_NODE("master_node")`
* Add YAML REST tests for the new parameter value for "cluster state" and "cluster reroute" API
Signed-off-by: Tianli Feng <ftianli@amazon.com>
Add a new node role `cluster_manager`, as the replacement for `master` role, that used in setting `node.roles: [ master ]`.
They have got the same functionality, but can NOT be assigned to `node.roles` together.
* Add `CLUSTER_MANAGER_ROLE` in `DiscoveryNodeRole` class, and deprecate `MASTER_ROLE`
* Remove `MASTER_ROLE` from the `DiscoveryNodeRole.BUILT_IN_ROLES`, and temporarily load the role by the existing method `DiscoveryNode.setAdditionalRoles`
* Add a method `validateRole()` in `DiscoveryNodeRole` class, it's used to validate a specific role is compatible with the other roles to be assigned to a node together.
* Add deprecation message when assigning `master` role in setting `node.roles`
* Replace most `MASTER_ROLE` with `CLUSTER_MANAGER_ROLE` in current unit and integration tests
* Add new unit and integration tests to validate `CLUSTER_MANAGER_ROLE` and `MASTER_ROLE` can be treated as the same.
More explanation:
* New node will have "cluster_manager", "data", and "ingest" roles by default. Which means the default value of setting "node.roles" will be `["cluster_manager","data","ingest"]`, instead of `["master","data","ingest"]`
* "cluster_manager" role will be treated as "master" role in the OpenSearch node of previous versions.
* "cluster_manager” role and "master" role have got the same abbreviation name: "m"
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Replace 'master' with 'cluster_manager' in the table header of [GET Cat Nodes](https://opensearch.org/docs/latest/opensearch/rest-api/cat/cat-nodes/) API, to promote inclusive language.
* Add `cm` as the alias for the table header `cluster_manager`
* Add `master` as the alias for the table header `cluster_manager`, for keeping compatibility when using `GET _cat/nodes?v&h=master` to show the specific column only.
* Add YAML rest test
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Replace 'discovered_master' with 'discovered_cluster_manager' in the table header of `GET Cat Health` API, to promote inclusive language.
* Add `dcm` as the alias for the table header `discovered_cluster_manager`
* Add `discovered_master` as the alias for the table header `discovered_cluster_manager`, for keeping compatibility when using `GET _cat/nodes?v&h=discovered_master` to show the specific column only.
* Add YAML rest test
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Add a new field discovered_cluster_manager to the response of `GET _cluster/health` API, which has got the same value with the existing discovered_master field, aims to replace the non-inclusive term "master" in the API response field.
* Add yaml rest test to validate `discovered_cluster_manager` and `discovered_master` field have got the same value.
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* `master_timeout` is being used in multiple APIs, such as cat allocation and cat nodes APIs. The PR take CAT Nodes API as a sample.
* Add a new request parameter `cluster_manager_timeout`, as the alternative for the existing `master_timeout`, in CAT Nodes API
* Throw an exception when both master_timeout and cluster_manager_timeout are assigned but with different values, such as `GET _cat/nodes?v&master_timeout=1s&cluster_manager_timeout=2s`. This is addressed by the temporary method `validateParamValuesAreEqual()` added in `RestRequest` class.
* Add deprecation log when using request parameter `master_timeout`
* Deprecate request parameter `master_timeout` of in rest api spec
* Add unit tests
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Add a new path `/_cat/cluster_manager` for the REST request handler `RestMasterAction`
* Deprecate the existing path `/_cat/master`
* Change the name of the `RestMasterAction` handler from `cat_master_action` to `cat_cluster_manager_action`
* Change the response of `GET _cat` to only show the new path `/_cat/cluster_manager`
* Rename the JSON rest-api-spec file `cat.master.json` to `cat.cluster_manager.json` and update the paths.
* Add YAML REST test. By default, it will run test for both endpoints `GET _cat/cluster_manager` and `GET _cat/master`
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Add a new setting `cluster.no_cluster_manager_block`, aims to replace the existing `cluster.no_master_block`
* Set the value of the old setting `cluster.no_master_block` as the fallback to the new setting
* Deprecate the existing `setting cluster.no_master_block`
* Add unit tests
Signed-off-by: Tianli Feng <ftianli@amazon.com>
Removes the type variable as input to CreateIndexRequest.mapping(Object...)
along with the CreateIndexRequestBuilder helper class. This also refactors the
method name to setMapping for consistency with other methods (e.g.,
setSettings).
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Continues removal of types from PutIndexTemplateRequest and
PutIndexTemplateRequestBuilder.mapping. Delegated mapping method in
PutIndexTemplateRequestBuilder is refactored to setMapping for consistency with
similar methods (e.g., setSettings, setAliases).
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
First pass to remove types from CreateIndexRequest and CreateIndexRequestBuilder
mapping method. This method is overloaded several times so the most widely used
methods in the RequestBuilder are refactored from mapping to setMapping to avoid
confusion, conflicts, and to be consistent with other method names (e.g.,
setSettings, setCause, setAlias).
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* [Upgrade] Lucene 9.1.0-snapshot-ea989fe8f30
Upgrades from Lucene 9.0.0 to 9.1.0-snapshot-ea989fe8f30 in preparation for
9.1.0 GA.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* Add spanishplural token filter
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* fix KNOWN_TOKENIZERS
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Unmutes IndexPrimaryRelocationIT.testPrimaryRelocationWhileIndexing which was
fixed by LuceneChangesSnapshot using accurate ops history.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Adds default implementation for getEngineFactory in EnginePlugin. The
default just returns Optional.empty(), allowing plugin developers to
implement this plugin without implementing this method.
Signed-off-by: John Mazanec <jmazane@amazon.com>
This commit upgrades the core codebase from Lucene 8.10.1 to
lucene 9.0.0. It includes all necessary refactoring of features and
API changes when upgrading to a new major Lucene release.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Co-authored-by: Andriy Redko <drreta@gmail.com>
Improves the LuceneChangesSnapshot to get an accurate count of recovery
operations using sort by sequence number optimization.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Removes the obsolete types based .addMapping method from CreateIndexRequest and
CreateIndexRequestBuilder. Tests are refactored to use the new source only
setMapping method.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Removes types from the TaskResults internal index along with the getMappings
method from IndexMetadata. This is needed to further remove types from
CreateIndexRequest.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* [Remove] Type from TermsLookUp
Signed-off-by: Suraj Singh <surajrider@gmail.com>
* Fix unit test failure
Signed-off-by: Suraj Singh <surajrider@gmail.com>
* Discrepancy in result from _validate/query API and actual query validity
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
* Moved the validate() check later into the flow to allow range validation to trigger first
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
Removes types from UID class along with cleaning up all obsolete MapperService
dependendices. This includes removing UID from the Translog Delete operation
which is no longer needed due to type dependency removal.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Removes multi-type support from IndexTemplateMetadata so that instead of holding
a map of multiple types to mappings, it only returns a single mapping for a
single type. Also removes type from documentMapper() method to avoid any
accidental NullPointerExceptions in the internal mapping retrieval.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
This inner class is no longer required because its functionality has been moved to the generic GatedCloseable class.
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
Nick Knize
Tianli Feng
Kartik Ganesh
Daniel Doubrovkine (dB.)
Peng Huo
Tushar Kharbanda
Suraj Singh
Wenjun Ruan
Andriy Redko
Ankit Jain
Vacha Shah
Poojita Raj
Owais Kazi
opensearch-trigger-bot[bot]
Suraj Singh
Mohit Godwani
Sruti Parthiban
Viggo
John Mazanec