Commit Graph

2894 Commits

Author SHA1 Message Date
dependabot[bot] 81e2455df1
Bump asm-analysis from 9.2 to 9.3 in /test/logger-usage (#2829)
Bumps asm-analysis from 9.2 to 9.3.

---
updated-dependencies:
- dependency-name: org.ow2.asm:asm-analysis
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-04-11 11:47:50 -04:00
Yoann Rodière b5d5616d44
Update commons-logging to 1.2 (#2806)
* Upgrade to Apache Commons Logging 1.2

Signed-off-by: Yoann Rodière <yoann@hibernate.org>

* Clarify that Apache HTTP/commons-* dependencies are not just for tests

Signed-off-by: Yoann Rodière <yoann@hibernate.org>
2022-04-08 16:43:51 -04:00
Tianli Feng 47a22bb08d
Replace remaining 'blacklist' with 'denylist' in internal class and method names (#2784)
* Replace blacklist with denylist in BlacklistedPathPatternMatcher

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* Replace blacklist with denylist in assumption message

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* Replace all Blacklisted with Denylisted

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* Replace all blacklist(key) with denylist(key)

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* Adjust format by spotlessApply task

Signed-off-by: Tianli Feng <ftianli@amazon.com>
2022-04-07 15:45:20 -07:00
Andriy Redko 0244b2a2df
Update azure-storage-blob to 12.15.0: fix test flakiness (#2795)
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2022-04-06 16:28:14 -05:00
Tianli Feng ed040e9f1a
Replace blacklist in Gradle build environment configuration (#2752)
- Replace `blacklist` with `denylist` in all `tests.rest.blacklist` and `REST_TESTS_BLACKLIST`
Signed-off-by: Tianli Feng <ftianli@amazon.com>
2022-04-05 17:48:39 -07:00
Andriy Redko 1dbeda0f32
Update azure-storage-blob to 12.15.0 (#2774)
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2022-04-05 18:49:42 -04:00
Wenjun Ruan 0b1f4a2069
Rename `file` to `dir` in Environment (#2730)
* Rename `file` to `dir` in Environment

Signed-off-by: ruanwenjun <wenjun@apache.org>

* Fix compile error

Signed-off-by: ruanwenjun <wenjun@apache.org>

* fix compile error

Signed-off-by: ruanwenjun <wenjun@apache.org>
2022-04-05 09:23:08 -04:00
Nick Knize cb238aae61
Bump version to 3.0.0 (#2556)
Bump version to 3.0.0

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-04-04 17:40:31 -05:00
Tianli Feng 6a2a33d187
Make Rest-High-Rest-Level tests allow deprecation warning temporarily, during deprecation of request parameter 'master_timeout' (#2702)
Temporarily build rest client with setStrictDeprecationMode(false) to allow 
deprecation warning in HLRC tests while master_timeout parameters is
being refactored.

Signed-off-by: Tianli Feng <ftianli@amazon.com>
2022-04-04 10:22:22 -05:00
Nick Knize 6c195abc59
[Main] Bump Version 2.0 to 2.1 (#2715)
Bumps from version 2.0 to 2.1

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-04-02 14:40:25 -05:00
Andriy Redko d8a1ba6912
[CVE-2020-36518] Update jackson-databind to 2.13.2.2 (#2599)
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2022-03-29 12:24:37 -04:00
Vacha Shah 5dd75bb0aa
Removing SLM check in tests for OpenSearch versions (#2604)
Signed-off-by: Vacha Shah <vachshah@amazon.com>
2022-03-28 11:32:18 -05:00
Owais Kazi cc0e66b1dc
Replaced "master" terminology in Log message (#2575)
Changed the log message to cluster-manager from master

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>
2022-03-24 21:50:54 -05:00
Andriy Redko b6ca0d1f78
Concurrent Searching (Experimental) (#1500)
* Concurrent Searching (Experimental)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* Addressingf code review comments

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2022-03-24 14:20:31 -04:00
Andriy Redko 82fb7abb71
Introduce QueryPhaseSearcher extension point (SearchPlugin) (#1931)
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2022-03-21 23:11:23 -05:00
dependabot[bot] 8737b677a0
Bump asm-analysis from 7.1 to 9.2 in /test/logger-usage (#2273)
Bumps asm-analysis from 7.1 to 9.2.

---
updated-dependencies:
- dependency-name: org.ow2.asm:asm-analysis
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-21 15:05:26 -04:00
Tianli Feng ea31483126
Add a new node role 'cluster_manager' as the alternative for 'master' role and deprecate 'master' role (#2424)
Add a new node role `cluster_manager`, as the replacement for `master` role, that used in setting `node.roles: [ master ]`. 
They have got the same functionality, but can NOT be assigned to `node.roles` together.
* Add `CLUSTER_MANAGER_ROLE` in `DiscoveryNodeRole` class, and deprecate `MASTER_ROLE`
* Remove `MASTER_ROLE` from the `DiscoveryNodeRole.BUILT_IN_ROLES`, and temporarily load the role by the existing method `DiscoveryNode.setAdditionalRoles`
* Add a method `validateRole()` in `DiscoveryNodeRole` class, it's used to validate a specific role is compatible with the other roles to be assigned to a node together.
* Add deprecation message when assigning `master` role in setting `node.roles`
* Replace most `MASTER_ROLE` with `CLUSTER_MANAGER_ROLE` in current unit and integration tests
* Add new unit and integration tests to validate `CLUSTER_MANAGER_ROLE` and `MASTER_ROLE` can be treated as the same.

More explanation:
* New node will have "cluster_manager", "data", and "ingest" roles by default. Which means the default value of setting "node.roles" will be `["cluster_manager","data","ingest"]`, instead of `["master","data","ingest"]`
* "cluster_manager" role will be treated as "master" role in the OpenSearch node of previous versions.
* "cluster_manager” role and "master" role have got the same abbreviation name: "m"

Signed-off-by: Tianli Feng <ftianli@amazon.com>
2022-03-18 15:06:06 -07:00
Tianli Feng 1193ec101b
Deprecate setting 'cluster.no_master_block' and introduce the alternative setting 'cluster.no_cluster_manager_block' (#2453)
* Add a new setting `cluster.no_cluster_manager_block`, aims to replace the existing `cluster.no_master_block`
* Set the value of the old setting `cluster.no_master_block` as the fallback to the new setting
* Deprecate the existing `setting cluster.no_master_block`
* Add unit tests

Signed-off-by: Tianli Feng <ftianli@amazon.com>
2022-03-18 14:57:08 -07:00
Tianli Feng 19eadb46ba
Deprecate setting 'cluster.initial_master_nodes' and introduce the alternative setting 'cluster.initial_cluster_manager_nodes' (#2463)
* Deprecate setting cluster.initial_master_nodes, and add setting cluster.initial_cluster_manager_nodes

Signed-off-by: Tianli Feng <ftianli@amazon.com>
2022-03-18 14:55:49 -07:00
Nick Knize b9f04405d6
[Remove] types from CreateIndexRequest and companion Builder's mapping method (#2498)
Removes the type variable as input to CreateIndexRequest.mapping(Object...)
along with the CreateIndexRequestBuilder helper class. This also refactors the
method name to setMapping for consistency with other methods (e.g.,
setSettings).

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-03-17 18:13:27 -05:00
Nick Knize 2e3d3fef13
[Remove] Type from PutIndexTemplateRequest and PITRB (#2497)
Continues removal of types from PutIndexTemplateRequest and
PutIndexTemplateRequestBuilder.mapping. Delegated mapping method in
PutIndexTemplateRequestBuilder is refactored to setMapping for consistency with
similar methods (e.g., setSettings, setAliases).

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-03-17 12:05:24 -05:00
Nick Knize 3675400b92
[Remove] type from CIR.mapping and CIRB.mapping (#2478)
First pass to remove types from CreateIndexRequest and CreateIndexRequestBuilder
mapping method. This method is overloaded several times so the most widely used
methods in the RequestBuilder are refactored from mapping to setMapping to avoid
confusion, conflicts, and to be consistent with other method names (e.g.,
setSettings, setCause, setAlias).

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-03-16 17:35:08 -05:00
Nick Knize 05a5819243
[Upgrade] Lucene 9.1.0-snapshot-ea989fe8f30 (#2487)
* [Upgrade] Lucene 9.1.0-snapshot-ea989fe8f30

Upgrades from Lucene 9.0.0 to 9.1.0-snapshot-ea989fe8f30 in preparation for
9.1.0 GA.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* Add spanishplural token filter

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* fix KNOWN_TOKENIZERS

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-03-16 15:47:25 -04:00
Lukáš Vlček 02ffd4c548
Rename reference to project OpenSearch was forked from (#2483)
Fixing log message and renaming internal variable.
Improving internal JavaDoc.

Signed-off-by: Lukáš Vlček <lukas.vlcek@aiven.io>
2022-03-16 12:03:22 -05:00
Nick Knize 006c832c5f
[Upgrade] Lucene 9.0.0 release (#1109)
This commit upgrades the core codebase from Lucene 8.10.1 to
lucene 9.0.0. It includes all necessary refactoring of features and
API changes when upgrading to a new major Lucene release.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Co-authored-by: Andriy Redko <drreta@gmail.com>
2022-03-15 15:48:13 -05:00
Nick Knize 757abdb9a0
[Refactor] LuceneChangesSnapshot to use accurate ops history (#2452)
Improves the LuceneChangesSnapshot to get an accurate count of recovery
operations using sort by sequence number optimization.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-03-15 13:13:54 -05:00
Nick Knize bdcaec5caf
[Remove] types from Uid and remaining types/Uid from translog (#2450)
Removes types from UID class along with cleaning up all obsolete MapperService
dependendices. This includes removing UID from the Translog Delete operation
which is no longer needed due to type dependency removal.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-03-13 12:49:51 -05:00
Nick Knize 95d4750249
[Remove] types from translog (#2439)
Removes persisting the type in the translog since types are no longer supported.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-03-12 14:03:00 -06:00
Kartik 9cfa395128
Remove the IndexCommitRef class (#2421)
This inner class is no longer required because its functionality has been moved to the generic GatedCloseable class.

Signed-off-by: Kartik Ganesh <gkart@amazon.com>
2022-03-10 10:12:17 -08:00
Kartik fb9e150076
Refactoring gated and ref-counted interfaces and their implementations (#2396)
* Reducing duplication in plugins around ref-counted releasable classes

Both AmazonEc2Reference and AmazonS3Reference duplicate the same logic - a subclass of AbstractRefCounted that also implements Releasable. This change centralizes this paradigm into a AbstractRefCountedReleasable class and supports both clients via generics. It also updates all fetching implementations to use the get() method instead of client()

Signed-off-by: Kartik Ganesh <gkart@amazon.com>

* Introduce Reference classes for the Closeable and AutoCloseable interfaces

These classes allow you to wrap a reference instance with an onClose runnable that is executed when close() is invoked. Two separate classes are needed because the close() signatures for the two interfaces are different. This change takes the first step to have implementing classes extend from these generic superclasses, before attempting to remove the subclasses entirely. The get() method is also replaced throughout the code base.

Note that there is also a separate Releasable interface that has a similar access pattern, but is implemented separately. This is used in AbstractRefCountedReleasable introduced in a prior commit

Signed-off-by: Kartik Ganesh <gkart@amazon.com>

* More improvements and refactoring

* Functionality around one-way gating is now moved to a dedicated class - OneWayGate. This replaces duplicate functionality throughout the code.
* The two *Reference classes have been renamed to Gated* since that better represents their functionality
* The AbstractRefCountedReleasable has been improved to no longer be abstract by accepting the shutdown hook. This removes the need for the inner class in ReleasableBytesReference, and further simplifies the plugin subclasses (these could probably be removed entirely).
* Finally, unit tests have been added for some classes

Signed-off-by: Kartik Ganesh <gkart@amazon.com>

* Added tests for GatedCloseable

Also updated the license information in GatedAutoCloseableTests

Signed-off-by: Kartik Ganesh <gkart@amazon.com>

* Fixing license information in new files

Signed-off-by: Kartik Ganesh <gkart@amazon.com>

* Added unit tests for RefCountedReleasable

Signed-off-by: Kartik Ganesh <gkart@amazon.com>
2022-03-09 12:21:09 -08:00
Nick Knize 5a9a11416d
[Remove] TrimUnsafeCommit logic for legacy 6.x indexes (#2225)
* [Remove] TrimUnsafeCommit logic for legacy 6.x indexes

Multiple txlog commits was introduced in legacy 7.x. Legacy 6.x indexes could
therefore not have a safe commit. Since OpenSearch 2.0 is no longer compatible
with legacy 6.x indexes, the logic to trim these unsafe commits is safely
removed.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* fix assertion typo

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* rebase and incorporate pr feedback

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-03-09 10:50:05 -08:00
Nick Knize c3712a51b3
[Remove] include_type_name from HLRC (#2397)
Removes include_type_name from the high level reset client along with relevant
deprecated methods in IndicesClient. All tests are updated to remove the
parameter from the rest requests along with various toXContent methods that are
no longer required.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-03-08 11:30:27 -06:00
aponb be64af2f90
Replace exclusionary words whitelist and blacklist in the places that won't impact backwards compatibility (#2178)
* Replace the exclusionary word whitelist with allowlist, and blacklist with denylist, in code commet and internal variable/method/class/package name.

Signed-off-by: Andreas <apre@gmx.at>
2022-03-04 16:34:09 -08:00
Nick Knize 0cc2c9be6a
[Remove] types from PutMappingRequest (#2335)
Remove type support from putMappingRequest, dependencies, and all tests.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-03-04 13:30:43 -06:00
Nick Knize 897f4e7295
[Remove] deprecated getMapping API from IndicesClient (#2262)
Removes the deprecated types based get, getMapping, getAsync, and
getMappingAsync methods from IndicesClient. It also removes extra nesting of
mappings belong the deprecated type named object and removes the types based
methods from the affected request classes.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-03-02 13:44:04 -06:00
Nick Knize 0bd7850bed
[Remove] remaining type usage in Client and AbstractClient (#2258)
Removes type parameter from remaining prepareIndex in Client and AbstractClient.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-02-25 13:35:48 -06:00
Nick Knize 8b48207540
[Remove] Type from Client.prepare(Index,Delete,Update) (#2253)
Removes the type parameter from Client.prepare(Index,Delete,Update) and
everywhere it's used throughout the codebase except for prepareIndex(index,
type, id) which is removed in a follow up.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-02-24 21:20:03 -06:00
Nick Knize d892c51d66
[Remove] Types from DocWrite Request and Response (#2239)
Removes type support from DocWrite Request and Response, all derived classes,
and all places used.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-02-24 15:57:44 -06:00
Nick Knize 3445befd36
[Remove] types from SearchHit and Explain API (#2205)
* [Remove] types from SearchHit and Explain API

Removes type support from SearchHit and Explain API.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* fix failing tests

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* move TYPES_DEPRECATION_MESSAGE from RestExplainAction to FullClusterRestartIT

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* fix FullClusterRestartIT

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* fix failing tests

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* fix testSoftDelete

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-02-23 11:40:30 -08:00
Nick Knize 44441d8fc5
Always use Lucene index in peer recovery (#2077)
With soft deletes no longer optional, peer recovery is switched to always use the
lucene index instead of replaying operations from the translog.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-02-23 11:05:40 -06:00
dependabot[bot] c5b0c21921
Bump asm-tree from 7.1 to 9.2 in /test/logger-usage (#2184)
Bumps asm-tree from 7.1 to 9.2.

---
updated-dependencies:
- dependency-name: org.ow2.asm:asm-tree
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-21 11:54:47 -08:00
dependabot[bot] 6d52ac7334
Bump gson from 2.8.9 to 2.9.0 in /test/fixtures/hdfs-fixture (#2190)
Bumps [gson](https://github.com/google/gson) from 2.8.9 to 2.9.0.
- [Release notes](https://github.com/google/gson/releases)
- [Changelog](https://github.com/google/gson/blob/master/CHANGELOG.md)
- [Commits](https://github.com/google/gson/compare/gson-parent-2.8.9...gson-parent-2.9.0)

---
updated-dependencies:
- dependency-name: com.google.code.gson:gson
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-21 11:48:57 -08:00
Nick Knize 2e3f410957
[Remove] Types from GET/MGET (#2168)
Removes type support from the get and mget transport and rest actions along
with removal from ShardGetService.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-02-21 11:37:27 -06:00
dependabot[bot] f703de9394
Bump asm from 7.1 to 9.2 in /test/logger-usage (#2134)
Bumps asm from 7.1 to 9.2.

---
updated-dependencies:
- dependency-name: org.ow2.asm:asm
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-18 10:24:52 -08:00
dependabot[bot] 8d69c77081
Bump json-smart from 2.4.7 to 2.4.8 in /test/fixtures/hdfs-fixture (#2136)
Bumps [json-smart](https://github.com/netplex/json-smart-v2) from 2.4.7 to 2.4.8.
- [Release notes](https://github.com/netplex/json-smart-v2/releases)
- [Commits](https://github.com/netplex/json-smart-v2/commits/2.4.8)

---
updated-dependencies:
- dependency-name: net.minidev:json-smart
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-02-17 14:31:45 -08:00
Nick Knize 7fe642fda5
[Remove] Type from Search Internals (#2109)
With types deprecation the type support is removed from internal search API
(SearchRequest and QueryShardContext).

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-02-17 16:17:31 -06:00
dependabot[bot] 9689a27b63
Bump com.diffplug.spotless from 5.6.1 to 6.2.0 (#1919)
* Bump com.diffplug.spotless from 5.6.1 to 6.2.0

Bumps com.diffplug.spotless from 5.6.1 to 6.2.0.

---
updated-dependencies:
- dependency-name: com.diffplug.spotless
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* spotlessApply

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Nicholas Walter Knize <nknize@apache.org>
2022-02-16 22:08:40 -08:00
Suraj Singh 892801a074
Remove type mapping from document index API (#2026)
* Remove doc type specific indexing APIs and relevant changes

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Remove type param from yml and integ test files

Signed-off-by: Suraj Singh <surajrider@gmail.com>
2022-02-09 13:34:10 -06:00
Nick Knize 6428d1a2a6
[Refactor] MapperService to QueryShardContext in valueFetcher (#2027)
* [Refactor] MapperService to QueryShardContext in valueFetcher

QueryShardContext encapsulates MapperService along with other mechanisms useful
for analyzers, highlighters, and readers. This commit refactors the valueFetcher
to use the QueryShardContext as a parameter instead of direct access to the
MapperService. This is an API change. Behavior of the method is unaffected.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* change unnecessary Optional usage

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* spotlessApply

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-02-08 08:38:20 -05:00
Nick Knize 8288c9010c
[Remove] CircuitBreaker Accounting (#2056)
RAM Accounting of segments is removed so remove the associated Accounting
circuit breaker.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-02-04 19:46:27 -06:00