Commit Graph

2775 Commits

Author SHA1 Message Date
Andriy Redko 82fb7abb71
Introduce QueryPhaseSearcher extension point (SearchPlugin) (#1931)
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2022-03-21 23:11:23 -05:00
Tianli Feng ea31483126
Add a new node role 'cluster_manager' as the alternative for 'master' role and deprecate 'master' role (#2424)
Add a new node role `cluster_manager`, as the replacement for `master` role, that used in setting `node.roles: [ master ]`. 
They have got the same functionality, but can NOT be assigned to `node.roles` together.
* Add `CLUSTER_MANAGER_ROLE` in `DiscoveryNodeRole` class, and deprecate `MASTER_ROLE`
* Remove `MASTER_ROLE` from the `DiscoveryNodeRole.BUILT_IN_ROLES`, and temporarily load the role by the existing method `DiscoveryNode.setAdditionalRoles`
* Add a method `validateRole()` in `DiscoveryNodeRole` class, it's used to validate a specific role is compatible with the other roles to be assigned to a node together.
* Add deprecation message when assigning `master` role in setting `node.roles`
* Replace most `MASTER_ROLE` with `CLUSTER_MANAGER_ROLE` in current unit and integration tests
* Add new unit and integration tests to validate `CLUSTER_MANAGER_ROLE` and `MASTER_ROLE` can be treated as the same.

More explanation:
* New node will have "cluster_manager", "data", and "ingest" roles by default. Which means the default value of setting "node.roles" will be `["cluster_manager","data","ingest"]`, instead of `["master","data","ingest"]`
* "cluster_manager" role will be treated as "master" role in the OpenSearch node of previous versions.
* "cluster_manager” role and "master" role have got the same abbreviation name: "m"

Signed-off-by: Tianli Feng <ftianli@amazon.com>
2022-03-18 15:06:06 -07:00
Tianli Feng 1193ec101b
Deprecate setting 'cluster.no_master_block' and introduce the alternative setting 'cluster.no_cluster_manager_block' (#2453)
* Add a new setting `cluster.no_cluster_manager_block`, aims to replace the existing `cluster.no_master_block`
* Set the value of the old setting `cluster.no_master_block` as the fallback to the new setting
* Deprecate the existing `setting cluster.no_master_block`
* Add unit tests

Signed-off-by: Tianli Feng <ftianli@amazon.com>
2022-03-18 14:57:08 -07:00
Tianli Feng 19eadb46ba
Deprecate setting 'cluster.initial_master_nodes' and introduce the alternative setting 'cluster.initial_cluster_manager_nodes' (#2463)
* Deprecate setting cluster.initial_master_nodes, and add setting cluster.initial_cluster_manager_nodes

Signed-off-by: Tianli Feng <ftianli@amazon.com>
2022-03-18 14:55:49 -07:00
Nick Knize b9f04405d6
[Remove] types from CreateIndexRequest and companion Builder's mapping method (#2498)
Removes the type variable as input to CreateIndexRequest.mapping(Object...)
along with the CreateIndexRequestBuilder helper class. This also refactors the
method name to setMapping for consistency with other methods (e.g.,
setSettings).

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-03-17 18:13:27 -05:00
Nick Knize 2e3d3fef13
[Remove] Type from PutIndexTemplateRequest and PITRB (#2497)
Continues removal of types from PutIndexTemplateRequest and
PutIndexTemplateRequestBuilder.mapping. Delegated mapping method in
PutIndexTemplateRequestBuilder is refactored to setMapping for consistency with
similar methods (e.g., setSettings, setAliases).

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-03-17 12:05:24 -05:00
Nick Knize 3675400b92
[Remove] type from CIR.mapping and CIRB.mapping (#2478)
First pass to remove types from CreateIndexRequest and CreateIndexRequestBuilder
mapping method. This method is overloaded several times so the most widely used
methods in the RequestBuilder are refactored from mapping to setMapping to avoid
confusion, conflicts, and to be consistent with other method names (e.g.,
setSettings, setCause, setAlias).

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-03-16 17:35:08 -05:00
Nick Knize 05a5819243
[Upgrade] Lucene 9.1.0-snapshot-ea989fe8f30 (#2487)
* [Upgrade] Lucene 9.1.0-snapshot-ea989fe8f30

Upgrades from Lucene 9.0.0 to 9.1.0-snapshot-ea989fe8f30 in preparation for
9.1.0 GA.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* Add spanishplural token filter

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* fix KNOWN_TOKENIZERS

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-03-16 15:47:25 -04:00
Lukáš Vlček 02ffd4c548
Rename reference to project OpenSearch was forked from (#2483)
Fixing log message and renaming internal variable.
Improving internal JavaDoc.

Signed-off-by: Lukáš Vlček <lukas.vlcek@aiven.io>
2022-03-16 12:03:22 -05:00
Nick Knize 006c832c5f
[Upgrade] Lucene 9.0.0 release (#1109)
This commit upgrades the core codebase from Lucene 8.10.1 to
lucene 9.0.0. It includes all necessary refactoring of features and
API changes when upgrading to a new major Lucene release.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Co-authored-by: Andriy Redko <drreta@gmail.com>
2022-03-15 15:48:13 -05:00
Nick Knize 757abdb9a0
[Refactor] LuceneChangesSnapshot to use accurate ops history (#2452)
Improves the LuceneChangesSnapshot to get an accurate count of recovery
operations using sort by sequence number optimization.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-03-15 13:13:54 -05:00
Nick Knize bdcaec5caf
[Remove] types from Uid and remaining types/Uid from translog (#2450)
Removes types from UID class along with cleaning up all obsolete MapperService
dependendices. This includes removing UID from the Translog Delete operation
which is no longer needed due to type dependency removal.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-03-13 12:49:51 -05:00
Nick Knize 95d4750249
[Remove] types from translog (#2439)
Removes persisting the type in the translog since types are no longer supported.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-03-12 14:03:00 -06:00
Kartik 9cfa395128
Remove the IndexCommitRef class (#2421)
This inner class is no longer required because its functionality has been moved to the generic GatedCloseable class.

Signed-off-by: Kartik Ganesh <gkart@amazon.com>
2022-03-10 10:12:17 -08:00
Kartik fb9e150076
Refactoring gated and ref-counted interfaces and their implementations (#2396)
* Reducing duplication in plugins around ref-counted releasable classes

Both AmazonEc2Reference and AmazonS3Reference duplicate the same logic - a subclass of AbstractRefCounted that also implements Releasable. This change centralizes this paradigm into a AbstractRefCountedReleasable class and supports both clients via generics. It also updates all fetching implementations to use the get() method instead of client()

Signed-off-by: Kartik Ganesh <gkart@amazon.com>

* Introduce Reference classes for the Closeable and AutoCloseable interfaces

These classes allow you to wrap a reference instance with an onClose runnable that is executed when close() is invoked. Two separate classes are needed because the close() signatures for the two interfaces are different. This change takes the first step to have implementing classes extend from these generic superclasses, before attempting to remove the subclasses entirely. The get() method is also replaced throughout the code base.

Note that there is also a separate Releasable interface that has a similar access pattern, but is implemented separately. This is used in AbstractRefCountedReleasable introduced in a prior commit

Signed-off-by: Kartik Ganesh <gkart@amazon.com>

* More improvements and refactoring

* Functionality around one-way gating is now moved to a dedicated class - OneWayGate. This replaces duplicate functionality throughout the code.
* The two *Reference classes have been renamed to Gated* since that better represents their functionality
* The AbstractRefCountedReleasable has been improved to no longer be abstract by accepting the shutdown hook. This removes the need for the inner class in ReleasableBytesReference, and further simplifies the plugin subclasses (these could probably be removed entirely).
* Finally, unit tests have been added for some classes

Signed-off-by: Kartik Ganesh <gkart@amazon.com>

* Added tests for GatedCloseable

Also updated the license information in GatedAutoCloseableTests

Signed-off-by: Kartik Ganesh <gkart@amazon.com>

* Fixing license information in new files

Signed-off-by: Kartik Ganesh <gkart@amazon.com>

* Added unit tests for RefCountedReleasable

Signed-off-by: Kartik Ganesh <gkart@amazon.com>
2022-03-09 12:21:09 -08:00
Nick Knize 5a9a11416d
[Remove] TrimUnsafeCommit logic for legacy 6.x indexes (#2225)
* [Remove] TrimUnsafeCommit logic for legacy 6.x indexes

Multiple txlog commits was introduced in legacy 7.x. Legacy 6.x indexes could
therefore not have a safe commit. Since OpenSearch 2.0 is no longer compatible
with legacy 6.x indexes, the logic to trim these unsafe commits is safely
removed.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* fix assertion typo

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* rebase and incorporate pr feedback

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-03-09 10:50:05 -08:00
Nick Knize c3712a51b3
[Remove] include_type_name from HLRC (#2397)
Removes include_type_name from the high level reset client along with relevant
deprecated methods in IndicesClient. All tests are updated to remove the
parameter from the rest requests along with various toXContent methods that are
no longer required.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-03-08 11:30:27 -06:00
aponb be64af2f90
Replace exclusionary words whitelist and blacklist in the places that won't impact backwards compatibility (#2178)
* Replace the exclusionary word whitelist with allowlist, and blacklist with denylist, in code commet and internal variable/method/class/package name.

Signed-off-by: Andreas <apre@gmx.at>
2022-03-04 16:34:09 -08:00
Nick Knize 0cc2c9be6a
[Remove] types from PutMappingRequest (#2335)
Remove type support from putMappingRequest, dependencies, and all tests.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-03-04 13:30:43 -06:00
Nick Knize 897f4e7295
[Remove] deprecated getMapping API from IndicesClient (#2262)
Removes the deprecated types based get, getMapping, getAsync, and
getMappingAsync methods from IndicesClient. It also removes extra nesting of
mappings belong the deprecated type named object and removes the types based
methods from the affected request classes.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-03-02 13:44:04 -06:00
Nick Knize 0bd7850bed
[Remove] remaining type usage in Client and AbstractClient (#2258)
Removes type parameter from remaining prepareIndex in Client and AbstractClient.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-02-25 13:35:48 -06:00
Nick Knize 8b48207540
[Remove] Type from Client.prepare(Index,Delete,Update) (#2253)
Removes the type parameter from Client.prepare(Index,Delete,Update) and
everywhere it's used throughout the codebase except for prepareIndex(index,
type, id) which is removed in a follow up.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-02-24 21:20:03 -06:00
Nick Knize d892c51d66
[Remove] Types from DocWrite Request and Response (#2239)
Removes type support from DocWrite Request and Response, all derived classes,
and all places used.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-02-24 15:57:44 -06:00
Nick Knize 3445befd36
[Remove] types from SearchHit and Explain API (#2205)
* [Remove] types from SearchHit and Explain API

Removes type support from SearchHit and Explain API.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* fix failing tests

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* move TYPES_DEPRECATION_MESSAGE from RestExplainAction to FullClusterRestartIT

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* fix FullClusterRestartIT

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* fix failing tests

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* fix testSoftDelete

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-02-23 11:40:30 -08:00
Nick Knize 44441d8fc5
Always use Lucene index in peer recovery (#2077)
With soft deletes no longer optional, peer recovery is switched to always use the
lucene index instead of replaying operations from the translog.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-02-23 11:05:40 -06:00
Nick Knize 2e3f410957
[Remove] Types from GET/MGET (#2168)
Removes type support from the get and mget transport and rest actions along
with removal from ShardGetService.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-02-21 11:37:27 -06:00
Nick Knize 7fe642fda5
[Remove] Type from Search Internals (#2109)
With types deprecation the type support is removed from internal search API
(SearchRequest and QueryShardContext).

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-02-17 16:17:31 -06:00
dependabot[bot] 9689a27b63
Bump com.diffplug.spotless from 5.6.1 to 6.2.0 (#1919)
* Bump com.diffplug.spotless from 5.6.1 to 6.2.0

Bumps com.diffplug.spotless from 5.6.1 to 6.2.0.

---
updated-dependencies:
- dependency-name: com.diffplug.spotless
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* spotlessApply

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Nicholas Walter Knize <nknize@apache.org>
2022-02-16 22:08:40 -08:00
Suraj Singh 892801a074
Remove type mapping from document index API (#2026)
* Remove doc type specific indexing APIs and relevant changes

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Remove type param from yml and integ test files

Signed-off-by: Suraj Singh <surajrider@gmail.com>
2022-02-09 13:34:10 -06:00
Nick Knize 6428d1a2a6
[Refactor] MapperService to QueryShardContext in valueFetcher (#2027)
* [Refactor] MapperService to QueryShardContext in valueFetcher

QueryShardContext encapsulates MapperService along with other mechanisms useful
for analyzers, highlighters, and readers. This commit refactors the valueFetcher
to use the QueryShardContext as a parameter instead of direct access to the
MapperService. This is an API change. Behavior of the method is unaffected.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* change unnecessary Optional usage

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* spotlessApply

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-02-08 08:38:20 -05:00
Nick Knize 8288c9010c
[Remove] CircuitBreaker Accounting (#2056)
RAM Accounting of segments is removed so remove the associated Accounting
circuit breaker.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-02-04 19:46:27 -06:00
Nick Knize fc0d3a368d
[Remove] Segment memory estimation and tracking (#2029)
Lucene 9 removed CodecReader#ramBytesUsed and all file formats that no longer
consume large amounts of memory. As a result RAM estimation for segments is no
longer possible and is removed by this commit. backwards compatibility is retained 
for the tranport layer.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-02-04 13:47:48 -06:00
Andriy Redko 1e5d98329e
Another attempt to fix o.o.transport.netty4.OpenSearchLoggingHandlerIT fails w/ stack overflow (#2051)
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2022-02-04 11:23:36 -05:00
Nick Knize 0791c88af1
[Remove] Deprecated Synced Flush API (#1761)
Remove the deprecated sync flush API which was replaced by sequence
number and retention lease mechanisms and no longer used in 2.0.0.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-01-26 22:18:48 -06:00
Rishikesh Pasham 3f92aecddf
Enabling Sort Optimization to make use of Lucene (#1974)
Remove sort optimization. From Lucence 8.10.1, Numeric sort optimization is 
already done in Lucene Comparators. Lucene also supports sort optimization 
with search_after queries.

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>
2022-01-26 14:36:07 -06:00
Nick Knize 57ac788bb6
[Refactor] InternalEngine to always use soft deletes (#1933)
Soft Deletes have been enabled by default since Legacy version 7.0 and made
mandatory in Version 2.0.0. This commit refactors the InternalEngine to always
use soft-deletes. It is a follow on to making soft deletes mandatory in 2.0.0.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-01-26 12:28:24 -06:00
Nick Knize 7058256e45
Enforce soft deletes (#1903)
This commit enforces soft deletes as required in 2.0.0.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-01-18 12:54:42 -06:00
Nick Knize 81d998dd7a
[BWC] Ensure 2.x compatibility with Legacy 7.10.x (#1902)
* [BWC] Ensure 2.x compatibility with Legacy 7.10.x

This commit fixes TransportHandshaker to send a spoofed Legacy 7.10.2 mincompat
version to ensure OpenSearch 2.x nodes can join a Legacy 7.10.x cluster for
rolling upgrade support. Without this change 7.10.x and OpenSearch 2.x mixed
cluster bwc tests would fail.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* fix AbstractSimpleTransportTestCase failures

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* change v3 to local static

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* make V_3_0_0 package private and reuse in test

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-01-17 12:26:28 -05:00
Andriy Redko 385b268bc0
Update Mockito to 4.2.x (#1830)
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2022-01-03 12:00:45 -05:00
Rabi Panda 7c73ed8d4c
Remove deprecated transport client (#1781)
This commit removes all usage of transport client in OpenSearch.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-12-28 11:28:51 -06:00
Rabi Panda 24d0c9b818
Remove transport client from tests. (#1809)
Transport client has been deprecated. In this commit, we are removing all its usages from the tests.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-12-27 17:43:18 -06:00
Andrew Ross 96d55966a2
Use try-with-resources with MockLogAppender (#1595)
I previously added a helper that started a MockLogAppender to ensure it
was never added to a Logger before it was started. I subsequently found
the opposite case in RolloverIT.java where the appender was stopped
before it was closed, therefore creating a race where a concurrently
running test in the same JVM could cause a logging failure. This seems
like a really easy mistake to make when writing a test or introduce when
refactoring a test. I've made a change to use try-with-resources to
ensure that proper setup and teardown is done. This should make it much
harder to introduce this particular test bug in the future.
Unfortunately, it did involve touching a lot of files. The changes here
are purely structural to leverage try-with-resources; no testing logic
has been changed.

Signed-off-by: Andrew Ross <andrross@amazon.com>
2021-12-21 15:00:55 -06:00
Nick Knize 3f210dcfce
Remove LegacyESVersion.V_6_3_x constants (#1691)
This commit removes LegacyESVersion.V_6_3_x constants including all
pre-release versions and bug fixes.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-12-17 16:24:50 -06:00
Vacha e66ea2c4f3
Avoid logging duplicate deprecation warnings multiple times (#1660)
* Avoid logging duplicate deprecation warnings multiple times

Signed-off-by: Vacha <vachshah@amazon.com>

* Fixes test failures

Signed-off-by: Vacha <vachshah@amazon.com>

* Adding deprecation logger tests

Signed-off-by: Vacha <vachshah@amazon.com>

* Using ConcurrentHashMap keySet

Signed-off-by: Vacha Shah <vachshah@amazon.com>
2021-12-15 15:26:44 -08:00
Nick Knize 538b40a0dc
Remove LegacyESVersion.V_6_2_x constants (#1686)
This commit removes LegacyESVersion.V_6_2_x constants including all
pre-release versions and bug fixes.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-12-09 16:08:00 -06:00
Nick Knize 821417b9ab
Remove LegacyESVersion.V_6_1_x constants (#1681)
This commit removes LegacyESVersion.V_6_1_x constants including all
pre-release versions and bug fixes.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-12-08 23:11:36 -06:00
Nick Knize b74d71fb74
Remove 6.0.* version constants (#1658)
This PR removes LegacyESVersion.V_6_0_* constants including all pre-release
versions and bug fixes.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-12-08 13:37:21 -06:00
Nick Knize 33d8677796
Refactor LegacyESVersion tests from Version tests (#1662)
In preparation for removing all LegacyESVersion support by 3.0; this commit
largely refactors the LegacyESVersion test logic from the OpenSearch Version
test logic into an independent test class. This PR also updates Version.fromString
to ensure a proper legacy version is returned when major is > 3 (to support
legacy yaml test and build scripts).

Note that bwc w/ legacy versions are still supported so some cross compatibility
testing is retained in the Version test class.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-12-07 08:27:17 -06:00
Nick Knize 4db97aa470
[Remove] Deprecated Zen1 Discovery (#1216)
Zen1 discovery was deprecated in Legacy 7.x for eventual removal. OpenSearch 1.x
carries this deprecation. This commit completely removes all support for Zen1
discovery in favor of Zen2.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-12-03 15:27:10 -06:00
Andrew Ross d467283475
Start MockLogAppender before adding to static context (#1587)
I observed a test failure with the message
'Attempted to append to non-started appender mock' from an assertion in
`OpenSearchTestCase::after`. I believe this indicates that a
MockLogAppender (which is named "mock") was added as an appender to the
static logging context and some other test in the same JVM happened to
cause a logging statement to hit that appender and cause an error, which
then caused an unrelated test to fail (because they share static state
with the logger). Almost all usages of MockLogAppender start it
immediately after creation. I found a few that did not and fixed those.
I also made a static helper in MockLogAppender to start it upon
creation.

Signed-off-by: Andrew Ross <andrross@amazon.com>
2021-11-19 13:37:46 -05:00