Commit Graph

6007 Commits

Author SHA1 Message Date
Peng Huo 9ea25c4f2e
Adjust main version after backport to 1.x (#2147)
Signed-off-by: Peng Huo <penghuo@gmail.com>
2022-02-17 12:17:10 -06:00
dependabot[bot] 9689a27b63
Bump com.diffplug.spotless from 5.6.1 to 6.2.0 (#1919)
* Bump com.diffplug.spotless from 5.6.1 to 6.2.0

Bumps com.diffplug.spotless from 5.6.1 to 6.2.0.

---
updated-dependencies:
- dependency-name: com.diffplug.spotless
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* spotlessApply

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Nicholas Walter Knize <nknize@apache.org>
2022-02-16 22:08:40 -08:00
Nick Knize c0c7d7aedb
[Remove] Type parameter from TermVectors API (#2104)
Remove document type support from the TermVectors API.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-02-15 14:49:08 -06:00
Andriy Redko 6b6f03368f
Mapping update for “date_range” field type is not idempotent (#2094)
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2022-02-14 16:56:02 -06:00
Andriy Redko a942b275ef
Fixing org.opensearch.monitor.os.OsProbeTests::testLogWarnCpuMessageOnlyOnes when CGroups are not available (#2101)
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2022-02-14 12:10:16 -05:00
Andrey Pleskach 27ed6fc82c
Fix integration tests failure (#2067)
Fixed integration tests failure on Linux with Kernel 5.16.x

Signed-off-by: Andrey Pleskach <ples@aiven.io>
2022-02-11 09:40:04 -05:00
Suraj Singh 892801a074
Remove type mapping from document index API (#2026)
* Remove doc type specific indexing APIs and relevant changes

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* Remove type param from yml and integ test files

Signed-off-by: Suraj Singh <surajrider@gmail.com>
2022-02-09 13:34:10 -06:00
Ankit Jain f0984eb409
Added timeout to ensureGreen() for testClusterGreenAfterPartialRelocation (#2074)
Signed-off-by: Ankit Jain <jain.ankitk@gmail.com>
2022-02-09 00:55:30 -08:00
Nick Knize 6428d1a2a6
[Refactor] MapperService to QueryShardContext in valueFetcher (#2027)
* [Refactor] MapperService to QueryShardContext in valueFetcher

QueryShardContext encapsulates MapperService along with other mechanisms useful
for analyzers, highlighters, and readers. This commit refactors the valueFetcher
to use the QueryShardContext as a parameter instead of direct access to the
MapperService. This is an API change. Behavior of the method is unaffected.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* change unnecessary Optional usage

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* spotlessApply

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-02-08 08:38:20 -05:00
Ankit Jain 343b82fe24
Stabilizing org.opensearch.cluster.routing.MovePrimaryFirstTests.test… (#2048)
* Stabilizing org.opensearch.cluster.routing.MovePrimaryFirstTests.testClusterGreenAfterPartialRelocation

Signed-off-by: Ankit Jain <jain.ankitk@gmail.com>

* Removing unused import

Signed-off-by: Ankit Jain <jain.ankitk@gmail.com>

* Making code more readable

Signed-off-by: Ankit Jain <jain.ankitk@gmail.com>
2022-02-07 19:10:07 -08:00
Matt Weber b9420d8f70
Add regexp interval source (#1917)
* Add regexp interval source

Add a regexp interval source provider so people can use regular
expressions inside of intervals queries.

Signed-off-by: Matt Weber <matt@mattweber.org>

* Fixes

- register regexp interval in SearchModule
- use fully-qualified name for lucene RegExp
- get rid of unnecessary variable

Signed-off-by: Matt Weber <matt@mattweber.org>
2022-02-07 17:18:43 -06:00
Nick Knize 8288c9010c
[Remove] CircuitBreaker Accounting (#2056)
RAM Accounting of segments is removed so remove the associated Accounting
circuit breaker.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-02-04 19:46:27 -06:00
Nicholas Walter Knize ee9e5ba579
Add Version.V_1_2_5 constant
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-02-04 18:21:51 -06:00
Nick Knize 25955782b9
[Refactor] Lucene DataInput and DataOutput to StreamInput and StreamOutput (#2035)
Lucene 9 changes from BigEndian to LittleEndian in DataInput and DataOutput.
The use of Lucene's Data IO classes are refactored in this commit to use
StreamInput which maintains the same method calls while preserving the byte
order of the data.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-02-04 14:23:18 -06:00
Nick Knize fc0d3a368d
[Remove] Segment memory estimation and tracking (#2029)
Lucene 9 removed CodecReader#ramBytesUsed and all file formats that no longer
consume large amounts of memory. As a result RAM estimation for segments is no
longer possible and is removed by this commit. backwards compatibility is retained 
for the tranport layer.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-02-04 13:47:48 -06:00
Peng Huo 2fc8acd409
Fix composite aggregation failed test cases introduce by missing_order parameter (#1942) (#2005)
Fix composite aggregation failed test cases introduce by missing_order parameter by using MissingOrder to decide null values's order in InternalBucket.

Signed-off-by: Peng Huo <penghuo@gmail.com>
2022-02-01 14:19:10 -06:00
Lukáš Vlček a597d4daac
Fixing invalid Java code example in JavaDoc (#2008)
Including several other minor typos in JavaDoc.

Signed-off-by: Lukáš Vlček <lukas.vlcek@aiven.io>
2022-01-30 16:31:42 -08:00
Nick Knize 5569ce0025
[Remove] index.merge.policy.max_merge_at_once_explicit (#1988)
Remove deprecated max_merge_at_once_explicit parameter since it has been
removed in lucene 9 and deprecated in opensearch 1.3

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-01-27 14:05:13 -06:00
Nick Knize 334f0d9908
[Remove] Setting explicit version on analysis component (#1986)
Lucene 9 removes the ability to define an explicit version on an analysis
component so remove the the version parameter and deprecation warning
prohibiting users from explicitly setting versioned analyzers.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-01-27 14:04:26 -06:00
Owais Kazi 054595b140
Added timeout to ensureGreen() for testClusterGreenAfterPartialRelocation (#1983)
* Added timeout to ensureGreen() for testClusterGreenAfterPartialRelocation

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

* Fixed spotless imports

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>
2022-01-27 09:26:32 -08:00
Nick Knize 0791c88af1
[Remove] Deprecated Synced Flush API (#1761)
Remove the deprecated sync flush API which was replaced by sequence
number and retention lease mechanisms and no longer used in 2.0.0.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-01-26 22:18:48 -06:00
Nick Knize 0f3b72b7bb
[Deprecate] Setting explicit version on analysis component (#1978)
Lucene 9 removes the ability to define an explicit version on an analysis 
component. The version parameter is deprecated at parse time and a warning is 
issued 
to the user through the deprecation logger.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-01-26 20:46:28 -06:00
Nick Knize a18b4e35b8
[Deprecate] index.merge.policy.max_merge_at_once_explicit (#1981)
max_merge_at_once_explicit is removed in lucene 9 so the index setting is
deprecated for removal in the next major release.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-01-26 20:33:52 -06:00
Matt Weber 0bc30b4314
Wildcard max_expansion version check update (#1980)
Change from 2x to 1.3 for wildcard interval max_expansion version check
since #1916 was backported.

Signed-off-by: Matt Weber <matt@mattweber.org>
2022-01-26 18:43:20 -06:00
Peng Huo 7a9314a3bb
Support _first and _last parameter for missing bucket ordering in composite aggregation (#1942)
Adds support for _first and _last parameters for missing bucket ordering in composite aggregation. 
By default, if order is asc, missing_bucket at first, if order is desc, missing_bucket at last. If 
missing_order is "_first" or "_last", regardless order, missing_bucket is at first or last respectively.

Signed-off-by: Peng Huo <penghuo@gmail.com>
2022-01-26 16:36:50 -06:00
Saurabh Singh 447b20c457
Delay the request size calculation until required by the indexing pressure framework (#1592)
Logically delay the request size calculation until it's absolutely required by the 
indexing pressure framework.

Signed-off-by: Saurabh Singh <sisurab@amazon.com>
2022-01-26 15:06:01 -06:00
Rishikesh Pasham 3f92aecddf
Enabling Sort Optimization to make use of Lucene (#1974)
Remove sort optimization. From Lucence 8.10.1, Numeric sort optimization is 
already done in Lucene Comparators. Lucene also supports sort optimization 
with search_after queries.

Signed-off-by: Rishikesh1159 <rishireddy1159@gmail.com>
2022-01-26 14:36:07 -06:00
Matt Weber 3b8db91abe
Add max_expansions option to wildcard interval (#1916)
Add support for setting the max expansions on a wildcard interval.
The default value is still 128 and the max value is bounded by
`BooleanQuery.getMaxClauseCount()`.

Signed-off-by: Matt Weber <matt@mattweber.org>
2022-01-26 13:06:39 -06:00
Nick Knize 57ac788bb6
[Refactor] InternalEngine to always use soft deletes (#1933)
Soft Deletes have been enabled by default since Legacy version 7.0 and made
mandatory in Version 2.0.0. This commit refactors the InternalEngine to always
use soft-deletes. It is a follow on to making soft deletes mandatory in 2.0.0.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-01-26 12:28:24 -06:00
Venkata Jyothsna Donapati 02c6a7f156
Prefer adaptive replica selection over awareness attribute based routing (#1107)
* Prefer adaptive replica selection over awareness attribute based routing

Signed-off-by: jyothsna <donapv@amazon.com>

* Minor changes

Signed-off-by: jyothsna <donapv@amazon.com>

* Address review comments

Signed-off-by: jyothsna <donapv@amazon.com>

* Fix checkstyle

Signed-off-by: jyothsna <donapv@amazon.com>

* Minor fix

Signed-off-by: jyothsna <donapv@amazon.com>
2022-01-21 11:11:49 -06:00
Sarat Vemulapalli 781156471a
Removing lingering transportclient (#1955)
Signed-off-by: Sarat Vemulapalli <vemulapallisarat@gmail.com>
2022-01-21 01:00:50 -08:00
Suraj Singh 27e05cc1dc
[BUG] Wait for outstanding requests to complete in LastSuccessfulSett… (#1939)
* [BUG] Wait for outstanding requests to complete in LastSuccessfulSettingsUpdate test

Signed-off-by: Suraj Singh <surajrider@gmail.com>

* [BUG] Wait for outstanding requests to complete in LastSuccessfulSettingsUpdate test

Signed-off-by: Suraj Singh <surajrider@gmail.com>
2022-01-20 17:26:53 -06:00
Ankit Jain 6eb8f6f307
Prioritize primary shard movement during shard allocation (#1445)
When some node or set of nodes is excluded (based on some cluster setting) 
BalancedShardsAllocator iterates over them in breadth first order picking 1 shard from 
each node and repeating the process until all shards are balanced. Since shards from 
each node are picked randomly it's possible the p and r of shard1 is relocated first 
leaving behind both p and r of shard2. If the excluded nodes were to go down the 
cluster becomes red. 

This commit introduces a new setting  "cluster.routing.allocation.move.primary_first" 
that prioritizes the p of both shard1 and shard2 first so the cluster does not become 
red if the excluded nodes were to go down before relocating other shards. Note that
with this setting enabled performance of this change is a direct function of number 
of indices, shards, replicas, and nodes. The larger the indices, replicas, and 
distribution scale, the slower the allocation becomes. This should be used with care.

Signed-off-by: Ankit Jain <jain.ankitk@gmail.com>
2022-01-20 17:02:54 -06:00
Nick Knize 7058256e45
Enforce soft deletes (#1903)
This commit enforces soft deletes as required in 2.0.0.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-01-18 12:54:42 -06:00
Suraj Singh 7691f075cb
[Bug] Wait for outstanding requests to complete (#1925)
Signed-off-by: Suraj Singh <surajrider@gmail.com>
2022-01-18 10:49:47 -05:00
Nick Knize 81d998dd7a
[BWC] Ensure 2.x compatibility with Legacy 7.10.x (#1902)
* [BWC] Ensure 2.x compatibility with Legacy 7.10.x

This commit fixes TransportHandshaker to send a spoofed Legacy 7.10.2 mincompat
version to ensure OpenSearch 2.x nodes can join a Legacy 7.10.x cluster for
rolling upgrade support. Without this change 7.10.x and OpenSearch 2.x mixed
cluster bwc tests would fail.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* fix AbstractSimpleTransportTestCase failures

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* change v3 to local static

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* make V_3_0_0 package private and reuse in test

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-01-17 12:26:28 -05:00
Andriy Redko dd8ad29770
Fixing org.opensearch.common.network.InetAddressesTests.testForStringIPv6WithScopeIdInput (#1913)
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2022-01-14 15:23:33 -06:00
Matt Weber 524448fc41
Expand SearchPlugin javadocs. (#1909)
Add and clarify some search plugin point documentation.

Signed-off-by: Matt Weber <matt@mattweber.org>
2022-01-14 13:06:32 -06:00
Andriy Redko f059738aac
[BUG] Serialization bugs can cause node drops (#1885)
This commit restructures InboundHandler to ensure all data 
is consumed over the wire.

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2022-01-14 13:02:34 -06:00
Matt Weber e7d44c20e9
Make SortBuilders pluggable (#1856)
Add the ability for plugin authors to add custom sort builders.

Signed-off-by: Matt Weber <matt@mattweber.org>
2022-01-14 11:06:13 -06:00
Nick Knize 0887ac6d92
[Remove] Old Translog Checkpoint Format (#1884)
This commit removes support for old translog checkpoint formats prior to 7.0.0.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2022-01-11 12:58:35 -06:00
Rabi Panda 40442aae7f
Remove unwanted unreleased versions (#1877)
Only last major (2.0.0), last minor(1.3.0) and (1.2.4) can be unreleased

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2022-01-11 09:19:24 -05:00
Rabi Panda 28f688fbc5
Use Version.compareMajor instead of using equals operator (#1876)
This test fails due to equality check of majors between OpenSearch Version 1.x and LegacyESVersion 7.x. They should be equivalent but in this case the check fails and LegacyVersion.v6.8.x is chosen which was removed in 52508d5.

This change fixes the comparison logic by using `Version.compareMajor` method instead.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2022-01-10 17:59:08 -08:00
Andriy Redko 4dbf1d268c
Concurrent Searching (Experimental): modify profiling implementation to support concurrent data collection (#1673)
* Concurrent Searching (Experimental): modify profiling implementation to support concurrent data collection

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* Addressing code review comments

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2022-01-10 17:11:48 -06:00
Rabi Panda ec24ee56e8
Remove deprecated search.remote settings (#1870)
This commit removes the deprecated search.remote.* settings.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2022-01-10 11:45:56 -06:00
Rabi Panda 52508d57f5
Remove LegacyESVersion.V_6_8_x constants (#1869)
This commit removes all usages of the `LegacyESVersion.V_6_8_x` constants from the codebase.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2022-01-10 11:43:15 -06:00
Andriy Redko c71ff87303
Fixing build failures after Flavor Serialization backport (#1867)
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2022-01-07 16:43:56 -08:00
sharp-pixel 1bb8864178
Add darwin-arm64-tar and no-jdk-darwin-arm64-tar archive distributions. (#1668)
* Add darwin-arm64-tar and no-jdk-darwin-arm64-tar archive distributions.

Signed-off-by: Cédric Pelvet <cedric.pelvet@gmail.com>

* Fix dangling HTML tags in javadoc to solve build issues with JDK 17.

Signed-off-by: Cédric Pelvet <cedric.pelvet@gmail.com>
2022-01-06 12:39:13 -05:00
Andriy Redko 385b268bc0
Update Mockito to 4.2.x (#1830)
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2022-01-03 12:00:45 -05:00
Rabi Panda 06cbc47136
Remove LegacyESVersion.V_6_7_x constants (#1807)
This commit removes all usages of the `LegacyESVersion.V_6_7_x` constants from the codebase.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2022-01-02 14:09:55 -08:00
Rabi Panda 5ccb22fb4b
Remove LegacyESVersion.V_6_6_x constants (#1804)
This commit removes all usages of the `LegacyESVersion.V_6_6_x` constants from the codebase.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-12-29 22:43:53 -08:00
Tianli Feng c05e0d90bd
[TEST] Fix FsHealthServiceTest by increasing the timeout period before checking the FS health after restoring the FS status (#1813)
Signed-off-by: Tianli Feng <ftl94@live.com>
2021-12-28 20:50:59 -08:00
Andriy Redko 65804d25a6
Update to log4j 2.17.1 (#1820)
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2021-12-28 17:06:42 -05:00
Rabi Panda 01e1b94e47
Remove LegacyESVersion.V_6_5_x constants (#1794)
* Remove v6.5.x constants

Signed-off-by: Rabi Panda <adnapibar@gmail.com>

* Address code review comments.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>

* Add back version assumption for testRemoteClusterSettingsUpgraded

Signed-off-by: Rabi Panda <adnapibar@gmail.com>

* Remove unused import after rebase.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-12-28 11:52:51 -06:00
Rabi Panda 7c73ed8d4c
Remove deprecated transport client (#1781)
This commit removes all usage of transport client in OpenSearch.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-12-28 11:28:51 -06:00
Rabi Panda 24d0c9b818
Remove transport client from tests. (#1809)
Transport client has been deprecated. In this commit, we are removing all its usages from the tests.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-12-27 17:43:18 -06:00
Suraj Singh b00b45b25d
[Tests] ClusterHealthIT:testHealthOnMasterFailover - Increase master node timeout (#1812)
Signed-off-by: Suraj Singh <surajrider@gmail.com>
2021-12-27 13:26:18 -06:00
Rabi Panda 8b076b81ee
Remove LegacyVersion.v6.4.x constants (#1787)
This commit removes the `LegacyVersion.v6.4.x` constants and their usages, since we no longer need to support those in OpenSearch 2.0 onwards.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-12-23 16:54:37 -08:00
Rabi Panda 186959a6f5
Add bwc version 1.2.4 (#1796)
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-12-23 10:05:30 -05:00
Andrew Ross 96d55966a2
Use try-with-resources with MockLogAppender (#1595)
I previously added a helper that started a MockLogAppender to ensure it
was never added to a Logger before it was started. I subsequently found
the opposite case in RolloverIT.java where the appender was stopped
before it was closed, therefore creating a race where a concurrently
running test in the same JVM could cause a logging failure. This seems
like a really easy mistake to make when writing a test or introduce when
refactoring a test. I've made a change to use try-with-resources to
ensure that proper setup and teardown is done. This should make it much
harder to introduce this particular test bug in the future.
Unfortunately, it did involve touching a lot of files. The changes here
are purely structural to leverage try-with-resources; no testing logic
has been changed.

Signed-off-by: Andrew Ross <andrross@amazon.com>
2021-12-21 15:00:55 -06:00
Andriy Redko ca27c8fd4f
Update to log4j 2.17.0 (#1771) 2021-12-18 09:36:59 -08:00
Ankit Jain 3ed3fca4ae
Fixing allocation filters to persist existing state on settings update (#1718)
* Fixing allocation filters to persist existing state on settings update

Signed-off-by: Ankit Jain <jain.ankitk@gmail.com>

* Adding test for filter settings update

Signed-off-by: Ankit Jain <jain.ankitk@gmail.com>

* Adding more tests and review comments

Signed-off-by: Ankit Jain <jain.ankitk@gmail.com>

* Adding assertion and unit test for operation type mismatch

Signed-off-by: Ankit Jain <jain.ankitk@gmail.com>

* Updating test names

Signed-off-by: Ankit Jain <jain.ankitk@gmail.com>
2021-12-18 09:06:05 -05:00
Nick Knize 3f210dcfce
Remove LegacyESVersion.V_6_3_x constants (#1691)
This commit removes LegacyESVersion.V_6_3_x constants including all
pre-release versions and bug fixes.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-12-17 16:24:50 -06:00
Daniel Doubrovkine (dB.) bbb6e656c0
Add version 1.2.3. (#1760)
Signed-off-by: dblock <dblock@dblock.org>
2021-12-17 09:14:37 -08:00
Poojita Raj 4b97713a2f
Bridging the gap in network overhead measurement in the profiler (#1360)
* Bridging the gap in network overhead measurement in the profiler

Signed-off-by: Poojita Raj <poojiraj@amazon.com>

* add tests + remove total n/w time field

Signed-off-by: Poojita Raj <poojiraj@amazon.com>

* refactor network time into a class

Signed-off-by: Poojita Raj <poojiraj@amazon.com>

* gradle test fix

Signed-off-by: Poojita Raj <poojiraj@amazon.com>
2021-12-16 19:50:04 -08:00
Nick Knize 6cc462b92d
[Remove] various builder and mapping deprecations (#1752)
This commit removes the following deprecations and invalid tests:

* LegacyUpdateMappingIntegrationIT
* ShapeBuilder new z axis checks
* MatchPhrase new zeroTermsQuery checks
* TypeQuery utf8
* CompositeValuesSource format checks
* LegacyESVersionTests bumped to 6.8.15
* LegacyDynamicMapping, LegacySimilarity, LegacyMapperService tests

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-12-16 18:42:39 -06:00
Nick Knize 8c2d8f5431
[Remove] Remaining Flavor Serialization (#1751)
This commit removes unnecessary serialization of unused flavor variable in build
metadata from V_1_3_0+

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-12-16 17:31:25 -06:00
Nick Knize ef44182731
[Remove] DynamicTemplate deprecations (#1742)
This commit removes legacy version checks in DynamicTemplate parsing that are no
longer valid in OpenSearch 2.0.0.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-12-16 12:52:25 -06:00
Nick Knize 5550f8d7e2
[Remove] Analyzer Deprecations (#1741)
This commit removes deprecated analyzer instantiation that is no longer
permitted in OpenSearch 2.0.0.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-12-16 12:51:49 -06:00
Megha Sai Kavikondala 22bfadf324
Adding a cancelled field to tell if a cancellable task is cancelled (#1732)
* Changes made by adding the Task indicator and the IllegalArgs.

Signed-off-by: meghasaik <kavmegha@amazon.com>

* Changes made by changing the String message for uncancellable task.

Signed-off-by: meghasaik <kavmegha@amazon.com>
2021-12-16 09:49:49 -05:00
Vacha e66ea2c4f3
Avoid logging duplicate deprecation warnings multiple times (#1660)
* Avoid logging duplicate deprecation warnings multiple times

Signed-off-by: Vacha <vachshah@amazon.com>

* Fixes test failures

Signed-off-by: Vacha <vachshah@amazon.com>

* Adding deprecation logger tests

Signed-off-by: Vacha <vachshah@amazon.com>

* Using ConcurrentHashMap keySet

Signed-off-by: Vacha Shah <vachshah@amazon.com>
2021-12-15 15:26:44 -08:00
Sarat Vemulapalli a632b86e4c
Adding 1.2.2 (#1731)
Signed-off-by: Sarat Vemulapalli <vemulapallisarat@gmail.com>
2021-12-15 10:30:06 -05:00
Andriy Redko 6db435412b
Upgrade to log4j 2.16.0 (#1721)
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2021-12-14 07:34:45 -05:00
Andrew Ross 309649ce8a
Upgrade to logj4 2.15.0 (#1698)
Signed-off-by: Andrew Ross <andrross@amazon.com>
2021-12-10 13:03:41 -08:00
Daniel Doubrovkine (dB.) ec00b86859
Add version 1.2.1. (#1701)
Signed-off-by: dblock <dblock@dblock.org>
2021-12-10 14:47:52 -05:00
Tianli Feng baa10b9b20
Fix unit test testFailsHealthOnHungIOBeyondHealthyTimeout() by incresing the max waiting time before assertion (#1692)
* Increase the max waiting time before checking the FS health after restoring the FS status in the test

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* Remove another duplicate assetion statement

Signed-off-by: Tianli Feng <ftianli@amazon.com>
2021-12-10 12:25:57 -05:00
Sarat Vemulapalli e0e6995c4a
Updating Log4j to 2.11.2 (#1696)
Signed-off-by: Sarat Vemulapalli <vemulapallisarat@gmail.com>
2021-12-10 08:03:45 -08:00
Nick Knize 538b40a0dc
Remove LegacyESVersion.V_6_2_x constants (#1686)
This commit removes LegacyESVersion.V_6_2_x constants including all
pre-release versions and bug fixes.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-12-09 16:08:00 -06:00
Nick Knize 821417b9ab
Remove LegacyESVersion.V_6_1_x constants (#1681)
This commit removes LegacyESVersion.V_6_1_x constants including all
pre-release versions and bug fixes.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-12-08 23:11:36 -06:00
Nick Knize b74d71fb74
Remove 6.0.* version constants (#1658)
This PR removes LegacyESVersion.V_6_0_* constants including all pre-release
versions and bug fixes.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-12-08 13:37:21 -06:00
Nick Knize 33d8677796
Refactor LegacyESVersion tests from Version tests (#1662)
In preparation for removing all LegacyESVersion support by 3.0; this commit
largely refactors the LegacyESVersion test logic from the OpenSearch Version
test logic into an independent test class. This PR also updates Version.fromString
to ensure a proper legacy version is returned when major is > 3 (to support
legacy yaml test and build scripts).

Note that bwc w/ legacy versions are still supported so some cross compatibility
testing is retained in the Version test class.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-12-07 08:27:17 -06:00
Nick Knize 4db97aa470
[Remove] Deprecated Zen1 Discovery (#1216)
Zen1 discovery was deprecated in Legacy 7.x for eventual removal. OpenSearch 1.x
carries this deprecation. This commit completely removes all support for Zen1
discovery in favor of Zen2.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-12-03 15:27:10 -06:00
Nick Knize 17e7a4962d
Remove Deprecated SimpleFS (#1639)
Lucene 9 removes support for SimpleFS File System format. This PR completely
removes SimpleFS support which was deprecated in a previous PR.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-12-03 09:57:59 -06:00
Xue Zhou 74943c43c6
Rename field_masking_span to span_field_masking (#1606)
* Rename field_masking_span to span_field_masking

Signed-off-by: Xue Zhou <xuezhou@amazon.com>

* Update SearchModuleTests.java

Signed-off-by: Xue Zhou <xuezhou@amazon.com>

* Rename field_masking_span to span_field_masking

Signed-off-by: Xue Zhou <xuezhou@amazon.com>
2021-11-29 14:01:43 -05:00
Megha Sai Kavikondala 12a229410c
Giving informative error messages for double slashes in API call URLs (#1568)
* Signed-off-by: Megha Sai Kavikondala <kavmegha@amazon.com>

Changes related to Informative error messages.

* Signed-off-by: Megha Sai Kavikondala <kavmegha@amazon.com>

Spotless changes

* Signed-off-by: Megha Sai Kavikondala <kavmegha@amazon.com>

naming and message changes.

Signed-off-by: Megha Sai Kavikondala <kavmegha@amazon.com>

* Signed-off-by: Megha Sai Kavikondala <kavmegha@amazon.com>

Deleting the log message.
2021-11-22 18:01:39 -05:00
Poojita Raj a15c526612
Improve leader node-left logging to indicate timeout/coordination state rejection (#1584) 2021-11-19 13:15:01 -08:00
Andrew Ross d467283475
Start MockLogAppender before adding to static context (#1587)
I observed a test failure with the message
'Attempted to append to non-started appender mock' from an assertion in
`OpenSearchTestCase::after`. I believe this indicates that a
MockLogAppender (which is named "mock") was added as an appender to the
static logging context and some other test in the same JVM happened to
cause a logging statement to hit that appender and cause an error, which
then caused an unrelated test to fail (because they share static state
with the logger). Almost all usages of MockLogAppender start it
immediately after creation. I found a few that did not and fixed those.
I also made a static helper in MockLogAppender to start it upon
creation.

Signed-off-by: Andrew Ross <andrross@amazon.com>
2021-11-19 13:37:46 -05:00
Nick Knize 99a356bfc6
[main] Add staged version 1.3.0 for bwc (#1510)
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-11-04 15:56:08 -05:00
Nick Knize ac37cc99b2
Add staged version 1.1.1 (#1506)
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-11-04 14:19:21 -05:00
Ryan Bogan 286121f0dc
Added logic to allow {dot} files on startup (#1437)
* Added logic to allow {dot} files on startup

Signed-off-by: Ryan Bogan <rbogan@amazon.com>

* Ensures that only plugin directories are returned by findPluginDirs()

Signed-off-by: Ryan Bogan <rbogan@amazon.com>

* Prevents . files from being returned as plugins

Signed-off-by: Ryan Bogan <rbogan@amazon.com>
2021-11-04 10:01:55 -04:00
Xue Zhou f54cc382d5
remove codeQL warning about implicit narrowing conversion in compound assignment (#1403)
Signed-off-by: Xue Zhou <xuezhou@amazon.com>
2021-11-02 13:49:25 -07:00
Rabi Panda d45f5dfcab
Make TranslogDeletionPolicy abstract for extension (#1456)
As part of the commit 2ebd0e04, we added a new method to the EnginePlugin to provide a 
custom TranslogDeletionPolicy. This commit makes minTranslogGenRequired method 
abstract in this class for implementation by child classes. The default implementation 
is provided by DefaultTranslogDeletionPolicy.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-11-01 10:25:30 -05:00
Nick Knize 294ad0eb6f
[Upgrade] 1.2 BWC to Lucene 8.10.1 (#1460)
This commit upgrades the BWC 1.2 lucene dependency to version 8.10.1

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-10-29 10:45:29 -07:00
Rabi Panda 2cb71a33b8
Remove deprecated settings and logic for translog pruning by retention lease. (#1416)
The settings and the corresponding logic for translog pruning by retention lease which were added as part of #1100 have been deprecated. This commit removes those deprecated code in favor of an extension point for providing a custom TranslogDeletionPolicy.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-10-29 08:25:12 -07:00
Nick Knize 53334b2ce4
Upgrade to Lucene 8.10.1 (#1440)
This commit upgrades to the latest release of lucene 8.10

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-10-28 10:06:53 -05:00
Andrew Ross 12789f89a3
Close first engine instance before creating second (#1457)
When creating the second instance of an InternalEngine using the same
translog config of the default InternalEngine instance, the second
instance will attempt to delete all the existing translog files. I found
a deterministic test failure when running with the seed
`E3E6AAD95ABD299B`.

As opposed to creating a second engine instance with a different
translog location, just close the first one before creating the second.

Signed-off-by: Andrew Ross <andrross@amazon.com>
2021-10-28 01:11:05 -07:00
Rabi Panda c86d765e7c
Add extension point for custom TranslogDeletionPolicy in EnginePlugin. (#1404)
This commit adds a method that can be used to provide a custom TranslogDeletionPolicy
from within plugins that implement the EnginePlugin interface. This enables plugins to
provide a custom deletion policy with the current limitation that only one plugin can
override the policy. An exception will be thrown if more than one plugin overrides the
policy.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-10-26 09:45:10 -05:00
Andriy Redko acac3cc285
Fixing org.opensearch.repositories.azure.AzureBlobContainerRetriesTests and org.opensearch.action.admin.cluster.node.stats.NodeStatsTests (#1390)
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2021-10-20 15:27:18 -04:00
Saurabh Singh 284968bb85
Update node attribute check to version update (1.2) check for shard indexing pressure serialization. (#1395)
This commit adds an explicit version check for shard indexing pressure 
serialization. This is required to not mandate test have the cluster service 
initialized while asserting node attributes.

Signed-off-by: Saurabh Singh <sisurab@amazon.com>
Co-authored-by: Saurabh Singh <sisurab@amazon.com>
2021-10-20 12:18:46 -05:00
Nick Knize ecac8d3c38
Add EngineConfig extensions to EnginePlugin (#1387)
This commit adds an extension point to EngineConfig through EnginePlugin using
a new EngineConfigFactory mechanism. EnginePlugin provides interface methods to
override configurations in EngineConfig. The EngineConfigFactory produces a new
instance of the EngineConfig using these overrides. Defaults are used absent
overridden configurations.

This serves as a mechanism to override Engine configurations (e.g., CodecService,
TranslogConfig) enabling Plugins to have higher fidelity for changing Engine
behavior without having to override the entire Engine (which is only permitted for
a single plugin).

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-10-19 23:04:28 -05:00
Owais Kazi 996d33adb2
Run spotless and exclude checkstyle on server module (#1380)
Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>
2021-10-19 08:32:54 -04:00
Saurabh Singh 24fd89a1fd
Minor fix for the flaky test to reduce concurrency (#1361) (#1364)
Fixes flakiness for test testReplicaThreadedThroughputDegradationAndRejection.

Reduced the number of threads concurrently executing from the initial range between (100-120) to a new range between (80-100), as the previous range was breaking the node limits set as 10kb, for every execution where number of threads were greater than 110.

Signed-off-by: Saurabh Singh <sisurab@amazon.com>
2021-10-14 15:25:17 -07:00
Andriy Redko 3779576c51
Modernize and consolidate JDKs usage across all stages of the build. Use JDK-17 as bundled JDK distribution to run tests (#1358)
* Modernize and consolidate JDKs usage across all stages of the build. Use JDK-17 as bundled JDK distribution to run tests

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* Using -Djava.security.egd=file:/dev/urandom explicitly for cli tests

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2021-10-13 17:25:48 -04:00
Andriy Redko e9635d6bfe
Replace securemock with mock-maker (test support), update Mockito to 3.12.4 (#1332)
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2021-10-10 14:18:54 -04:00
Saurabh Singh 3665daf5d0
Add Shard Level Indexing Pressure (#1336)
Shard level indexing pressure improves the current Indexing Pressure framework which performs memory accounting at node level and rejects the requests. This takes a step further to have rejections based on the memory accounting at shard level along with other key performance factors like throughput and last successful requests. 

**Key features**
- Granular tracking of indexing tasks performance, at every shard level, for each node role i.e. coordinator, primary and replica.
- Smarter rejections by discarding the requests intended only for problematic index or shard, while still allowing others to continue (fairness in rejection).
- Rejections thresholds governed by combination of configurable parameters (such as memory limits on node) and dynamic parameters (such as latency increase, throughput degradation).
- Node level and shard level indexing pressure statistics exposed through stats api.
- Integration of Indexing pressure stats with Plugins for for metric visibility and auto-tuning in future.
- Control knobs to tune to the key performance thresholds which control rejections, to address any specific requirement or issues.
- Control knobs to run the feature in shadow-mode or enforced-mode. In shadow-mode only internal rejection breakdown metrics will be published while no actual rejections will be performed.

The changes were divided into small manageable chunks as part of the following PRs against a feature branch.

- Add Shard Indexing Pressure Settings. #716
- Add Shard Indexing Pressure Tracker. #717
- Refactor IndexingPressure to allow extension. #718
- Add Shard Indexing Pressure Store #838
- Add Shard Indexing Pressure Memory Manager #945
- Add ShardIndexingPressure framework level construct and Stats #1015
- Add Indexing Pressure Service which acts as orchestrator for IP #1084
- Add plumbing logic for IndexingPressureService in Transport Actions. #1113
- Add shard indexing pressure metric/stats via rest end point. #1171
- Add shard indexing pressure integration tests. #1198

Signed-off-by: Saurabh Singh <sisurab@amazon.com>
Co-authored-by: Saurabh Singh <sisurab@amazon.com>
Co-authored-by: Rabi Panda <adnapibar@gmail.com>
2021-10-07 11:06:15 -07:00
Andriy Redko d46c206f29
[BUG] ConcurrentSnapshotsIT#testAssertMultipleSnapshotsAndPrimaryFailOver fails intermittently (#1311)
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2021-09-30 12:44:29 -04:00
Andriy Redko 180db5cd09
Support for Heap after GC stats (correction after backport to 1.2.0) (#1315)
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2021-09-30 09:48:24 -04:00
Andriy Redko 80388a8a29
Support for Heap after GC stats (#1265)
* Support for Heap after GC stats

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* Addressing code review comments

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* Using the right version 2.0.0 (instead of 1.2.0) for the change

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2021-09-28 14:40:00 -04:00
Shivansh Arora 416220f510
Making GeneralScriptException an Implementation of OpensearchWrapperException (#1066)
Signed-off-by: Shivansh Arora <hishiv@amazon.com>
2021-09-28 12:17:43 -04:00
Nick Knize 46e0f63539
[Tests] Translog Pruning tests to MetadataCreateIndexServiceTests (#1295)
This commit adds test coverage for translog pruning setting to
MetadataCreateIndexServiceTests

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-09-25 23:05:32 -05:00
Sai 29c88c6900
Rename translog pruning setting to CCR specific setting and addressed Bug in the test case (#1243)
* Rename translog pruing setting to CCR specific setting

Signed-off-by: Sai Kumar <karanas@amazon.com>

* Rename to index.plugins.replication.translog.retention_lease.pruning.enabled as
index settings needs "index." as prefix

Signed-off-by: Sai Kumar <karanas@amazon.com>

* Add deprecations to retention pruning controls

This commit adds deprecation flags to all added settings, variables, and methods
specific to ccr's retention lease pruning mechanism.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* Addressed CR comments

Signed-off-by: Sai Kumar <karanas@amazon.com>

* fix javadoc deprecation

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* fix deprecation tag in TranslogDeletionPolicy

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* Addressed test issue under translog tests

Signed-off-by: Sai Kumar <karanas@amazon.com>

Co-authored-by: Nicholas Walter Knize <nknize@apache.org>
2021-09-24 16:09:59 -07:00
Bukhtawar Khan 390e678f92
Handle shard over allocation during partial zone/rack or independent node failures (#1149)
The changes ensure that in the event of a partial zone failure, the surviving nodes in the minority zone don't get overloaded with shards, this is governed by a skewness limit.

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>
2021-09-20 10:32:23 -07:00
Bukhtawar Khan f7e2984248
Introduce FS Health HEALTHY threshold to fail stuck node (#1167)
This will cause the leader stuck on IO during publication to step down and eventually trigger a leader election.

Issue Description
---
The publication of cluster state is time bound to 30s by a cluster.publish.timeout settings. If this time is reached before the new cluster state is committed, then the cluster state change is rejected and the leader considers itself to have failed. It stands down and starts trying to elect a new master.

There is a bug in leader that when it tries to publish the new cluster state it first tries acquire a lock to flush the new state under a mutex to disk. The same lock is used to cancel the publication on timeout. Below is the state of the timeout scheduler meant to cancel the publication. So essentially if the flushing of cluster state is stuck on IO, so will the cancellation of the publication since both of them share the same mutex. So leader will not step down and effectively block the cluster from making progress.

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>
2021-09-16 17:02:25 -07:00
Andriy Redko b6c8bdf872
Drop mocksocket in favour of custom security manager checks (tests only) (#1205)
* Drop mocksocket in favour of custom security manager checks (tests only)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* Slightly relaxed host checks to allow all local addresses

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2021-09-16 17:21:47 -04:00
Nick Knize cbbf967d76
[Version] Add 1.2 for BWC testing (#1241)
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-09-15 09:07:53 -07:00
Sai 7e440af9e4
Changes to support retrieval of operations from translog based on specified range (#1210)
* Changes to support retrieval of operations from translog based on specified range

Signed-off-by: Sai Kumar <karanas@amazon.com>

* Addressed CR comments

Signed-off-by: Sai Kumar <karanas@amazon.com>

* Added testcases for internal engine

Signed-off-by: Sai Kumar <karanas@amazon.com>
2021-09-08 20:22:24 +05:30
Sai a9e1913bd5
Support for translog pruning based on retention leases (#1038)
* Support for translog pruning based on retention leases

Signed-off-by: Sai Kumar <karanas@amazon.com>

* Addressed CR Comments

Signed-off-by: Sai Kumar <karanas@amazon.com>

* Addressed test case issue

Signed-off-by: Sai Kumar <karanas@amazon.com>
2021-09-08 16:23:00 +05:30
Vacha 83332c8ab6
Kept the original constructor for PluginInfo to maintain bwc (#1206) 2021-09-02 18:09:15 -07:00
Nick Knize 4b31e1ba04
[Version] Increment main to 2.0 (#1192)
Increment version on main to 2.0.0.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Co-authored-by: dblock <dblock@amazon.com>
2021-09-01 22:35:53 -05:00
Andriy Redko 6e199d223b
Restoring alpha/beta/rc version semantics (#1112)
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2021-09-01 18:22:20 -05:00
Nick Knize 3a7f0762ac
[Bug] Fix mixed cluster support for OpenSearch 2+ (#1191)
The version framework only added support for OpenSearch 1.x bwc with legacy
clusters. This commit adds support for v2.0 which will be the last version with
bwc support for legacy clusters (v7.10)

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-09-01 14:45:46 -05:00
Abbas Hussain 91d9922f3b
Reduce iterations to improve test run time (#1168)
Signed-off-by: Abbas Hussain <abbas_10690@yahoo.com>
2021-08-30 23:51:52 +05:30
Abbas Hussain 9b4440d1f6
Tune datanode count and shards count to improve test run time (#1170)
Signed-off-by: Abbas Hussain <abbas_10690@yahoo.com>
2021-08-30 22:33:31 +05:30
Nick Knize 7ed024e2c8
Add 1.0.1 revision (#1152)
This commit stages the branch to the next 1.0.1 patch release. BWC testing needs
this even if the next revision is never actually released.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-08-25 23:11:43 -05:00
Nicholas Walter Knize 33cabe2e45
[Bug] Change 1.0.0 version check in PluginInfo
PluginInfo should use .onOrAfter(Version.V_1_1_0) instead of
.after(Version.V_1_0_0) for the new custom folder name for plugin feature.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-08-25 16:30:11 -05:00
Nick Knize 5ae00456a0
Upgrade to Lucene 8.9 (#1080)
This commit upgrades to the official lucene 8.9 release

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-08-20 11:28:06 -05:00
Nick Knize ff7e7904ca
[DEPRECATE] SimpleFS in favor of NIOFS (#1073)
Lucene 9 removes support for SimpleFS File System format. This commit deprecates
the SimpleFS format in favor of NIOFS.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-08-19 17:56:55 -05:00
Sorabh 93a621f54e
Fix failure in SearchCancellationIT.testMSearchChildReqCancellationWithHybridTimeout (#1103)
In some cases as one shared with issue #1099, the maxConcurrentSearchRequests was chosen as 0 which
will compute the final value during execution of the request based on processor counts. When this
computed value is less than number of search request in msearch request, it will execute all the
requests in multiple iterations causing the failure since test will only wait for one such
iteration. Hence setting the maxConcurrentSearchRequests explicitly to number of search requests
being added in the test to ensure correct behavior

Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com>
2021-08-17 14:10:31 -04:00
Sorabh 9b6e621452
Part 1: Support for cancel_after_timeinterval parameter in search and msearch request (#986)
* Part 1: Support for cancel_after_timeinterval parameter in search and msearch request

This commit introduces the new request level parameter to configure the timeout interval after which
a search request will be cancelled. For msearch request the parameter is supported both at parent
request and at sub child search requests. If it is provided at parent level and child search request
doesn't have it then the parent level value is set at such child request. The parent level msearch
is not used to cancel the parent request as it may be tricky to come up with correct value in cases
when child search request can have different runtimes

TEST: Added test for ser/de with new parameter

Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com>

* Part 2: Support for cancel_after_timeinterval parameter in search and msearch request

This commit adds the handling of the new request level parameter and schedule cancellation task. It
also adds a cluster setting to set a global cancellation timeout for search request which will be
used in absence of request level timeout.

TEST: Added new tests in SearchCancellationIT
Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com>

* Address Review feedback for Part 1

Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com>

* Address review feedback for Part 2

Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com>

* Update CancellableTask to remove the cancelOnTimeout boolean flag

Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com>

* Replace search.cancellation.timeout cluster setting with search.enforce_server.timeout.cancellation to control if cluster level cancel_after_time_interval should take precedence over request level cancel_after_time_interval value

Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com>

* Removing the search.enforce_server.timeout.cancellation cluster setting and just keeping search.cancel_after_time_interval setting with request level parameter taking the precedence.

Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com>

Co-authored-by: Sorabh Hamirwasia <hsorabh@amazon.com>
2021-08-12 11:01:28 -04:00
frotsch c34f9ab75c
Avoid crashing on using the index.lifecycle.name in the API body (#1060)
* Avoid crashing on using the index.lifecycle.name in the API body

Signed-off-by: frotsch <frotsch@mailbox.org>
2021-08-10 11:03:19 -04:00
Bukhtawar Khan fbdc180944
Max scroll limit breach to throw a OpenSearchRejectedExecutionException (#1054)
* Changes the Exception to throw a OpenSearchRejectedExecutionException on max scroll limit breach

Signed-off-by: Bukhtawar Khan bukhtawa@amazon.com
2021-08-10 11:02:04 -04:00
xuezhou25 7d18eaf3a4
Replaced custom built JNA by official JNA distribution. (#1003)
Signed-off-by: Xue Zhou <xuezhou@amazon.com>
2021-08-03 13:52:53 -04:00
Bukhtawar Khan 1a87534c20
Prevent /_cat/master from getting tripped by the CB (#1036)
_cat/master is a fundamental API to know the master instance in the cluster. Given RestClusterState is exempted from tripping already, doesn't make sense for RestMasterAction to trip

Signed-off-by: Bukhtawar Khan bukhtawa@amazon.com
2021-08-02 10:15:42 -07:00
Vlad Rozov b7cf1fae41
Introduce RestHandler.Wrapper to help with delegate implementations (#1004)
Signed-off-by: Vlad Rozov <vrozov@users.noreply.github.com>
2021-07-30 15:17:27 -04:00
Vlad Rozov ee644bdc40
Add unit test for RestActionListener. Validate that onFailure() sends response even when BytesRestResponse can not be constructed using passed exception. Follow up on #923. (#1024)
Signed-off-by: Vlad Rozov <vrozov@users.noreply.github.com>
2021-07-30 14:38:14 -04:00
Vacha 19e54d6624
Allowing custom folder name for plugin installation (#848)
Signed-off-by: Vacha Shah <vachshah@amazon.com>
2021-07-22 04:28:49 +05:30
afazel 23d604571c
Introduce replaceRoutes() method and 2 new constructors to RestHandler.java (#947)
* Add addRoutesPrefix() method to RestHandler.java

Signed-off-by: Azar Fazel <azar.fazel@gmail.com>
2021-07-15 19:14:15 -07:00
Vlad Rozov 854967ff92
Fail fast when BytesRestResponse ctor throws exception (#923)
Signed-off-by: Vlad Rozov <vrozov@users.noreply.github.com>
2021-07-15 14:52:28 -05:00
Chang Liu 7241127c1d
Avoid override of routes() in BaseRestHandler to respect the default behavior defined in RestHandler (#889) 2021-07-14 16:22:39 -05:00
Rabi Panda 4906a3c1b0
A CLI tool to assist during an upgrade to OpenSearch. (#846)
This change adds the initial version of a new CLI tool `opensearch-upgrade` as part of the OpenSearch distribution. This tool is meant for assisting during an upgrade from an existing Elasticsearch v7.10.2/v6.8.0 node to OpenSearch. It automates the process of importing existing configurations and installing of core plugins.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-07-13 09:51:02 -07:00
Daniel Doubrovkine (dB.) 88f17b1822
Cleanup TESTING and DEVELOPER_GUIDE markdowns (#946) 2021-07-12 15:02:56 -05:00
Daniel Doubrovkine (dB.) 5d020170ff
Lower build requirement from Java 14+ to Java 11+ (#940)
* Lower build requirement from Java 14+ to Java 11+

Avoid use of -Werror -Xlint:all, which may change significantly across
java releases (new warnings could be added). Instead, just list the
warnings individually.

Workaround JDK 11 compiler bug (JDK-8209058) that only impacts test fixture
code in the build itself.

Signed-off-by: Robert Muir <rmuir@apache.org>

* Disable warning around -source 7 -release 7 for java version checker

The java version checker triggers some default warnings because it
targets java7:

```
> Task :distribution:tools:java-version-checker:compileJava FAILED
warning: [options] source value 7 is obsolete and will be removed in a future release
warning: [options] target value 7 is obsolete and will be removed in a future release
warning: [options] To suppress warnings about obsolete options, use -Xlint:-options.
error: warnings found and -Werror specified
```

Suppress this warning explicitly for this module.

Signed-off-by: Robert Muir <rmuir@apache.org>

* more java14 -> java11 cleanup

Signed-off-by: Robert Muir <rmuir@apache.org>

Co-authored-by: Robert Muir <rmuir@apache.org>
2021-07-06 11:55:23 -05:00
Rabi Panda f652e95236
Update favicon for OpenSearch (#932)
Signed-off-by: Rabi Panda <pandarab@amazon.com>
2021-07-02 13:40:28 -07:00
shwetathareja 8082604ec2
Refresh OpenSearch nodes version in cluster state after upgrade (#865)
Signed-off-by: Shweta Thareja <tharejas@amazon.com>

Co-authored-by: Shweta Thareja <tharejas@amazon.com>
2021-07-02 09:03:34 -07:00
Tianli Feng 4a1add9481
Add 'tagline' back to MainResponse in server that was removed in PR #427 (#913)
* Add "tagline" field back to "MainResponse" in sever side (not in rest-high-level-client side) that removed in PR #427 .
* Replace with a new tagline "The OpenSearch Project: https://opensearch.org/".
* Turn the tagline into a constant in server/src/main/java/org/opensearch/action/main/MainResponse.java.
2021-07-01 15:30:29 -07:00
Marc Handalian c91c29e5c2
Remove distribution from main response in compatibility mode (#898)
This Change removes version.distribution when the version.number is
overridden with the cluster setting compatibility.override_main_response_version.

Signed-off-by: Marc Handalian <handalm@amazon.com>
2021-06-30 17:36:21 -07:00
Vlad Rozov 362f116abe
Replace metadata keys in OpenSearchException during serialization and deserialization (#905)
Signed-off-by: Vlad Rozov <vrozov@users.noreply.github.com>
2021-06-30 11:39:26 -07:00
Nick Knize ef1cbd9e3d
Version bump for 1.1 release (#772)
Bumps the minor to 1.1 for the 1.0 release.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-06-28 09:58:27 -05:00
Marc Handalian c1250c963d
Add cluster setting to spoof version number returned from MainResponse (#847)
This change adds a new cluster setting "compatibility.override_main_response_version"
that when enabled spoofs the version.number returned from MainResponse
for REST clients expecting legacy version 7.10.2.

Signed-off-by: Marc Handalian <handalm@amazon.com>
2021-06-22 10:42:23 -07:00
itiyamas 11babe7782
Decouple throttling limits for new and old indices. (#778)
* Decouple throttling limits for new and old indices.

Signed-off-by: Itiyama <itiyamas@amazon.com>

* Precommit fixes.

Signed-off-by: Itiyama <itiyamas@amazon.com>

* Review comments.

Signed-off-by: Itiyama <itiyamas@amazon.com>

* Review comments and test fix.

Signed-off-by: Itiyama <itiyamas@amazon.com>

* Checkstyle fixes.

Signed-off-by: Itiyama <itiyamas@amazon.com>

* Review comments.

Signed-off-by: Itiyama <itiyamas@amazon.com>
2021-06-17 10:58:50 +05:30