Commit Graph

55191 Commits

Author SHA1 Message Date
Andriy Redko d46c206f29
[BUG] ConcurrentSnapshotsIT#testAssertMultipleSnapshotsAndPrimaryFailOver fails intermittently (#1311)
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2021-09-30 12:44:29 -04:00
Andriy Redko 180db5cd09
Support for Heap after GC stats (correction after backport to 1.2.0) (#1315)
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2021-09-30 09:48:24 -04:00
Andriy Redko 80388a8a29
Support for Heap after GC stats (#1265)
* Support for Heap after GC stats

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* Addressing code review comments

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* Using the right version 2.0.0 (instead of 1.2.0) for the change

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2021-09-28 14:40:00 -04:00
Daniel Doubrovkine (dB.) 457c1cd6ec
fix typo (#1305)
Signed-off-by: sgbasaraner <sarpbasaraner@gmail.com>

Co-authored-by: Sarp Güney Başaraner <sarpbasaraner@gmail.com>
2021-09-28 13:54:35 -04:00
Shivansh Arora 416220f510
Making GeneralScriptException an Implementation of OpensearchWrapperException (#1066)
Signed-off-by: Shivansh Arora <hishiv@amazon.com>
2021-09-28 12:17:43 -04:00
Nick Knize 46e0f63539
[Tests] Translog Pruning tests to MetadataCreateIndexServiceTests (#1295)
This commit adds test coverage for translog pruning setting to
MetadataCreateIndexServiceTests

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-09-25 23:05:32 -05:00
Sai 29c88c6900
Rename translog pruning setting to CCR specific setting and addressed Bug in the test case (#1243)
* Rename translog pruing setting to CCR specific setting

Signed-off-by: Sai Kumar <karanas@amazon.com>

* Rename to index.plugins.replication.translog.retention_lease.pruning.enabled as
index settings needs "index." as prefix

Signed-off-by: Sai Kumar <karanas@amazon.com>

* Add deprecations to retention pruning controls

This commit adds deprecation flags to all added settings, variables, and methods
specific to ccr's retention lease pruning mechanism.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* Addressed CR comments

Signed-off-by: Sai Kumar <karanas@amazon.com>

* fix javadoc deprecation

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* fix deprecation tag in TranslogDeletionPolicy

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* Addressed test issue under translog tests

Signed-off-by: Sai Kumar <karanas@amazon.com>

Co-authored-by: Nicholas Walter Knize <nknize@apache.org>
2021-09-24 16:09:59 -07:00
Xue Zhou 82d1d0ec08
fix gradle check fail due to renameing -min in #1094 (#1289)
Signed-off-by: Xue Zhou <xuezhou@amazon.com>
2021-09-24 12:06:16 -05:00
Tianli Feng 20728c3725
Add guide for generating code coverage report in TESTING.md (#1264)
Signed-off-by: Tianli Feng <ftianli@amazon.com>
2021-09-21 19:44:08 -04:00
Xue Zhou 0ab8e34022
Rename artifact produced by the build to include -min (#1251)
Signed-off-by: Xue Zhou <xuezhou@amazon.com>
2021-09-21 19:43:35 -04:00
Andriy Redko cdbc84f09d
Update Jackson to 2.12.5 (#1247)
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2021-09-21 18:33:20 -04:00
CEHENKLE e66b717c98
adding components to DEVELOPER_GUIDE (#1200)
* adding components to DEVELOPER_GUIDE

Signed-off-by: CEHENKLE <henkle@amazon.com>

* small tweaks

Signed-off-by: CEHENKLE <henkle@amazon.com>
2021-09-20 20:59:49 -07:00
Bukhtawar Khan 390e678f92
Handle shard over allocation during partial zone/rack or independent node failures (#1149)
The changes ensure that in the event of a partial zone failure, the surviving nodes in the minority zone don't get overloaded with shards, this is governed by a skewness limit.

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>
2021-09-20 10:32:23 -07:00
Bukhtawar Khan f7e2984248
Introduce FS Health HEALTHY threshold to fail stuck node (#1167)
This will cause the leader stuck on IO during publication to step down and eventually trigger a leader election.

Issue Description
---
The publication of cluster state is time bound to 30s by a cluster.publish.timeout settings. If this time is reached before the new cluster state is committed, then the cluster state change is rejected and the leader considers itself to have failed. It stands down and starts trying to elect a new master.

There is a bug in leader that when it tries to publish the new cluster state it first tries acquire a lock to flush the new state under a mutex to disk. The same lock is used to cancel the publication on timeout. Below is the state of the timeout scheduler meant to cancel the publication. So essentially if the flushing of cluster state is stuck on IO, so will the cancellation of the publication since both of them share the same mutex. So leader will not step down and effectively block the cluster from making progress.

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>
2021-09-16 17:02:25 -07:00
Andriy Redko b6c8bdf872
Drop mocksocket in favour of custom security manager checks (tests only) (#1205)
* Drop mocksocket in favour of custom security manager checks (tests only)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* Slightly relaxed host checks to allow all local addresses

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2021-09-16 17:21:47 -04:00
Nick Knize cbbf967d76
[Version] Add 1.2 for BWC testing (#1241)
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-09-15 09:07:53 -07:00
Tianli Feng 53d60b72b2
Fix defects in code-coverage.gralde to generate code coverage report properly (#1214)
* Refactor the logic to control the format for code coverage report and rename the system property

* Remove outdated code of giving JaCoCo files permission when Java security manager enabled

Signed-off-by: Tianli Feng <ftianli@amazon.com>
2021-09-13 10:56:11 -07:00
Nick Knize aecc7bd005
[Bug] Fix InstallPluginCommand to use proper key signatures (#1233)
The public key has changed since the initial release. This commit fixes the
public key and uses the .sig files that are published to the artifacts site.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-09-10 14:58:31 -07:00
Abbas Hussain 65abe4a45c
Generate release notes for 1.1 (#1230)
Signed-off-by: Abbas Hussain <abbas_10690@yahoo.com>
2021-09-09 00:51:53 +05:30
Rabi Panda 3870522228
Run link checker GitHub action on schedule. (#1221)
Most PRs do not add/update links, however sites go down often. This change makes sure that we catch any broken link in the repository and fix it, but at the same time we do not want to block PRs because of some unrelated broken links.

This PR updates the workflow to run everyday at midnight UTC.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-09-08 11:24:19 -07:00
Sai 7e440af9e4
Changes to support retrieval of operations from translog based on specified range (#1210)
* Changes to support retrieval of operations from translog based on specified range

Signed-off-by: Sai Kumar <karanas@amazon.com>

* Addressed CR comments

Signed-off-by: Sai Kumar <karanas@amazon.com>

* Added testcases for internal engine

Signed-off-by: Sai Kumar <karanas@amazon.com>
2021-09-08 20:22:24 +05:30
Andriy Redko 0b773c2cec
Fix org.opensearch.index.reindex.ReindexRestClientSslTests#testClientSucceedsWithCertificateAuthorities - javax.net.ssl.SSLPeerUnverifiedException (#1212)
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2021-09-08 09:08:05 -04:00
Andriy Redko ddb37dea32
Added Eclipse import instructions to DEVELOPER_GUIDE.md (#1215)
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2021-09-08 09:05:02 -04:00
Sai a9e1913bd5
Support for translog pruning based on retention leases (#1038)
* Support for translog pruning based on retention leases

Signed-off-by: Sai Kumar <karanas@amazon.com>

* Addressed CR Comments

Signed-off-by: Sai Kumar <karanas@amazon.com>

* Addressed test case issue

Signed-off-by: Sai Kumar <karanas@amazon.com>
2021-09-08 16:23:00 +05:30
Sai 919c5e05aa
Exclude failing links from plugins/modules (#1223)
Signed-off-by: Sai Kumar <karanas@amazon.com>
2021-09-08 13:39:53 +05:30
Sven R 268d93b375
FreeBSD Java support (#1014)
Signed-off-by: hackacad <admin@hackacad.net>
2021-09-07 18:17:58 -04:00
Vacha 83332c8ab6
Kept the original constructor for PluginInfo to maintain bwc (#1206) 2021-09-02 18:09:15 -07:00
Anas Alkouz f37f29c996
Add more instructions how to install/configure git secrets (#1202)
Signed-off-by: Anas Alkouz <aalkouz@amazon.com>

Co-authored-by: Anas Alkouz <aalkouz@amazon.com>
2021-09-02 17:24:12 -04:00
Nick Knize 4b31e1ba04
[Version] Increment main to 2.0 (#1192)
Increment version on main to 2.0.0.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Co-authored-by: dblock <dblock@amazon.com>
2021-09-01 22:35:53 -05:00
Abbas Hussain fa8126004c
Upgrade apache commons-compress to 1.21 (#1197)
Signed-off-by: Abbas Hussain <abbas_10690@yahoo.com>
2021-09-02 08:35:42 +05:30
Daniel Doubrovkine (dB.) ecd9875e7c
Clarify opensearch.version to not include -SNAPSHOT. (#1186)
Signed-off-by: dblock <dblock@dblock.org>
2021-09-01 21:54:41 -04:00
Daniel Doubrovkine (dB.) 20c6f786ac
Added all icu-project.org websites to the link checker exclusions. (#1201)
Signed-off-by: dblock <dblock@dblock.org>
2021-09-01 21:13:48 -04:00
Andriy Redko 6e199d223b
Restoring alpha/beta/rc version semantics (#1112)
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2021-09-01 18:22:20 -05:00
Tianli Feng 652cdbd5af
Clarify JDK requirement in the developer guide (#1153)
* Explicitly point out the JDK 8 requirement is for runtime, but not for compiling.
* Clarify the JAVAx_HOME env variables are for the "backwards compatibility test".
* Add explanation on how the backwards compatibility tests get the OpenSearch distributions for a specific version.
Signed-off-by: Tianli Feng <ftianli@amazon.com>
2021-09-01 14:15:22 -07:00
Nick Knize 3a7f0762ac
[Bug] Fix mixed cluster support for OpenSearch 2+ (#1191)
The version framework only added support for OpenSearch 1.x bwc with legacy
clusters. This commit adds support for v2.0 which will be the last version with
bwc support for legacy clusters (v7.10)

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-09-01 14:45:46 -05:00
Daniel Doubrovkine (dB.) f298a41e4b
Extract excludes into a file, fix the link checker by adding http://site.icu-project.org/. (#1189) 2021-08-31 18:20:48 -04:00
Andriy Redko f60d093c63
Drop mocksocket & securemock dependencies from sniffer and rest client (no needed) (#1174)
* Drop mocksocket & securemock dependencies from sniffer and rest client (not needed)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* Removing .gitignore

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2021-08-31 14:18:37 -04:00
Minal Shah a43e6cf424
Fixed copyright to OpenSearch (#1175) 2021-08-31 12:55:28 -04:00
Abbas Hussain 91d9922f3b
Reduce iterations to improve test run time (#1168)
Signed-off-by: Abbas Hussain <abbas_10690@yahoo.com>
2021-08-30 23:51:52 +05:30
Abbas Hussain 9b4440d1f6
Tune datanode count and shards count to improve test run time (#1170)
Signed-off-by: Abbas Hussain <abbas_10690@yahoo.com>
2021-08-30 22:33:31 +05:30
Nicholas Walter Knize 93cce825eb
[BWC] Re-enable bwc testing after 1.0.1 version bump
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-08-26 07:11:53 -05:00
Nick Knize 7ed024e2c8
Add 1.0.1 revision (#1152)
This commit stages the branch to the next 1.0.1 patch release. BWC testing needs
this even if the next revision is never actually released.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-08-25 23:11:43 -05:00
Nicholas Walter Knize 26a4db6282
[BWC] Temporarily disable bwc testing while bumping 1.0.1
BWC tests are temporarily disabled to unblock main PRs while bumping 1.0.1

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-08-25 22:33:59 -05:00
Nicholas Walter Knize 33cabe2e45
[Bug] Change 1.0.0 version check in PluginInfo
PluginInfo should use .onOrAfter(Version.V_1_1_0) instead of
.after(Version.V_1_0_0) for the new custom folder name for plugin feature.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-08-25 16:30:11 -05:00
Nick Knize 5ae00456a0
Upgrade to Lucene 8.9 (#1080)
This commit upgrades to the official lucene 8.9 release

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-08-20 11:28:06 -05:00
Nick Knize ff7e7904ca
[DEPRECATE] SimpleFS in favor of NIOFS (#1073)
Lucene 9 removes support for SimpleFS File System format. This commit deprecates
the SimpleFS format in favor of NIOFS.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-08-19 17:56:55 -05:00
Sorabh 93a621f54e
Fix failure in SearchCancellationIT.testMSearchChildReqCancellationWithHybridTimeout (#1103)
In some cases as one shared with issue #1099, the maxConcurrentSearchRequests was chosen as 0 which
will compute the final value during execution of the request based on processor counts. When this
computed value is less than number of search request in msearch request, it will execute all the
requests in multiple iterations causing the failure since test will only wait for one such
iteration. Hence setting the maxConcurrentSearchRequests explicitly to number of search requests
being added in the test to ensure correct behavior

Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com>
2021-08-17 14:10:31 -04:00
Miki 911c13a395
Add themed logo to README (#988)
Signed-off-by: Miki <mehranb@amazon.com>
2021-08-14 13:19:06 -04:00
Vacha 072ccda932
Support for bwc tests for plugins (#1051)
* Support for bwc tests for plugins

Signed-off-by: Vacha <vachshah@amazon.com>

* Adding support for restart upgrades for plugins bwc

Signed-off-by: Vacha <vachshah@amazon.com>
2021-08-12 13:28:08 -07:00
kartg af6fbc77eb
Improving the Grok circular reference check to prevent stack overflow (#1079)
This change refactors the circular reference check in the Grok processor class
to use a formal depth-first traversal. It also includes a logic update to
prevent a stack overflow in one scenario and a check for malformed patterns.
This bugfix addresses CVE-2021-22144.

Signed-off-by: Kartik Ganesh <85275476+kartg@users.noreply.github.com>
2021-08-12 12:52:02 -04:00