Commit Graph

55103 Commits

Author SHA1 Message Date
Andriy Redko 05dc4bf4b3
Fixing post merge 3rd party audit issues (#1384)
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2021-10-18 22:06:50 -04:00
Andriy Redko 9612fe80b5
[repository-azure] plugin should use Azure Storage SDK v12 for Java (#1302)
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2021-10-18 19:48:32 -04:00
Vacha c7f5c90a5f
Upgrading netty version to 4.1.69.Final (#1363)
Signed-off-by: Vacha <vachshah@amazon.com>
2021-10-18 17:30:19 -04:00
Andriy Redko 8ea3364bc5
Modernize and consolidate JDKs usage across all stages of the build. Update JDK-14 requirement, switch to JDK-17 instead (#1368)
* Modernize and consolidate JDKs usage across all stages of the build. Update JDK-14 requirement, switch to JDK-17 instead

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* Updating phrasing based on review feedback

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* Fixed runtime Java version usage, it has to be respected when RUNTIME_JAVA_HOME == JAVA_HOME

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* Addressing review comments

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2021-10-15 09:04:05 -04:00
Saurabh Singh 24fd89a1fd
Minor fix for the flaky test to reduce concurrency (#1361) (#1364)
Fixes flakiness for test testReplicaThreadedThroughputDegradationAndRejection.

Reduced the number of threads concurrently executing from the initial range between (100-120) to a new range between (80-100), as the previous range was breaking the node limits set as 10kb, for every execution where number of threads were greater than 110.

Signed-off-by: Saurabh Singh <sisurab@amazon.com>
2021-10-14 15:25:17 -07:00
Vacha d151082832
Upgrade hadoop dependencies for hdfs plugin (#1335)
* Upgrade hadoop dependencies for hdfs plugin

Signed-off-by: Vacha <vachshah@amazon.com>

* Fixing gradle check failures

Signed-off-by: Vacha <vachshah@amazon.com>

* Upgrading htrace-core4 to 4.1.0

Signed-off-by: Vacha <vachshah@amazon.com>
2021-10-14 14:43:49 -04:00
Romain Tartière ea0fe7bfae
Allow building on FreeBSD (#1091)
* Allow building on FreeBSD

With this set of change, we are able to successfuly run:

```
./gradlew publishToMavenLocal -Dbuild.snapshot=false
```

This step is used in the OpenSearch repository context when building
plugins in the current state of the CI.

While here, reorder OS conditions alphabetically.

Before building, the openjdk14 package was installed and the environment
was adjusted to use it:

```
sudo pkg install openjdk14
export JAVA_HOME=/usr/local/openjdk14/
export PATH=$JAVA_HOME/bin:$PATH
```

Signed-off-by: Romain Tartière <romain@blogreen.org>

* Unbreak CI with FreeBSD support

Signed-off-by: dblock <dblock@dblock.org>

Co-authored-by: dblock <dblock@dblock.org>
2021-10-14 14:42:28 -04:00
Andriy Redko 3779576c51
Modernize and consolidate JDKs usage across all stages of the build. Use JDK-17 as bundled JDK distribution to run tests (#1358)
* Modernize and consolidate JDKs usage across all stages of the build. Use JDK-17 as bundled JDK distribution to run tests

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* Using -Djava.security.egd=file:/dev/urandom explicitly for cli tests

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2021-10-13 17:25:48 -04:00
CEHENKLE 5a29b4797f
initial commit to add in a dependabot.yml file (#1353)
Signed-off-by: CEHENKLE <henkle@amazon.com>
2021-10-11 09:31:57 -07:00
Andriy Redko e9635d6bfe
Replace securemock with mock-maker (test support), update Mockito to 3.12.4 (#1332)
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2021-10-10 14:18:54 -04:00
Saurabh Singh 3665daf5d0
Add Shard Level Indexing Pressure (#1336)
Shard level indexing pressure improves the current Indexing Pressure framework which performs memory accounting at node level and rejects the requests. This takes a step further to have rejections based on the memory accounting at shard level along with other key performance factors like throughput and last successful requests. 

**Key features**
- Granular tracking of indexing tasks performance, at every shard level, for each node role i.e. coordinator, primary and replica.
- Smarter rejections by discarding the requests intended only for problematic index or shard, while still allowing others to continue (fairness in rejection).
- Rejections thresholds governed by combination of configurable parameters (such as memory limits on node) and dynamic parameters (such as latency increase, throughput degradation).
- Node level and shard level indexing pressure statistics exposed through stats api.
- Integration of Indexing pressure stats with Plugins for for metric visibility and auto-tuning in future.
- Control knobs to tune to the key performance thresholds which control rejections, to address any specific requirement or issues.
- Control knobs to run the feature in shadow-mode or enforced-mode. In shadow-mode only internal rejection breakdown metrics will be published while no actual rejections will be performed.

The changes were divided into small manageable chunks as part of the following PRs against a feature branch.

- Add Shard Indexing Pressure Settings. #716
- Add Shard Indexing Pressure Tracker. #717
- Refactor IndexingPressure to allow extension. #718
- Add Shard Indexing Pressure Store #838
- Add Shard Indexing Pressure Memory Manager #945
- Add ShardIndexingPressure framework level construct and Stats #1015
- Add Indexing Pressure Service which acts as orchestrator for IP #1084
- Add plumbing logic for IndexingPressureService in Transport Actions. #1113
- Add shard indexing pressure metric/stats via rest end point. #1171
- Add shard indexing pressure integration tests. #1198

Signed-off-by: Saurabh Singh <sisurab@amazon.com>
Co-authored-by: Saurabh Singh <sisurab@amazon.com>
Co-authored-by: Rabi Panda <adnapibar@gmail.com>
2021-10-07 11:06:15 -07:00
Rishikesh Pasham 808dbfd2ec
Adding Security Reporting Instructions in README.md file Signed-off-by: Rishikesh Reddy Pasham rishireddy1159@gmail.com (#1326)
Signed-off-by: Rishikesh Pasham <rishireddy1159@gmail.com>
2021-10-02 10:41:30 -04:00
Andriy Redko d46c206f29
[BUG] ConcurrentSnapshotsIT#testAssertMultipleSnapshotsAndPrimaryFailOver fails intermittently (#1311)
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2021-09-30 12:44:29 -04:00
Andriy Redko 180db5cd09
Support for Heap after GC stats (correction after backport to 1.2.0) (#1315)
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2021-09-30 09:48:24 -04:00
Andriy Redko 80388a8a29
Support for Heap after GC stats (#1265)
* Support for Heap after GC stats

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* Addressing code review comments

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* Using the right version 2.0.0 (instead of 1.2.0) for the change

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2021-09-28 14:40:00 -04:00
Daniel Doubrovkine (dB.) 457c1cd6ec
fix typo (#1305)
Signed-off-by: sgbasaraner <sarpbasaraner@gmail.com>

Co-authored-by: Sarp Güney Başaraner <sarpbasaraner@gmail.com>
2021-09-28 13:54:35 -04:00
Shivansh Arora 416220f510
Making GeneralScriptException an Implementation of OpensearchWrapperException (#1066)
Signed-off-by: Shivansh Arora <hishiv@amazon.com>
2021-09-28 12:17:43 -04:00
Nick Knize 46e0f63539
[Tests] Translog Pruning tests to MetadataCreateIndexServiceTests (#1295)
This commit adds test coverage for translog pruning setting to
MetadataCreateIndexServiceTests

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-09-25 23:05:32 -05:00
Sai 29c88c6900
Rename translog pruning setting to CCR specific setting and addressed Bug in the test case (#1243)
* Rename translog pruing setting to CCR specific setting

Signed-off-by: Sai Kumar <karanas@amazon.com>

* Rename to index.plugins.replication.translog.retention_lease.pruning.enabled as
index settings needs "index." as prefix

Signed-off-by: Sai Kumar <karanas@amazon.com>

* Add deprecations to retention pruning controls

This commit adds deprecation flags to all added settings, variables, and methods
specific to ccr's retention lease pruning mechanism.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* Addressed CR comments

Signed-off-by: Sai Kumar <karanas@amazon.com>

* fix javadoc deprecation

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* fix deprecation tag in TranslogDeletionPolicy

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

* Addressed test issue under translog tests

Signed-off-by: Sai Kumar <karanas@amazon.com>

Co-authored-by: Nicholas Walter Knize <nknize@apache.org>
2021-09-24 16:09:59 -07:00
Xue Zhou 82d1d0ec08
fix gradle check fail due to renameing -min in #1094 (#1289)
Signed-off-by: Xue Zhou <xuezhou@amazon.com>
2021-09-24 12:06:16 -05:00
Tianli Feng 20728c3725
Add guide for generating code coverage report in TESTING.md (#1264)
Signed-off-by: Tianli Feng <ftianli@amazon.com>
2021-09-21 19:44:08 -04:00
Xue Zhou 0ab8e34022
Rename artifact produced by the build to include -min (#1251)
Signed-off-by: Xue Zhou <xuezhou@amazon.com>
2021-09-21 19:43:35 -04:00
Andriy Redko cdbc84f09d
Update Jackson to 2.12.5 (#1247)
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2021-09-21 18:33:20 -04:00
CEHENKLE e66b717c98
adding components to DEVELOPER_GUIDE (#1200)
* adding components to DEVELOPER_GUIDE

Signed-off-by: CEHENKLE <henkle@amazon.com>

* small tweaks

Signed-off-by: CEHENKLE <henkle@amazon.com>
2021-09-20 20:59:49 -07:00
Bukhtawar Khan 390e678f92
Handle shard over allocation during partial zone/rack or independent node failures (#1149)
The changes ensure that in the event of a partial zone failure, the surviving nodes in the minority zone don't get overloaded with shards, this is governed by a skewness limit.

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>
2021-09-20 10:32:23 -07:00
Bukhtawar Khan f7e2984248
Introduce FS Health HEALTHY threshold to fail stuck node (#1167)
This will cause the leader stuck on IO during publication to step down and eventually trigger a leader election.

Issue Description
---
The publication of cluster state is time bound to 30s by a cluster.publish.timeout settings. If this time is reached before the new cluster state is committed, then the cluster state change is rejected and the leader considers itself to have failed. It stands down and starts trying to elect a new master.

There is a bug in leader that when it tries to publish the new cluster state it first tries acquire a lock to flush the new state under a mutex to disk. The same lock is used to cancel the publication on timeout. Below is the state of the timeout scheduler meant to cancel the publication. So essentially if the flushing of cluster state is stuck on IO, so will the cancellation of the publication since both of them share the same mutex. So leader will not step down and effectively block the cluster from making progress.

Signed-off-by: Bukhtawar Khan <bukhtawa@amazon.com>
2021-09-16 17:02:25 -07:00
Andriy Redko b6c8bdf872
Drop mocksocket in favour of custom security manager checks (tests only) (#1205)
* Drop mocksocket in favour of custom security manager checks (tests only)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* Slightly relaxed host checks to allow all local addresses

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2021-09-16 17:21:47 -04:00
Nick Knize cbbf967d76
[Version] Add 1.2 for BWC testing (#1241)
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-09-15 09:07:53 -07:00
Tianli Feng 53d60b72b2
Fix defects in code-coverage.gralde to generate code coverage report properly (#1214)
* Refactor the logic to control the format for code coverage report and rename the system property

* Remove outdated code of giving JaCoCo files permission when Java security manager enabled

Signed-off-by: Tianli Feng <ftianli@amazon.com>
2021-09-13 10:56:11 -07:00
Nick Knize aecc7bd005
[Bug] Fix InstallPluginCommand to use proper key signatures (#1233)
The public key has changed since the initial release. This commit fixes the
public key and uses the .sig files that are published to the artifacts site.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-09-10 14:58:31 -07:00
Abbas Hussain 65abe4a45c
Generate release notes for 1.1 (#1230)
Signed-off-by: Abbas Hussain <abbas_10690@yahoo.com>
2021-09-09 00:51:53 +05:30
Rabi Panda 3870522228
Run link checker GitHub action on schedule. (#1221)
Most PRs do not add/update links, however sites go down often. This change makes sure that we catch any broken link in the repository and fix it, but at the same time we do not want to block PRs because of some unrelated broken links.

This PR updates the workflow to run everyday at midnight UTC.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-09-08 11:24:19 -07:00
Sai 7e440af9e4
Changes to support retrieval of operations from translog based on specified range (#1210)
* Changes to support retrieval of operations from translog based on specified range

Signed-off-by: Sai Kumar <karanas@amazon.com>

* Addressed CR comments

Signed-off-by: Sai Kumar <karanas@amazon.com>

* Added testcases for internal engine

Signed-off-by: Sai Kumar <karanas@amazon.com>
2021-09-08 20:22:24 +05:30
Andriy Redko 0b773c2cec
Fix org.opensearch.index.reindex.ReindexRestClientSslTests#testClientSucceedsWithCertificateAuthorities - javax.net.ssl.SSLPeerUnverifiedException (#1212)
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2021-09-08 09:08:05 -04:00
Andriy Redko ddb37dea32
Added Eclipse import instructions to DEVELOPER_GUIDE.md (#1215)
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2021-09-08 09:05:02 -04:00
Sai a9e1913bd5
Support for translog pruning based on retention leases (#1038)
* Support for translog pruning based on retention leases

Signed-off-by: Sai Kumar <karanas@amazon.com>

* Addressed CR Comments

Signed-off-by: Sai Kumar <karanas@amazon.com>

* Addressed test case issue

Signed-off-by: Sai Kumar <karanas@amazon.com>
2021-09-08 16:23:00 +05:30
Sai 919c5e05aa
Exclude failing links from plugins/modules (#1223)
Signed-off-by: Sai Kumar <karanas@amazon.com>
2021-09-08 13:39:53 +05:30
Sven R 268d93b375
FreeBSD Java support (#1014)
Signed-off-by: hackacad <admin@hackacad.net>
2021-09-07 18:17:58 -04:00
Vacha 83332c8ab6
Kept the original constructor for PluginInfo to maintain bwc (#1206) 2021-09-02 18:09:15 -07:00
Anas Alkouz f37f29c996
Add more instructions how to install/configure git secrets (#1202)
Signed-off-by: Anas Alkouz <aalkouz@amazon.com>

Co-authored-by: Anas Alkouz <aalkouz@amazon.com>
2021-09-02 17:24:12 -04:00
Nick Knize 4b31e1ba04
[Version] Increment main to 2.0 (#1192)
Increment version on main to 2.0.0.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Co-authored-by: dblock <dblock@amazon.com>
2021-09-01 22:35:53 -05:00
Abbas Hussain fa8126004c
Upgrade apache commons-compress to 1.21 (#1197)
Signed-off-by: Abbas Hussain <abbas_10690@yahoo.com>
2021-09-02 08:35:42 +05:30
Daniel Doubrovkine (dB.) ecd9875e7c
Clarify opensearch.version to not include -SNAPSHOT. (#1186)
Signed-off-by: dblock <dblock@dblock.org>
2021-09-01 21:54:41 -04:00
Daniel Doubrovkine (dB.) 20c6f786ac
Added all icu-project.org websites to the link checker exclusions. (#1201)
Signed-off-by: dblock <dblock@dblock.org>
2021-09-01 21:13:48 -04:00
Andriy Redko 6e199d223b
Restoring alpha/beta/rc version semantics (#1112)
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2021-09-01 18:22:20 -05:00
Tianli Feng 652cdbd5af
Clarify JDK requirement in the developer guide (#1153)
* Explicitly point out the JDK 8 requirement is for runtime, but not for compiling.
* Clarify the JAVAx_HOME env variables are for the "backwards compatibility test".
* Add explanation on how the backwards compatibility tests get the OpenSearch distributions for a specific version.
Signed-off-by: Tianli Feng <ftianli@amazon.com>
2021-09-01 14:15:22 -07:00
Nick Knize 3a7f0762ac
[Bug] Fix mixed cluster support for OpenSearch 2+ (#1191)
The version framework only added support for OpenSearch 1.x bwc with legacy
clusters. This commit adds support for v2.0 which will be the last version with
bwc support for legacy clusters (v7.10)

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-09-01 14:45:46 -05:00
Daniel Doubrovkine (dB.) f298a41e4b
Extract excludes into a file, fix the link checker by adding http://site.icu-project.org/. (#1189) 2021-08-31 18:20:48 -04:00
Andriy Redko f60d093c63
Drop mocksocket & securemock dependencies from sniffer and rest client (no needed) (#1174)
* Drop mocksocket & securemock dependencies from sniffer and rest client (not needed)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* Removing .gitignore

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2021-08-31 14:18:37 -04:00
Minal Shah a43e6cf424
Fixed copyright to OpenSearch (#1175) 2021-08-31 12:55:28 -04:00