Commit Graph

54948 Commits

Author SHA1 Message Date
Rabi Panda 864466cbdb
Update instructions on debugging OpenSearch. (#689) (#738)
Add clear instructions on how to run OpenSearch with debugging mode in IntelliJ. 
Fixed a few minor typos and grammars.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-05-20 17:09:59 -04:00
Daniel Doubrovkine (dB.) a8c5e71eb9
Create group settings with fallback. (#743) (#745)
* Create group settings with fallback.

Signed-off-by: dblock <dblock@amazon.com>

* Use protected fallbackSetting in Setting.

Signed-off-by: dblock <dblock@amazon.com>
2021-05-20 17:06:58 -04:00
Ankit Jain edbe5ae7e8
Make allocation decisions at node level first for pending task optimi… (#534) (#739)
* Make allocation decisions at node level first for pending task optimization

Signed-off-by: Ankit Jain <akjain@amazon.com>

* Addressing review comments

Signed-off-by: Ankit Jain <akjain@amazon.com>

* Fixing benchmark and adding debug mode tests

Signed-off-by: Ankit Jain <akjain@amazon.com>

* Fixing typo in previous commit

Signed-off-by: Ankit Jain <akjain@amazon.com>

* Moving test file to correct package

Signed-off-by: Ankit Jain <akjain@amazon.com>

* Addressing review comments

Signed-off-by: Ankit Jain <akjain@amazon.com>
2021-05-20 14:02:18 -07:00
Rabi Panda e90bde5a05
[CVE] Upgrade dependencies to mitigate CVEs (#657) (#737)
This PR upgrade the following dependencies to fix CVEs.

- commons-codec:1.12 (->1.13) apache/commons-codec@48b6157
- ant:1.10.8 (->1.10.9) https://ant.apache.org/security.html
- jackson-databind:2.10.4 (->2.11.0) FasterXML/jackson-databind#2589
- jackson-dataformat-cbor:2.10.4 (->2.11.0) https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-28491
- apache-httpclient:4.5.10 (->4.5.13) https://bugzilla.redhat.com/show_bug.cgi?id=CVE-2020-13956
- checkstyle:8.20 (->8.29) https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-10782
- junit:4.12 (->4.13.1) https://github.com/junit-team/junit4/security/advisories/GHSA-269g-pwp5-87pp
- netty:4.1.49.Final (->4.1.59) https://github.com/netty/netty/security/advisories/GHSA-5mcr-gq6c-3hq2

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-05-19 21:20:53 -07:00
Abbas Hussain b574acf6b9
Added a link to the maintainer file in contribution guides (#589) (#731)
Co-authored-by: Dawn Foster <fosterd@vmware.com>
2021-05-19 13:41:32 -07:00
amitai stern f4f5374592
Fix snapshot deletion task getting stuck in the event of exceptions (#629) (#650)
Changes the behavior of the recursive deletion function `executeOneStaleIndexDelete()` stop
condition to be when the queue of `staleIndicesToDelete` is empty -- also in the error flow.
Otherwise the GroupedActionListener never responds and in the event of a few exceptions the
deletion task gets stuck.
Alters the test case to fail to delete in bulk many snapshots at the first attempt, and then
the next successful deletion also takes care of the previously failed attempt as the test
originally intended.
SNAPSHOT threadpool is at most 5. So in the event we get more than 5 exceptions there are no
more threads to handle the deletion task and there is still one more snapshot to delete in the
queue. Thus, in the test I made the number of extra snapshots be one more than the max in the
SNAPSHOT threadpool.

Signed-off-by: AmiStrn <amitai.stern@logz.io>
2021-05-13 09:02:57 -05:00
Daniel Doubrovkine (dB.) b1b563dc39
Standardize int, long, double and float Setting constructors. (#665) 2021-05-06 21:17:49 +00:00
Daniel Doubrovkine (dB.) b948315633
Converted all .asciidoc to .md. (#658)
Signed-off-by: dblock <dblock@amazon.com>
2021-05-06 20:12:49 +00:00
Daniel Doubrovkine (dB.) 4d9356e185
Fix #649: Properly escape @ in JavaDoc. (#651)
Signed-off-by: dblock <dblock@amazon.com>
2021-05-06 20:12:22 +00:00
Daniel Doubrovkine (dB.) 4892a389a2
Make -Dtests.output=always actually work. (#648)
Signed-off-by: dblock <dblock@amazon.com>
2021-05-06 20:11:33 +00:00
Daniel Doubrovkine (dB.) 8bc0a54d20
[WIP] Developer guide updates (#595)
* Add detail on how to install Java.

Signed-off-by: dblock <dblock@amazon.com>

* There's no password requirement for the instance.

Signed-off-by: dblock <dblock@amazon.com>

* Explain how to listen on an external IP.

Signed-off-by: dblock <dblock@amazon.com>
2021-04-28 17:28:17 +00:00
Nick Knize 3bbb87cbbc
Speedup snapshot stale indices delete (#613) (#616)
Instead of snapshot delete of stale indices being a single threaded operation this commit makes 
it a multithreaded operation and delete multiple stale indices in parallel using SNAPSHOT 
threadpool's workers.

Signed-off-by: Piyush Daftary <piyush.besu@gmail.com>
2021-04-28 11:02:21 -05:00
Nick Knize f085aa8e11
Speedup lang-painless tests (#605) (#617)
Painless tests would previously create script engine for every single test method.
Now the tests that need to tweak script engine settings create a class
level fixture (BeforeClass/AfterClass) that is used across all the test
methods in that suite.

RegexLimitTests was split into two suites (limit=1 and limit=2) rather
than dynamically applying different settings.

C2 compiler is no longer needed for tests to be fast, instead tests run
faster with C1 only as expected, like the rest of the unit tests.

Signed-off-by: Robert Muir <rmuir@apache.org>
2021-04-28 11:01:39 -05:00
Nick Knize a76bbca887
Replace elastic.co with opensearch.org (#611) (#623)
Signed-off-by: Abbas Hussain <abbas_10690@yahoo.com>
2021-04-28 10:59:36 -05:00
Sarat Vemulapalli 0e9f74e35f
Renaming CPU architecture to have consistent naming (#612)
* Renaming CPU architecture from x86_64, aarch64 to x64, arm64

Signed-off-by: Sarat Vemulapalli <vemulapallisarat@gmail.com>
2021-04-26 09:40:09 -07:00
Nick Knize c5a3c3cb41
Update lucene version to 8.8.2 (#557)
This commit updates the codebase to the latest released version of Lucene.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-04-23 09:48:41 -05:00
Rabi Panda 821c44c440
Fix failing test caused by versioning change. (#598)
The change in 0ba0e7cc, introduced the issue where randomly selecting an incompatible version fails the test. It caused the filtering logic to incorrectly identify all ES 7.*.* versions as bad versions for joining which should not be the case.

Additionally, split the test into two separate tests where earlier only one of them was run at random.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-04-23 07:31:41 -07:00
Nick Knize 6e04778d0c
[Versioning] Fix Version.fromString logic for legacy version (#604)
This commit fixes the Version.fromString logic to identify legacy versions. It
also adds an optional "distribution" field to the MainRespose for OpenSearch
version 1.0.0+. Any preceeding versions that do not contain the distribution
label will be handeled as legacy versions appropriately.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-04-22 23:04:44 -05:00
Rabi Panda 3fede8be3c
Rename the distribution used in test clusters. (#603)
For test clusters, we are using the archive(zip, tar), so we rename the distribution accordingly.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-04-22 14:21:32 -07:00
Sarat Vemulapalli 9701b226d8
Updating the Ivy repository to point to real url for Releases (#602)
Signed-off-by: Sarat Vemulapalli <vemulapallisarat@gmail.com>
2021-04-22 10:22:50 -07:00
Tom "spot" Callaway b8f9469cc9
clean up rpm artifact naming (#590)
Signed-off-by: Tom spot Callaway <spotaws@amazon.com>
2021-04-21 11:45:53 -07:00
Daniel Doubrovkine (dB.) 97698e082c
Cleaned up developer guide, added TOC. (#572)
Signed-off-by: dblock <dblock@amazon.com>
2021-04-21 09:39:31 -07:00
Abbas Hussain 9d73fecbe7
Remove all elastic.co references from javadocs (#586)
Signed-off-by: Abbas Hussain <abbas_10690@yahoo.com>
2021-04-21 10:41:14 +05:30
Rabi Panda 163a51fc5a
Remove the oss string from OpenSearch distributions (#575)
This commit removes the 'oss' string which was a remnant of the predecessor distribution flavors. As OpenSearch has no flavors for distributions, we are removing this tag from all the distribution names.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-04-20 09:27:40 -07:00
Robert Muir 5474e8d094
Speed ups to test suite and precommit tasks. (#580)
Use C1 compiler only for short-lived tasks and unit test execution. Tone
down some of the slowest unit tests.

Signed-off-by: Robert Muir <rmuir@apache.org>
2021-04-20 09:02:45 -05:00
Tyler Ohlsen feb435aeff
Fix DCO CLI example in CONTRIBUTING.md (#576)
Signed-off-by: Tyler Ohlsen <ohltyler@amazon.com>
2021-04-19 14:15:55 -07:00
Nick Knize 0ba0e7cc26
[Versioning] Rebase to OpenSearch version 1.0.0 (#555)
This commit rebases the versioning to OpenSearch 1.0.0

Co-authored-by: Rabi Panda <adnapibar@gmail.com>

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-04-15 17:06:47 -05:00
Nick Knize 4dde0f2a3b
[DOCKER] add apt update to test fixture krb5kdc (#565)
* [DOCKER] add apt update to test fixture krb5kdc

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-04-15 16:06:21 -05:00
Robert Muir bdb7aade9a
TEST BUG: MergeSchedulerSettingsTests fails always on small machines (#559)
MergeSchedulerSettingsTests tweaks the `node.processors` setting: sets it
explicitly to values of `2` and `8`. On a machine with only `4` threads
(e.g. my 2-core thinkpad), the test fails, because it creates unexpected
warnings about `node.processors` being set higher than the number of
cpus.

The problem can be reproduced always, by pretending to be single core:
```
./gradlew ':server:test' --tests "org.opensearch.index.MergeSchedulerSettingsTests.testMaxThreadAndMergeCount" -Dtests.jvm.argline="-XX:ActiveProcessorCount=1"
```

Instead, allow the test to provoke these specific warnings.

Signed-off-by: Robert Muir <rmuir@apache.org>
2021-04-15 13:57:57 -05:00
Robert Muir 6aa3675e9d
Prevent setting maxParallelForks=0 on single-cpu machines (#558)
The maxParallelForks is currently numJavaCpus / 2, but this evaluates to
zero (illegal value) if numJavaCpus is 1 and breaks the build.

Remove large chunk of complex dead code above this calculation, as it is
always overwritten by the simple calculation anyway.

Closes #554

Signed-off-by: Robert Muir <rmuir@apache.org>
2021-04-14 19:17:58 -07:00
Rabi Panda 0a66f9ace1
[TEST] Fix failing packaging tests for OpenSearch distributions. (#541)
While creating the archives and packages, the build tries to copy the non-existent file `README.asciidoc` instead of `README.md`. Consequently, the packaging tests fail during verification time. This commit addresses the issue by fixing the name.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-04-14 07:02:25 -07:00
Jonah Kowall ff02542c8b
Update README with getting started (#549)
Update README with getting started instructions

Signed-off-by: Jonah Kowall <jkowall@kowall.net>
2021-04-14 08:19:50 -05:00
Nick Knize ee6d15e26a
[License] Add SPDX License Header to security policies (#531)
This commit adds the SPDX license header and modifications copyright to security
policy files.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-04-12 22:59:36 -05:00
Dawn Foster 3563b72c7e
Added a maintainers file (#523)
based on the maintainers list that CEHENKLE posted in the forums here: https://discuss.opendistrocommunity.dev/t/preparing-opensearch-and-opensearch-dashboards-for-release/5567 

Signed-off-by: Dawn M. Foster <fosterd@vmware.com>
2021-04-12 19:41:03 -07:00
Kyle Conroy d7926100d4
Remove extra greater-thans from README (#527)
Signed-off-by: Kyle Conroy <kyle@conroy.org>
2021-04-12 16:20:11 -05:00
Manivannan Giridharan 05167b1621
Update Developers Guide. (#522)
Fixes Git Clone Link

Signed-off-by: Giridharan Manivannan <giridar123@gmail.com>
2021-04-12 11:31:24 -07:00
CEHENKLE 9a0050c454 changed to point to open issues rather than the project board
Signed-off-by: CEHENKLE <henkle@amazon.com>
2021-04-11 11:09:03 -07:00
Rabi Panda 8727afbcd3
Use the correct domain to fix failing integration tests. (#519)
This commit fixes a renaming issue (opensearch.co -> opensearch.org) which was causing few integration test failures.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-04-10 09:42:39 -07:00
Rabi Panda 2a3ce0bb75
Fix rename issues and failing repository-hdfs tests. (#518)
This commit fixes some partial rename issues and as a result fixes the failing secure repository-hdfs tests.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-04-09 17:51:27 -07:00
Nick Knize f14bc54c48
[Rename] Remove final references to legacy keystore (#514)
This commit refactors a remaining reference to the legacy keystore. This is 
likely not used but left for posterity.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-04-09 19:09:02 -05:00
CEHENKLE b52ed285f7 changed ''' to >
Signed-off-by: CEHENKLE <henkle@amazon.com>
2021-04-09 16:44:23 -07:00
CEHENKLE a945b9fd60 wrong ticks
Signed-off-by: CEHENKLE <henkle@amazon.com>
2021-04-09 16:39:59 -07:00
CEHENKLE dca6697aae changed Apache to Apache 2.0. Numbered principles
Signed-off-by: CEHENKLE <henkle@amazon.com>
2021-04-09 16:36:53 -07:00
Nick Knize e7528db91d
[Rename] Update Vagrantfile (#515)
This commit updates the Vagrantfile to identify any modifications copyright 
to OpenSearch contributors and refactor legacy to opensearch.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-04-09 18:27:48 -05:00
Rabi Panda d89e721da1
Remove the references to xpack and elastic in tests. (#516)
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-04-09 15:21:58 -07:00
CEHENKLE 73fc1db773 fixed apache to apache 2.0
Signed-off-by: CEHENKLE <henkle@amazon.com>
2021-04-09 14:40:04 -07:00
Nick Knize 5118e064b5
[README] Remove stale information (#513)
This commit removes status updates from the README and includes proper links to
the project borad, issues, and PR tabs for tracking progress of the project.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-04-09 16:00:21 -05:00
Nick Knize 05a6fde676
Update Plugin Signing Key (#512)
This commit updates the public_key.asc for verifying the signature of plugins in
the InstallPluginCommand utility. The key details are as follows:

e-mail: opensearch-infra@amazon.com
key id: 0934A65836A51424

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-04-09 15:14:33 -05:00
Nick Knize 9168f1fb43
[License] Add SPDX and OpenSearch Modification license header (#509)
This commit adds the SPDX Apache-2.0 license header along with an additional
copyright header for all modifications.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-04-09 14:28:18 -05:00
Nick Knize 8f4b2d58d1
[License] Update SPDX License Header (#510)
This commit updates the SPDX License Header for all new files created by
OpenSearch contributors.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-04-09 13:49:56 -05:00