Commit Graph

55103 Commits

Author SHA1 Message Date
Marc Handalian c91c29e5c2
Remove distribution from main response in compatibility mode (#898)
This Change removes version.distribution when the version.number is
overridden with the cluster setting compatibility.override_main_response_version.

Signed-off-by: Marc Handalian <handalm@amazon.com>
2021-06-30 17:36:21 -07:00
Vlad Rozov 362f116abe
Replace metadata keys in OpenSearchException during serialization and deserialization (#905)
Signed-off-by: Vlad Rozov <vrozov@users.noreply.github.com>
2021-06-30 11:39:26 -07:00
Marc Handalian 06228bc25b
Update DistributionDownloader to fetch snapshots and staging bundles. (#904)
This change updates DistributionDownloader by registering a second
repository containing release-candidate bundles.  This repository will
only be checked if the release repository does not contain the requested
version.

This change also updates the snapshot repository to point to a new
snapshot repository.

Signed-off-by: Marc Handalian <handalm@amazon.com>
2021-06-30 10:29:07 -07:00
Nick Knize ef1cbd9e3d
Version bump for 1.1 release (#772)
Bumps the minor to 1.1 for the 1.0 release.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-06-28 09:58:27 -05:00
Nick Knize 0317b604f6
[BUG] Fix versioning issues discovered through version bump (#884)
This commit fixes up the version framework to ensure a smooth version bump
across releases.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-06-25 16:53:06 -05:00
Tianli Feng 18625952a9
update external library 'pdfbox' version to 2.0.24 to reduce vulnerability (#883) 2021-06-25 13:18:15 -07:00
Ralph Ursprung 0079fde645
docker build: use OSS `log4j2.properties` (#878)
with Elasticsearch there were two flavours of the docker build: one for
the OSS build and one with X-Pack. accordingly, there were two
`log4j2.properties` files for the two cases and the `build.gradle`
picked up the correct one depending on the build.

commit 83e87f7e54 removed all non-OSS functionality from the build but
with that it also removed the logic which so far copied over the OSS
version of `log4j2.properties`.

what the diff doesn't show properly here is that `log4j2.properties` has
been deleted and `oss/log4j2.properties` has been moved into its place
without any changes to it.

this fixes #868

Signed-off-by: Ralph Ursprung <Ralph.Ursprung@avaloq.com>
2021-06-24 10:16:17 -07:00
Marc Handalian c1250c963d
Add cluster setting to spoof version number returned from MainResponse (#847)
This change adds a new cluster setting "compatibility.override_main_response_version"
that when enabled spoofs the version.number returned from MainResponse
for REST clients expecting legacy version 7.10.2.

Signed-off-by: Marc Handalian <handalm@amazon.com>
2021-06-22 10:42:23 -07:00
Rabi Panda ca564fd04f
Enable adding experimental features through sandbox modules (#691)
OpenSearch uses plugins to add new features. A plugin can be included in the distribution by default (as part of *modules* directory) or can be installed optionally from a plugin repository.

This change provides a separate space called *sandbox* inside OpenSearch for the community to easily experiment new ideas and innovate. Ideally, this is where an experimental feature will reside before it can be promoted to the *modules* directory. All the plugins in this module will only be included in the **snapshot distributions**. During assembling distributions, we will check if the following two conditions are met to include the sandbox modules,

* the distribution is a snapshot i.e. the build system property `build.snapshot` is set to true. We use this because it will prevent accidental bundling of these modules in a release distribution.
* the `enable.sandbox` system property is set to true. By default it is set to true. The purpose of adding this extra flag is that we can exclude the modules from snapshots if needed. For example, we may want to run performance tests on snapshots without the sandbox modules.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-06-22 11:45:18 -05:00
Marc Handalian c7e60b87f9
Add URL for lucene snapshots (#858)
Signed-off-by: Marc Handalian <handalm@amazon.com>
2021-06-22 11:40:53 -05:00
Daniel Doubrovkine (dB.) 6018a5d6d3
Updated READMEs on releasing, maintaining, admins and security. (#853)
* Updated READMEs on releasing, maintaining, admins and security.

Signed-off-by: dblock <dblock@amazon.com>

* Ensuing -> following.

Signed-off-by: dblock <dblock@amazon.com>
2021-06-17 18:03:11 -04:00
itiyamas 11babe7782
Decouple throttling limits for new and old indices. (#778)
* Decouple throttling limits for new and old indices.

Signed-off-by: Itiyama <itiyamas@amazon.com>

* Precommit fixes.

Signed-off-by: Itiyama <itiyamas@amazon.com>

* Review comments.

Signed-off-by: Itiyama <itiyamas@amazon.com>

* Review comments and test fix.

Signed-off-by: Itiyama <itiyamas@amazon.com>

* Checkstyle fixes.

Signed-off-by: Itiyama <itiyamas@amazon.com>

* Review comments.

Signed-off-by: Itiyama <itiyamas@amazon.com>
2021-06-17 10:58:50 +05:30
jayesh hathila 3cd4e7ff54
Verbose plugin not found exception (#849)
* Add Plugin name for verbose Plugin not found exception
* Make the plugin loading failure exception more verbose
* Throw Opensearch in place of RuntimeException for plugin load failure
* Nit fix, added ... to make logging standout

Signed-off-by: Jayesh Hathila <sharma.jayesh52@gmail.com>
2021-06-15 15:12:01 -04:00
Tianli Feng 110cef7882
Fix resource leak issues suggested by Amazon CodeGuru (#816)
* Address a kind of issue suggested by Amazon CodeGuru Reviewer:

* Add try-with-resources block to automatically close the resources after using to avoid resource leak, in `SymbolicLinkPreservingTarIT`, `LicenseAnalyzer`, `SymbolicLinkPreservingUntarTransform`, `ConcurrentSeqNoVersioningIT` in `VersionProperties`, `GeoFilterIT`, `XContentHelper`,  `Json` and `IndexShard` class

* Add try-finally block to close the resources after using to avoid resource leak, in `ServerChannelContext` class.

* Add try-catch block to close the resources when exception occurs in `FsBlobContainer` class (when XContentFactory.xContentType throws an exception).

* Close resources when assertion error occurs, in `ServerChannelContext` class.
2021-06-15 09:31:36 -07:00
Marc Handalian dff274414d
Add Snapshot maven repository (#829)
Signed-off-by: Marc Handalian <handalm@amazon.com>
2021-06-10 14:07:48 -07:00
Jesper Olsson e7812cd882
#699/first impression (#712)
* #699 Add logo

Signed-off-by: Jesper Olsson <software@jesperolsson.se>

* #699 Describe OpenSearch

Signed-off-by: Jesper Olsson <software@jesperolsson.se>

* #699 Reorganize content

Signed-off-by: Jesper Olsson <software@jesperolsson.se>

* #699 Simplify helping out

Signed-off-by: Jesper Olsson <software@jesperolsson.se>

* #699 Clarify forum

Signed-off-by: Jesper Olsson <software@jesperolsson.se>

* #699 Remove history

Signed-off-by: Jesper Olsson <software@jesperolsson.se>
2021-06-08 12:00:47 -04:00
Gregor Zurowski 4fff57014f
Remove `client/sniffer` from Javadoc exemption list (#818)
Signed-off-by: Gregor Zurowski <gregor@zurowski.org>
2021-06-04 17:15:08 -07:00
Peter Nied 74f29c7727
Fix arm architecture translation issue (#809)
Found when attempting to build on an `arm64` machine when I recieved an error message below.  Root cause is that string equality in java cannot be done with the `==` sign.

```
unknown architecture [arm64] for jdk [provisioned_runtime], must be one of [aarch64, x64]
```

Signed-off-by: Peter Nied <petern@amazon.com>
2021-06-04 12:26:46 -04:00
Gregor Zurowski 0f9060761c
Fix Javadoc errors in `client/sniffer` (#802)
* Adds a gradle plugin to validate missing javadocs

Use `./gradlew missingJavadoc` to validate missing javadocs.
Currently this task fails because several modules are missing
appropriate javadocs. Once added, this should pass.
Also, precommit PomValidation check currently fails with missing Javadoc
plugin, that needs to be fixed -
https://github.com/opensearch-project/OpenSearch/issues/449
Thus keeping this in a separate feature branch.

Signed-off-by: Himanshu Setia <setiah@amazon.com>

* Fix Javadoc errors in client/sniffer module

Signed-off-by: Gregor Zurowski <gregor@zurowski.org>

* Add package info to client/sniffer module

Signed-off-by: Gregor Zurowski <gregor@zurowski.org>

Co-authored-by: Himanshu Setia <setiah@amazon.com>
2021-06-03 17:08:16 -07:00
Daniel Doubrovkine (dB.) 6a8e303591
Removed pre-alpha notes. (#815)
Signed-off-by: dblock <dblock@amazon.com>
2021-06-03 14:50:59 -07:00
Daniel Doubrovkine (dB.) 3c92080ce4
Document running individual tests. (#741)
Signed-off-by: dblock <dblock@amazon.com>
2021-06-03 15:12:30 -04:00
Marc Handalian da78b889c8
Remove checks for legacy .yaml and .json config files. (#792)
Signed-off-by: Marc Handalian <handalm@amazon.com>
2021-06-03 15:09:57 -04:00
Daniel Doubrovkine (dB.) b3c8f71e2d
Remove reference to an EC2 instance type. (#812)
* Remove reference to an AWS instance type.

Signed-off-by: dblock <dblock@amazon.com>

* Move auto-TOC to top of doc.

Signed-off-by: dblock <dblock@amazon.com>
2021-06-03 12:01:09 -07:00
Rabi Panda 33708d29ab
Enable BWC checks (#796)
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-06-03 09:23:30 -07:00
Daniel Doubrovkine (dB.) bd9ca7cce5
Version checks are incorrectly returning versions < 1.0.0. (#797)
* Version checks are incorrectly returning versions < 1.0.0.

Signed-off-by: dblock <dblock@amazon.com>

* Removed V_7_10_3 which has not been released as of time of the fork.

Signed-off-by: dblock <dblock@amazon.com>

* Update check for current version to get unreleased versions.

- no unreleased version if the current version is "1.0.0"
- add unit tests for OpenSearch 1.0.0 with legacy ES versions.
- update VersionUtils to include all legacy ES versions as released.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>

Co-authored-by: Rabi Panda <adnapibar@gmail.com>
2021-06-02 16:03:50 -04:00
Sarat Vemulapalli e7d6dcd867
Adding release notes for 1.0.0-rc1 (#794)
Signed-off-by: Sarat Vemulapalli <vemulapallisarat@gmail.com>
2021-05-28 14:53:31 -04:00
Nick Knize d821b15a6b
[BWC] fix mixedCluster and rolling upgrades (#775)
This commit fixes mixedCluster and rolling upgrades by spoofing OpenSearch
version 1.0.0 as Legacy version 7.10.2. With this commit an OpenSearch 1.x node
can join a legacy (<= 7.10.2) cluster and rolling upgrades work as expected.
Mixed clusters will not work beyond the duration of the upgrade since shards
cannot be replicated from upgraded nodes to nodes running older versions.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

Co-authored-by: Shweta Thareja <tharejas@amazon.com>
2021-05-28 00:42:57 -05:00
Nick Knize c62884a1e9
[BUG] fix MainResponse to spoof version number for legacy clients (#708)
This commit changes MainResponse to spoof OpenSearch 1.x version numbers as
Legacy version number 7.10.2 for legacy clients.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-05-26 18:16:23 -05:00
Chloe 141d6b4e86
Add a method to use fallback setting to set the memory size (#755)
* Overwrote a memory size setting method to use fallback setting

Signed-off-by: chloe-zh <chloezh1102@gmail.com>
2021-05-26 11:09:01 -04:00
ashwinpankaj 002ce9eddf
An allocation constraint mechanism, that de-prioritizes nodes from getting picked for allocation if they breach certain constraints (#680)
* An allocation constraint mechanism, that de-prioritizes nodes from getting picked for allocation if they breach certain constraints

Signed-off-by: Ashwin Pankaj <appankaj@amazon.com>
2021-05-26 12:42:37 +05:30
Abbas Hussain 3e92821c82
[CVE] Upgrade dependencies for Azure related plugins to mitigate CVEs (#688)
* Update commons-io-2.4.jar to 2.7 for plugins/discovery-azure-classic module
* Remove unused jackson dependency and respective LICENSE and NOTICE
* Update guava dependency to mitigate CVE for repository-azure plugin

Signed-off-by: Abbas Hussain <abbas_10690@yahoo.com>
2021-05-26 03:27:36 +05:30
Neal Gompa (ニール・ゴンパ) 381b76eaa9
distribution/packages: Fix RPM architecture name for 64-bit x86 (#620)
RPM uses the "x86_64" name for 64-bit x86, which is in-line with GCC
and other compilers.

Signed-off-by: Neal Gompa <ngompa13@gmail.com>
2021-05-26 02:41:11 +05:30
Himanshu Setia 971adc646d
Enabling missingJavadoc validation in gradle check (#721)
Excludes modules currently failing from missingJavadoc check.
These modules can be removed from exclusion list once javadocs are added.
Currently, only :client:rest module is enabled for missingJavadoc check.
Issue #221 tracks javadocs for legacy code.

These checks are enabled as part of gradle check. Enabling it as part of
precommit needs more work and will be done later.

Once this PR is merged all incoming PRs will undergo javadoc validations.

Signed-off-by: Himanshu Setia <setiah@amazon.com>
2021-05-25 11:20:17 -07:00
Rabi Panda 59563fbf72
Support running elasticsearch-oss distribution in test cluster for BWC (#764)
We need to install elasticsearch in the test cluster nodes to be able to run the BWC tests. This commit makes the following changes to enable that,
- updates DistributionDownloadPlugin to download elasticsearch-oss distributions.
- updates OpenSearchNode to be able to run both OpenSearch and Elasticsearch distributions.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-05-25 11:16:12 -07:00
Rabi Panda 9c95cf424b
[Bug] Fix gradle build on Windows failing from a recent change (#758)
A recent change as part of the commit c2e816ec introduced a bug where the build is failing on Windows. The change was made to include the NOTICE.txt file as read-only in the distributions. The code fails on Windows as it's not a POSIX-compliant. This commit adds a check on the current operating system.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-05-24 11:15:06 -07:00
Mohit Godwani 4e8c92f2aa
Apply fix for health API response to distinguish no master (#656)
Signed-off-by: Mohit Godwani <mgodwan@amazon.com>
2021-05-24 13:05:55 -04:00
Daniel Doubrovkine (dB.) 658dc18b5f
Create group settings with fallback. (#743)
* Create group settings with fallback.

Signed-off-by: dblock <dblock@amazon.com>

* Use protected fallbackSetting in Setting.

Signed-off-by: dblock <dblock@amazon.com>
2021-05-20 16:02:48 -04:00
Dhwanil Patel a3f29b3875
Add timeout on cat/stats API (#552)
Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>
2021-05-20 13:30:32 +05:30
Ankit Jain e0c8b7ea27
Make allocation decisions at node level first for pending task optimi… (#534)
* Make allocation decisions at node level first for pending task optimization

Signed-off-by: Ankit Jain <akjain@amazon.com>

* Addressing review comments

Signed-off-by: Ankit Jain <akjain@amazon.com>

* Fixing benchmark and adding debug mode tests

Signed-off-by: Ankit Jain <akjain@amazon.com>

* Fixing typo in previous commit

Signed-off-by: Ankit Jain <akjain@amazon.com>

* Moving test file to correct package

Signed-off-by: Ankit Jain <akjain@amazon.com>

* Addressing review comments

Signed-off-by: Ankit Jain <akjain@amazon.com>
2021-05-19 19:09:30 -07:00
CEHENKLE cbcae73934
Update/maintainers.md (#723)
* adding four new committers.

Signed-off-by: CEHENKLE <henkle@amazon.com>

* adding Gopala, Vengad, Shweta and Itiyama to maintainers list

Signed-off-by: CEHENKLE <henkle@amazon.com>

* Fixing typo >.<

Signed-off-by: CEHENKLE <henkle@amazon.com>

* Fixing OTHER typo >.<

Signed-off-by: CEHENKLE <henkle@amazon.com>

* adding dB

Signed-off-by: CEHENKLE <henkle@amazon.com>
2021-05-19 13:16:42 -07:00
itiyamas c116062909
Decouples primaries_recoveries limit from concurrent recoveries limit. (#546)
* Decouples initial primaries limit from node concurrent limits.

Signed-off-by: Itiyama <itiyamas@amazon.com>

* Checkstyle fixes.

Signed-off-by: Itiyama <itiyamas@amazon.com>

* Checkstyle test fixes.

Signed-off-by: Itiyama <itiyamas@amazon.com>

* Review comments

Signed-off-by: Itiyama <itiyamas@amazon.com>

* Fixes review comments.

Signed-off-by: Itiyama <itiyamas@amazon.com>
2021-05-19 18:07:26 +05:30
Himanshu Setia 6f893ed1cd
Merging javadoc feature branch changes to main (#715)
* Adds a gradle plugin to validate missing javadocs

Use `./gradlew missingJavadoc` to validate missing javadocs.
Currently this task fails because several modules are missing
appropriate javadocs. Once added, this should pass.
Also, precommit PomValidation check currently fails with missing Javadoc
plugin, that needs to be fixed -
https://github.com/opensearch-project/OpenSearch/issues/449
Thus keeping this in a separate feature branch.

Signed-off-by: Himanshu Setia <setiah@amazon.com>

* Fix Javadoc errors in module `client/rest` (#685)

* Fix Javadoc errors in client/rest module

Signed-off-by: Gregor Zurowski <gregor@zurowski.org>

* Add package info file in client/rest module

Signed-off-by: Gregor Zurowski <gregor@zurowski.org>

* Fix typos

Signed-off-by: Gregor Zurowski <gregor@zurowski.org>

* Add exception documentation to Javadoc

Signed-off-by: Gregor Zurowski <gregor@zurowski.org>

* Fixes precommit task configuration failures due to newly added missin… (#707)

* Fixes precommit task configuration failures due to newly added missingJavadoc task

Signed-off-by: Himanshu Setia <setiah@amazon.com>

* Fixes javadoc task errors due to PR#685

Signed-off-by: Himanshu Setia <setiah@amazon.com>

* Updated CONTRIBUTING.md for info on javadocs

Signed-off-by: Himanshu Setia <setiah@amazon.com>

* Correcting licenses and naming

Signed-off-by: Himanshu Setia <setiah@amazon.com>

* Correcting version info

Signed-off-by: Himanshu Setia <setiah@amazon.com>

Co-authored-by: Gregor Zurowski <gregor@zurowski.org>
2021-05-18 13:21:41 -07:00
Rabi Panda 50abf6d066
[CVE] Upgrade dependencies to mitigate CVEs (#657)
This PR upgrade the following dependencies to fix CVEs.

- commons-codec:1.12 (->1.13) apache/commons-codec@48b6157
- ant:1.10.8 (->1.10.9) https://ant.apache.org/security.html
- jackson-databind:2.10.4 (->2.11.0) FasterXML/jackson-databind#2589
- jackson-dataformat-cbor:2.10.4 (->2.11.0) https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-28491
- apache-httpclient:4.5.10 (->4.5.13) https://bugzilla.redhat.com/show_bug.cgi?id=CVE-2020-13956
- checkstyle:8.20 (->8.29) https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-10782
- junit:4.12 (->4.13.1) https://github.com/junit-team/junit4/security/advisories/GHSA-269g-pwp5-87pp
- netty:4.1.49.Final (->4.1.59) https://github.com/netty/netty/security/advisories/GHSA-5mcr-gq6c-3hq2

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-05-18 11:37:24 -07:00
Harmish 6732239f1c
Add read_only block argument to opensearch-node unsafe-bootstrap command (#599)
* apply user defined cluster wide read only block after unsafe bootstrap command

Signed-off-by: Harmish Lakhani <harmish.lakhani@gmail.com>
2021-05-18 10:15:37 -04:00
Ketan Verma ce12097095
Support Data Streams in OpenSearch (#690)
This commit adds support for data streams by adding a DataStreamFieldMapper, and making timestamp 
field name configurable. Backwards compatibility is supported.

Signed-off-by: Ketan Verma <ketan9495@gmail.com>
2021-05-17 09:09:25 -05:00
Rabi Panda c2e816ecf6
[TEST] Fix failing distro tests for linux packages (#569)
Changes to fix the failing OpenSearch distribution tests for packages (linux-archive, linux-archive-aarch64, debian, rpm, docker) on supported linux distros. 

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-05-15 21:45:48 -07:00
Neal Gompa (ニール・ゴンパ) 4d88210c12
distribution/packages: Fix filename format for deb archives (#621)
Debian packages are formatted with the following filename structure:

name_[epoch:]version-release_arch.deb

Make generated Debian packages follow this convention.

Signed-off-by: Neal Gompa <ngompa13@gmail.com>
2021-05-13 15:14:02 -07:00
Rabi Panda 943c778a7f
[CVE-2018-11765] Upgrade hadoop dependencies for hdfs plugin (#654)
Hadoop 2.8.5 has been reported to have CVEs (https://bugzilla.redhat.com/show_bug.cgi?id=1883549). We need to upgrade this to 2.10.1. This also updates the hadoop-minicluster version to 2.10.1 as well. This upgrade also brings in two additional dependencies, woodstox-core and stax2-api that are added along with the sha1s, licenses and notices.

Also upgrade guava to the latest as per the CVE https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-8908

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-05-13 14:56:47 -07:00
Rabi Panda 6550e099b3
[CVE-2020-7692] Upgrade google-oauth clients for goolge cloud plugins (#662)
For discovery-gce and repository-gcs plugins update the google-oauth-client library to version 1.31.0. See CVE details at https://nvd.nist.gov/vuln/detail/CVE-2020-7692

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-05-13 12:19:57 -07:00
Sarat Vemulapalli 053b793695
Updating README and CONTRIBUTING guide to get ready for beta1 release. (#672)
* Updating README and CONTRIBUTING guide to get ready for beta1 release.

Signed-off-by: Sarat Vemulapalli <vemulapallisarat@gmail.com>

* Addressing comments.

Signed-off-by: Sarat Vemulapalli <vemulapallisarat@gmail.com>
2021-05-13 10:23:27 -07:00