54968 Commits

Author SHA1 Message Date
Minal Shah
051aa13dab
Backporting the fix to 1.x for fixing Copyright licensing (#1188)
Signed-off-by: Minal Shah <minalsha@amazon.com>
2021-08-31 19:38:56 -04:00
Abbas Hussain
9b58ef5350
Reduce iterations to improve test run time (#1168) (#1177)
Signed-off-by: Abbas Hussain <abbas_10690@yahoo.com>
2021-08-31 01:06:29 +05:30
Abbas Hussain
04d56cf676
Tune datanode count and shards count to improve test run time (#1170) (#1176)
Signed-off-by: Abbas Hussain <abbas_10690@yahoo.com>
2021-08-31 00:14:38 +05:30
Nick Knize
079fd77062
Add 1.0.1 revision (#1152) (#1160)
This commit stages the branch to the next 1.0.1 patch release. BWC testing needs
this even if the next revision is never actually released.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-08-26 07:10:24 -05:00
Nick Knize
ec05dcdcf0
[Bug] Change 1.0.0 version check in PluginInfo (#1159)
PluginInfo should use .onOrAfter(Version.V_1_1_0) instead of
.after(Version.V_1_0_0) for the new custom folder name for plugin feature.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-08-25 23:58:30 -05:00
Rabi Panda
07a57d9bbb
Include sources and javadoc artifacts while publishing to a Maven repository (#1049) (#1139)
This change fixes the issue where the sources and javadoc artifacts were not built and included with the publish.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-08-23 17:12:20 -07:00
Vacha
f73e66e5e7
Allowing custom folder name for plugin installation (#848) (#1116)
Signed-off-by: Vacha Shah <vachshah@amazon.com>
2021-08-23 14:39:27 -07:00
Nick Knize
e153629871
Upgrade to Lucene 8.9 (#1080) (#1115)
This commit upgrades to the official lucene 8.9 release

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-08-23 10:06:34 -07:00
Nick Knize
c15e687bf9
[DEPRECATE] SimpleFS in favor of NIOFS (#1073) (#1114)
Lucene 9 removes support for SimpleFS File System format. This commit deprecates
the SimpleFS format in favor of NIOFS.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-08-20 11:26:22 -05:00
Sorabh
55c1026bb6
Fix failure in SearchCancellationIT.testMSearchChildReqCancellationWithHybridTimeout (#1105)
In some cases as one shared with issue #1099, the maxConcurrentSearchRequests was chosen as 0 which
will compute the final value during execution of the request based on processor counts. When this
computed value is less than number of search request in msearch request, it will execute all the
requests in multiple iterations causing the failure since test will only wait for one such
iteration. Hence setting the maxConcurrentSearchRequests explicitly to number of search requests
being added in the test to ensure correct behavior

Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com>
2021-08-17 16:23:55 -04:00
Vacha
8af1b91b7b
Support for bwc tests for plugins (#1051) (#1090)
* Support for bwc tests for plugins

Signed-off-by: Vacha <vachshah@amazon.com>

* Adding support for restart upgrades for plugins bwc

Signed-off-by: Vacha <vachshah@amazon.com>
2021-08-15 08:07:55 -07:00
kartg
f151bfff24
Improving the Grok circular reference check to prevent stack overflow (#1079) (#1087)
This change refactors the circular reference check in the Grok processor class
to use a formal depth-first traversal. It also includes a logic update to
prevent a stack overflow in one scenario and a check for malformed patterns.
This bugfix addresses CVE-2021-22144.

Signed-off-by: Kartik Ganesh <85275476+kartg@users.noreply.github.com>
2021-08-12 18:47:56 -04:00
Sorabh
e44e890d88
Part 1: Support for cancel_after_timeinterval parameter in search and msearch request (#986) (#1085)
* Part 1: Support for cancel_after_timeinterval parameter in search and msearch request

This commit introduces the new request level parameter to configure the timeout interval after which
a search request will be cancelled. For msearch request the parameter is supported both at parent
request and at sub child search requests. If it is provided at parent level and child search request
doesn't have it then the parent level value is set at such child request. The parent level msearch
is not used to cancel the parent request as it may be tricky to come up with correct value in cases
when child search request can have different runtimes

TEST: Added test for ser/de with new parameter

Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com>

* Part 2: Support for cancel_after_timeinterval parameter in search and msearch request

This commit adds the handling of the new request level parameter and schedule cancellation task. It
also adds a cluster setting to set a global cancellation timeout for search request which will be
used in absence of request level timeout.

TEST: Added new tests in SearchCancellationIT
Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com>

* Address Review feedback for Part 1

Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com>

* Address review feedback for Part 2

Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com>

* Update CancellableTask to remove the cancelOnTimeout boolean flag

Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com>

* Replace search.cancellation.timeout cluster setting with search.enforce_server.timeout.cancellation to control if cluster level cancel_after_time_interval should take precedence over request level cancel_after_time_interval value

Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com>

* Removing the search.enforce_server.timeout.cancellation cluster setting and just keeping search.cancel_after_time_interval setting with request level parameter taking the precedence.

Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com>

Co-authored-by: Sorabh Hamirwasia <hsorabh@amazon.com>

Co-authored-by: Sorabh Hamirwasia <hsorabh@amazon.com>
2021-08-12 13:52:28 -04:00
frotsch
987bfcf89c
Avoid crashing on using the index.lifecycle.name in the API body (#1060) (#1070)
* Avoid crashing on using the index.lifecycle.name in the API body

Signed-off-by: frotsch <frotsch@mailbox.org>
2021-08-10 14:16:44 -04:00
Vlad Rozov
a7ad3f7026
Introduce RestHandler.Wrapper to help with delegate implementations (#1004) (#1031)
Signed-off-by: Vlad Rozov <vrozov@users.noreply.github.com>
2021-08-03 09:02:40 -04:00
Yevhen Tienkaiev
a936e1e904
Rank feature - unknown field linear (#983) (#1025)
Signed-off-by: Yevhen Tienkaiev <hronom@gmail.com>
2021-07-30 15:17:47 -04:00
Poojita Raj
91248284c9
Replace Elasticsearch docs links in scripts (#994) (#1001)
Replace the docs links In scripts bin/opensearch-env and config/jvm.options, with OpenSearch docs links.

Signed-off-by: Poojita-Raj <poojiraj@amazon.com>
(cherry picked from commit 6bc4ce017ad654cc2c8d7d37553c82d61c61b964)
Signed-off-by: Poojita-Raj <poojiraj@amazon.com>
2021-07-23 14:21:31 -07:00
Chang Liu
96dabb31d3
Introduce replaceRoutes() method and 2 new constructors to RestHandler.java (#947) (#998)
* Add addRoutesPrefix() method to RestHandler.java

Signed-off-by: Azar Fazel <azar.fazel@gmail.com>
Signed-off-by: cliu123 <lc12251109@gmail.com>

Co-authored-by: afazel <afazel@users.noreply.github.com>
2021-07-22 14:26:16 -04:00
Chang Liu
34def5894e
Avoid override of routes() in BaseRestHandler to respect the default behavior defined in RestHandler (#889) (#991)
Signed-off-by: cliu123 <lc12251109@gmail.com>
2021-07-22 10:57:18 -04:00
Daniel Doubrovkine (dB.)
1740517905
Cleanup TESTING and DEVELOPER_GUIDE markdowns (#946) (#954) 2021-07-13 14:13:26 -05:00
Daniel Doubrovkine (dB.)
c63071b00f
Updated READMEs on releasing, maintaining, admins and security. (#853) (#950)
Signed-off-by: dblock <dblock@amazon.com>
2021-07-12 15:06:20 -05:00
Sooraj Sinha
2085fbcad6
Pass interceptor to super constructor (#876) (#937)
Signed-off-by: Sooraj Sinha <soosinha@amazon.com>
2021-07-12 11:48:09 -07:00
Marc Handalian
f0a2c4f038
Add Snapshot maven repository (#941)
* backport PR #829 to 1.x branch, so that we can publish snapshots to our maven repository.
Signed-off-by: Marc Handalian <handalm@amazon.com>
2021-07-07 11:00:05 -07:00
Nick Knize
da6455d567
[BUG] Fix cat.health test failures in pre 1.0.0 mixed cluster test (#928) (#935)
Fixes the cat.health yaml failures when running in a bwc mixed cluster with
legacy (pre 1.0.0) nodes.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-07-02 17:55:58 -05:00
Rabi Panda
5d07c256f9
Fix arm architecture translation issue (#809) (#927)
Found when attempting to build on an `arm64` machine when I recieved an error message below.  Root cause is that string equality in java cannot be done with the `==` sign.

```
unknown architecture [arm64] for jdk [provisioned_runtime], must be one of [aarch64, x64]
```

Signed-off-by: Peter Nied <petern@amazon.com>

Co-authored-by: Peter Nied <petern@amazon.com>
2021-07-02 17:19:44 -05:00
Marc Handalian
3c563fedd9
Update DistributionDownloader to support fetching arm64 bundles. (#929) (#934)
Signed-off-by: Marc Handalian <handalm@amazon.com>
2021-07-02 15:02:47 -07:00
Rabi Panda
1f7df7adac
Update favicon for OpenSearch (#933)
Signed-off-by: Rabi Panda <pandarab@amazon.com>
2021-07-02 14:39:32 -07:00
Tianli Feng
b383355f50
Refresh OpenSearch nodes version in cluster state after upgrade (#865) (#926)
Signed-off-by: Shweta Thareja <tharejas@amazon.com>
Co-authored-by: shwetathareja <shwetathareja@live.com>
2021-07-02 09:50:42 -07:00
Marc Handalian
36401d9edb
Fix Snapshot pattern in DistributionDownloader. (#916) (#920)
Snapshots are published with a -latest flag into s3, this updates
the pattern to correctly point to -latest.

Signed-off-by: Marc Handalian <handalm@amazon.com>
2021-07-01 21:59:41 -05:00
Marc Handalian
b749b38043
Remove distribution from main response in compatibility mode (#898) (#921)
This Change removes version.distribution when the version.number is
overridden with the cluster setting compatibility.override_main_response_version.

Signed-off-by: Marc Handalian <handalm@amazon.com>
2021-07-01 21:57:36 -05:00
Tianli Feng
230f90b554
Add 'tagline' back to MainResponse in server that was removed in PR #427 (#913) (#914)
* Add "tagline" field back to "MainResponse" in sever side (not in rest-high-level-client side) that removed in PR #427 .
* Replace with a new tagline "The OpenSearch Project: https://opensearch.org/".
* Turn the tagline into a constant in server/src/main/java/org/opensearch/action/main/MainResponse.java.
2021-07-01 19:45:28 -07:00
Vacha
3ed6a2ec2a
Replace metadata keys in OpenSearchException during serialization and deserialization (#905) (#910)
Signed-off-by: Vlad Rozov <vrozov@users.noreply.github.com>

Co-authored-by: Vlad Rozov <vrozov@users.noreply.github.com>
2021-06-30 14:45:23 -07:00
Marc Handalian
af4ad4fe8d
Update DistributionDownloader to fetch snapshots and staging bundles. (#904) (#907)
This change updates DistributionDownloader by registering a second
repository containing release-candidate bundles.  This repository will
only be checked if the release repository does not contain the requested
version.

This change also updates the snapshot repository to point to a new
snapshot repository.

Signed-off-by: Marc Handalian <handalm@amazon.com>
2021-06-30 11:32:55 -07:00
Nick Knize
15e6cb4db5
Version bump for 1.1 release (#772) (#892)
Bumps the minor to 1.1 for the 1.0 release.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-06-28 12:57:03 -05:00
Marc Handalian
f5199ee026
Add URL for lucene snapshots (#888)
Signed-off-by: Marc Handalian <handalm@amazon.com>
2021-06-28 09:58:54 -05:00
Tianli Feng
6e5e1aeeec
update external library 'pdfbox' version to 2.0.24 to reduce vulnerability (#883) (#886) 2021-06-25 16:04:28 -07:00
Nick Knize
1a4c2915a3
[BUG] Fix versioning issues discovered through version bump (#884) (#887)
This commit fixes up the version framework to ensure a smooth version bump
across releases.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-06-25 17:08:37 -05:00
Marc Handalian
a5fc4d4262
Add cluster setting to spoof version number returned from MainResponse (#847) (#870)
This change adds a new cluster setting "compatibility.override_main_response_version"
that when enabled spoofs the version.number returned from MainResponse
for REST clients expecting legacy version 7.10.2.

Signed-off-by: Marc Handalian <handalm@amazon.com>
2021-06-22 16:59:25 -05:00
Tianli Feng
b1aa3671b5
Fix resource leak issues suggested by Amazon CodeGuru (#816) (#859)
* Add try-with-resources block to automatically close the resources after using to avoid resource leak, in `SymbolicLinkPreservingTarIT`, `LicenseAnalyzer`, `SymbolicLinkPreservingUntarTransform`, `ConcurrentSeqNoVersioningIT` in `VersionProperties`, `GeoFilterIT`, `XContentHelper`,  `Json` and `IndexShard` class

* Add try-finally block to close the resources after using to avoid resource leak, in `ServerChannelContext` class.

* Add try-catch block to close the resources when exception occurs in `FsBlobContainer` class (when XContentFactory.xContentType throws an exception).

* Close resources when assertion error occurs, in `ServerChannelContext` class.
2021-06-22 10:19:01 -07:00
itiyamas
006d04a60f
Decouple throttling limits for new and old indices. (#778) (#863)
* Decouple throttling limits for new and old indices.

Signed-off-by: Itiyama <itiyamas@amazon.com>

* Precommit fixes.

Signed-off-by: Itiyama <itiyamas@amazon.com>

* Review comments.

Signed-off-by: Itiyama <itiyamas@amazon.com>

* Review comments and test fix.

Signed-off-by: Itiyama <itiyamas@amazon.com>

* Checkstyle fixes.

Signed-off-by: Itiyama <itiyamas@amazon.com>

* Review comments.

Signed-off-by: Itiyama <itiyamas@amazon.com>
2021-06-17 14:39:49 +05:30
itiyamas
fe40f8f360
Decouples primaries_recoveries limit from concurrent recoveries limit. (#546) (#862)
* Decouples initial primaries limit from node concurrent limits.

Signed-off-by: Itiyama <itiyamas@amazon.com>

* Checkstyle fixes.

Signed-off-by: Itiyama <itiyamas@amazon.com>

* Checkstyle test fixes.

Signed-off-by: Itiyama <itiyamas@amazon.com>

* Review comments

Signed-off-by: Itiyama <itiyamas@amazon.com>

* Fixes review comments.

Signed-off-by: Itiyama <itiyamas@amazon.com>
2021-06-17 13:45:37 +05:30
jayesh hathila
f3d5340b33
Add Plugin name for verbose Plugin not found exception (#841)
* Add Plugin name for verbose Plugin not found exception
* Make the plugin loading failure exception more verbose
* Throw Opensearch in place of RuntimeException for plugin load failure

Signed-off-by: Jayesh Hathila <sharma.jayesh52@gmail.com>
2021-06-15 12:09:41 -04:00
Mohit Godwani
733d9c6f4b
Apply fix for health API response to distinguish no master (#819)
Signed-off-by: Mohit Godwani <mgodwan@amazon.com>
2021-06-04 11:28:57 +05:30
ashwinpankaj
b4c697a29d
[1.x] An allocation constraint mechanism, that de-prioritizes nodes from getting picked for allocation if they breach certain constraints (#777)
* An allocation constraint mechanism, that de-prioritizes nodes from getting picked for allocation if they breach certain constraints

Signed-off-by: Ashwin Pankaj <appankaj@amazon.com>

* Precommit fixes

Signed-off-by: Ashwin Pankaj <appankaj@amazon.com>

* Fix license header in test file

Signed-off-by: Ashwin Pankaj <appankaj@amazon.com>

* Review comments

Signed-off-by: Ashwin Pankaj <appankaj@amazon.com>

Co-authored-by: Pankaj <appankaj@88665a1e205d.ant.amazon.com>
2021-06-04 10:47:47 +05:30
Rabi Panda
4f5b48f722
Enable BWC checks (#796) (#811)
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-06-03 15:09:12 -04:00
Daniel Doubrovkine (dB.)
b45b613472
Version checks are incorrectly returning versions < 1.0.0. (#797) (#807)
* Version checks are incorrectly returning versions < 1.0.0.
* Removed V_7_10_3 which has not been released as of time of the fork.
* Update check for current version to get unreleased versions.

- no unreleased version if the current version is "1.0.0"
- add unit tests for OpenSearch 1.0.0 with legacy ES versions.
- update VersionUtils to include all legacy ES versions as released.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
Signed-off-by: dblock <dblock@amazon.com>
Co-authored-by: Rabi Panda <adnapibar@gmail.com>
2021-06-03 09:56:33 -04:00
Nick Knize
403c511c17
[BUG] fix MainResponse to spoof version number for legacy clients (#708)
This commit changes MainResponse to spoof OpenSearch 1.x version numbers as
Legacy version number 7.10.2 for legacy clients.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-05-28 10:05:33 -05:00
Nick Knize
053ed7b66d
[BWC] fix mixedCluster and rolling upgrades (#775) (#793)
This commit fixes mixedCluster and rolling upgrades by spoofing OpenSearch
version 1.0.0 as Legacy version 7.10.2. With this commit an OpenSearch 1.x node
can join a legacy (<= 7.10.2) cluster and rolling upgrades work as expected.
Mixed clusters will not work beyond the duration of the upgrade since shards
cannot be replicated from upgraded nodes to nodes running older versions.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

Co-authored-by: Shweta Thareja <tharejas@amazon.com>
2021-05-28 08:02:43 -07:00
Rabi Panda
235cf29f90 Revert "Initial boilerplate for implementing the upgrade cli."
This reverts commit 3692d67c

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-05-26 11:54:59 -07:00
Rabi Panda
3692d67c1e Initial boilerplate for implementing the upgrade cli.
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-05-26 11:45:16 -07:00