* Support for bwc tests for plugins
Signed-off-by: Vacha <vachshah@amazon.com>
* Adding support for restart upgrades for plugins bwc
Signed-off-by: Vacha <vachshah@amazon.com>
This change refactors the circular reference check in the Grok processor class
to use a formal depth-first traversal. It also includes a logic update to
prevent a stack overflow in one scenario and a check for malformed patterns.
This bugfix addresses CVE-2021-22144.
Signed-off-by: Kartik Ganesh <85275476+kartg@users.noreply.github.com>
* Part 1: Support for cancel_after_timeinterval parameter in search and msearch request
This commit introduces the new request level parameter to configure the timeout interval after which
a search request will be cancelled. For msearch request the parameter is supported both at parent
request and at sub child search requests. If it is provided at parent level and child search request
doesn't have it then the parent level value is set at such child request. The parent level msearch
is not used to cancel the parent request as it may be tricky to come up with correct value in cases
when child search request can have different runtimes
TEST: Added test for ser/de with new parameter
Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com>
* Part 2: Support for cancel_after_timeinterval parameter in search and msearch request
This commit adds the handling of the new request level parameter and schedule cancellation task. It
also adds a cluster setting to set a global cancellation timeout for search request which will be
used in absence of request level timeout.
TEST: Added new tests in SearchCancellationIT
Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com>
* Address Review feedback for Part 1
Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com>
* Address review feedback for Part 2
Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com>
* Update CancellableTask to remove the cancelOnTimeout boolean flag
Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com>
* Replace search.cancellation.timeout cluster setting with search.enforce_server.timeout.cancellation to control if cluster level cancel_after_time_interval should take precedence over request level cancel_after_time_interval value
Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com>
* Removing the search.enforce_server.timeout.cancellation cluster setting and just keeping search.cancel_after_time_interval setting with request level parameter taking the precedence.
Signed-off-by: Sorabh Hamirwasia <sohami.apache@gmail.com>
Co-authored-by: Sorabh Hamirwasia <hsorabh@amazon.com>
* add jacoco plugin
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* config aggregated jacoco report
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Skip generating report if no test in subproject
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Add jacoco plugin into root project
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* test aggregate code coverage report
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Can generate aggregated unit test coverage report
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Can generate aggregated test report with source file linked
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Some cleanup, but jacocoReport is not dependson Test
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* cleanup
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* get unit test code coverage report
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* change 'enabled == true' to 'enabled'
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* add a comment for selectedsubprojects
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* add tasks to generate code coverage report for unit test and integration test
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* fix typo in variable
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Correct the task to get codecoverage for unit test and integtest
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Make code coverage report task run after test
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* apply gradle configuration aciidance api
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* apply jacoco plugin in BuildPlugin in all case
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Put file path list of integration test exec data in a variable
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Apply gradle configuration aviodance api to register task instead of create task
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* merge 2 jacocoreport configurations
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Add some comments in gradle script
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* Attach code coverage report task to gralde check task
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* add a space
Signed-off-by: Tianli Feng <ftianli@amazon.com>
* get code coverage report after check task on demand and get report in html format on demand
Signed-off-by: Tianli Feng <ftianli@amazon.com>
On February 3 2021, JFrog [announced](https://jfrog.com/blog/into-the-sunset-bintray-jcenter-gocenter-and-chartcenter/) the shutdown of JCenter. Later on April 27 2021, an update was provided that the repository will only be read only and new package and versions are no longer accepted on JCenter. This means we should no longer use JCenter for our central artifacts repository.
This change replaces JCenter with Maven Central as per the Gradle recommendation - https://blog.gradle.org/jcenter-shutdown
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
This change fixes the issue where the sources and javadoc artifacts were not built and included with the publish.
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
_cat/master is a fundamental API to know the master instance in the cluster. Given RestClusterState is exempted from tripping already, doesn't make sense for RestMasterAction to trip
Signed-off-by: Bukhtawar Khan bukhtawa@amazon.com
Add validation to check for official plugins during the plugins installation task for the upgrade tool.
Signed-off-by: Vacha Shah <vachshah@amazon.com>
Excluding all the links from license and notice files to be avoided to be checked by link checker GitHub action.
Signed-off-by: Vacha Shah <vachshah@amazon.com>
Replace the docs links In scripts bin/opensearch-env and config/jvm.options, with OpenSearch docs links.
Signed-off-by: Poojita-Raj <poojiraj@amazon.com>
distribution/bin/opensearch-env always sources the environment from the default environment file /etc/default/opensearch. This is an issue if we want to run multiple instances of OpenSearch on the same host. This change lets users override the default behavior by not sourcing the default environment file in case OPENSEARCH_PATH_CONF is set.
Signed-off-by: Xue Zhou <xuezhou@amazon.com>
Currently, the permissions for opensearch logs are -rw-r-r-, which gives read access to anyone. This weak permission
structure can lead to leakage of any sensitive information (if published) in the logs. This commit restricts read
access with -rw-r--- permission.
Signed-off-by: Himanshu Setia <setiah@amazon.com>
This change adds the initial version of a new CLI tool `opensearch-upgrade` as part of the OpenSearch distribution. This tool is meant for assisting during an upgrade from an existing Elasticsearch v7.10.2/v6.8.0 node to OpenSearch. It automates the process of importing existing configurations and installing of core plugins.
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
* Adding broken links checker
Signed-off-by: Vacha Shah <vachshah@amazon.com>
* Adding exclusions for links
Signed-off-by: Vacha Shah <vachshah@amazon.com>
* Correcting broken link
Signed-off-by: Vacha Shah <vachshah@amazon.com>
* Removing the benchmarks link
Signed-off-by: Vacha Shah <vachshah@amazon.com>
* Lower build requirement from Java 14+ to Java 11+
Avoid use of -Werror -Xlint:all, which may change significantly across
java releases (new warnings could be added). Instead, just list the
warnings individually.
Workaround JDK 11 compiler bug (JDK-8209058) that only impacts test fixture
code in the build itself.
Signed-off-by: Robert Muir <rmuir@apache.org>
* Disable warning around -source 7 -release 7 for java version checker
The java version checker triggers some default warnings because it
targets java7:
```
> Task :distribution:tools:java-version-checker:compileJava FAILED
warning: [options] source value 7 is obsolete and will be removed in a future release
warning: [options] target value 7 is obsolete and will be removed in a future release
warning: [options] To suppress warnings about obsolete options, use -Xlint:-options.
error: warnings found and -Werror specified
```
Suppress this warning explicitly for this module.
Signed-off-by: Robert Muir <rmuir@apache.org>
* more java14 -> java11 cleanup
Signed-off-by: Robert Muir <rmuir@apache.org>
Co-authored-by: Robert Muir <rmuir@apache.org>
Fixes the cat.health yaml failures when running in a bwc mixed cluster with
legacy (pre 1.0.0) nodes.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
* Add "tagline" field back to "MainResponse" in sever side (not in rest-high-level-client side) that removed in PR #427 .
* Replace with a new tagline "The OpenSearch Project: https://opensearch.org/".
* Turn the tagline into a constant in server/src/main/java/org/opensearch/action/main/MainResponse.java.
Snapshots are published with a -latest flag into s3, this updates
the pattern to correctly point to -latest.
Signed-off-by: Marc Handalian <handalm@amazon.com>
This Change removes version.distribution when the version.number is
overridden with the cluster setting compatibility.override_main_response_version.
Signed-off-by: Marc Handalian <handalm@amazon.com>
This change updates DistributionDownloader by registering a second
repository containing release-candidate bundles. This repository will
only be checked if the release repository does not contain the requested
version.
This change also updates the snapshot repository to point to a new
snapshot repository.
Signed-off-by: Marc Handalian <handalm@amazon.com>
with Elasticsearch there were two flavours of the docker build: one for
the OSS build and one with X-Pack. accordingly, there were two
`log4j2.properties` files for the two cases and the `build.gradle`
picked up the correct one depending on the build.
commit 83e87f7e54 removed all non-OSS functionality from the build but
with that it also removed the logic which so far copied over the OSS
version of `log4j2.properties`.
what the diff doesn't show properly here is that `log4j2.properties` has
been deleted and `oss/log4j2.properties` has been moved into its place
without any changes to it.
this fixes#868
Signed-off-by: Ralph Ursprung <Ralph.Ursprung@avaloq.com>
This change adds a new cluster setting "compatibility.override_main_response_version"
that when enabled spoofs the version.number returned from MainResponse
for REST clients expecting legacy version 7.10.2.
Signed-off-by: Marc Handalian <handalm@amazon.com>
OpenSearch uses plugins to add new features. A plugin can be included in the distribution by default (as part of *modules* directory) or can be installed optionally from a plugin repository.
This change provides a separate space called *sandbox* inside OpenSearch for the community to easily experiment new ideas and innovate. Ideally, this is where an experimental feature will reside before it can be promoted to the *modules* directory. All the plugins in this module will only be included in the **snapshot distributions**. During assembling distributions, we will check if the following two conditions are met to include the sandbox modules,
* the distribution is a snapshot i.e. the build system property `build.snapshot` is set to true. We use this because it will prevent accidental bundling of these modules in a release distribution.
* the `enable.sandbox` system property is set to true. By default it is set to true. The purpose of adding this extra flag is that we can exclude the modules from snapshots if needed. For example, we may want to run performance tests on snapshots without the sandbox modules.
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
Vacha
kartg
Sorabh
Daniel Doubrovkine (dB.)
frotsch
Bukhtawar Khan
Tianli Feng
Rabi Panda
Sven R
xuezhou25
Vlad Rozov
Yevhen Tienkaiev
Poojita Raj
afazel
Himanshu Setia
Chang Liu
Abbas Hussain
Sooraj Sinha
Nick Knize
Marc Handalian
Rabi Panda
shwetathareja
Ralph Ursprung