Commit Graph

1933 Commits

Author SHA1 Message Date
Andriy Redko b6c8bdf872
Drop mocksocket in favour of custom security manager checks (tests only) (#1205)
* Drop mocksocket in favour of custom security manager checks (tests only)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* Slightly relaxed host checks to allow all local addresses

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2021-09-16 17:21:47 -04:00
Nick Knize 4b31e1ba04
[Version] Increment main to 2.0 (#1192)
Increment version on main to 2.0.0.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Co-authored-by: dblock <dblock@amazon.com>
2021-09-01 22:35:53 -05:00
Abbas Hussain fa8126004c
Upgrade apache commons-compress to 1.21 (#1197)
Signed-off-by: Abbas Hussain <abbas_10690@yahoo.com>
2021-09-02 08:35:42 +05:30
Daniel Doubrovkine (dB.) ecd9875e7c
Clarify opensearch.version to not include -SNAPSHOT. (#1186)
Signed-off-by: dblock <dblock@dblock.org>
2021-09-01 21:54:41 -04:00
Andriy Redko f60d093c63
Drop mocksocket & securemock dependencies from sniffer and rest client (no needed) (#1174)
* Drop mocksocket & securemock dependencies from sniffer and rest client (not needed)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* Removing .gitignore

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
2021-08-31 14:18:37 -04:00
Minal Shah a43e6cf424
Fixed copyright to OpenSearch (#1175) 2021-08-31 12:55:28 -04:00
Nick Knize 5ae00456a0
Upgrade to Lucene 8.9 (#1080)
This commit upgrades to the official lucene 8.9 release

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-08-20 11:28:06 -05:00
Vacha 072ccda932
Support for bwc tests for plugins (#1051)
* Support for bwc tests for plugins

Signed-off-by: Vacha <vachshah@amazon.com>

* Adding support for restart upgrades for plugins bwc

Signed-off-by: Vacha <vachshah@amazon.com>
2021-08-12 13:28:08 -07:00
Tianli Feng ca3ced981f
Add support to generate code coverage report with JaCoCo (#971)
* add jacoco plugin

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* config aggregated jacoco report

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* Skip generating report if no test in subproject

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* Add jacoco plugin into root project

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* test aggregate code coverage report

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* Can generate aggregated unit test coverage report

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* Can generate aggregated test report with source file linked

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* Some cleanup, but jacocoReport is not dependson Test

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* cleanup

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* get unit test code coverage report

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* change 'enabled == true' to 'enabled'

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* add a comment for selectedsubprojects

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* add tasks to generate code coverage report for unit test and integration test

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* fix typo in variable

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* Correct the task to get codecoverage for unit test and integtest

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* Make code coverage report task run after test

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* apply gradle configuration aciidance api

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* apply jacoco plugin in BuildPlugin in all case

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* Put file path list of integration test exec data in a variable

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* Apply gradle configuration aviodance api to register task instead of create task

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* merge 2 jacocoreport configurations

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* Add some comments in gradle script

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* Attach code coverage report task to gralde check task

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* add a space

Signed-off-by: Tianli Feng <ftianli@amazon.com>

* get code coverage report after check task on demand and get report in html format on demand

Signed-off-by: Tianli Feng <ftianli@amazon.com>
2021-08-10 07:23:46 -07:00
Rabi Panda e78040aec4
Replace JCenter with Maven Central. (#1057)
On February 3 2021, JFrog [announced](https://jfrog.com/blog/into-the-sunset-bintray-jcenter-gocenter-and-chartcenter/) the shutdown of JCenter. Later on April 27 2021, an update was provided that the repository will only be read only and new package and versions are no longer accepted on JCenter.  This means we should no longer use JCenter for our central artifacts repository.

This change replaces JCenter with Maven Central as per the Gradle recommendation - https://blog.gradle.org/jcenter-shutdown

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-08-09 17:07:43 -07:00
Sven R dcd9cef56c
alt bash path support (#1047)
Signed-off-by: hackacad <admin@hackacad.net>
2021-08-06 11:09:29 -04:00
Rabi Panda 7469bf6e26
Include sources and javadoc artifacts while publishing to a Maven repository (#1049)
This change fixes the issue where the sources and javadoc artifacts were not built and included with the publish.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-08-05 16:43:31 -07:00
xuezhou25 7d18eaf3a4
Replaced custom built JNA by official JNA distribution. (#1003)
Signed-off-by: Xue Zhou <xuezhou@amazon.com>
2021-08-03 13:52:53 -04:00
Vacha 19e54d6624
Allowing custom folder name for plugin installation (#848)
Signed-off-by: Vacha Shah <vachshah@amazon.com>
2021-07-22 04:28:49 +05:30
Daniel Doubrovkine (dB.) 88f17b1822
Cleanup TESTING and DEVELOPER_GUIDE markdowns (#946) 2021-07-12 15:02:56 -05:00
Daniel Doubrovkine (dB.) 5d020170ff
Lower build requirement from Java 14+ to Java 11+ (#940)
* Lower build requirement from Java 14+ to Java 11+

Avoid use of -Werror -Xlint:all, which may change significantly across
java releases (new warnings could be added). Instead, just list the
warnings individually.

Workaround JDK 11 compiler bug (JDK-8209058) that only impacts test fixture
code in the build itself.

Signed-off-by: Robert Muir <rmuir@apache.org>

* Disable warning around -source 7 -release 7 for java version checker

The java version checker triggers some default warnings because it
targets java7:

```
> Task :distribution:tools:java-version-checker:compileJava FAILED
warning: [options] source value 7 is obsolete and will be removed in a future release
warning: [options] target value 7 is obsolete and will be removed in a future release
warning: [options] To suppress warnings about obsolete options, use -Xlint:-options.
error: warnings found and -Werror specified
```

Suppress this warning explicitly for this module.

Signed-off-by: Robert Muir <rmuir@apache.org>

* more java14 -> java11 cleanup

Signed-off-by: Robert Muir <rmuir@apache.org>

Co-authored-by: Robert Muir <rmuir@apache.org>
2021-07-06 11:55:23 -05:00
Marc Handalian 25bbdb7219
Update DistributionDownloader to support fetching arm64 bundles. (#929)
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-07-02 17:07:02 -05:00
Marc Handalian 2335194ebe
Fix Snapshot pattern in DistributionDownloader. (#916)
Snapshots are published with a -latest flag into s3, this updates
the pattern to correctly point to -latest.

Signed-off-by: Marc Handalian <handalm@amazon.com>
2021-07-01 16:53:17 -05:00
Marc Handalian 06228bc25b
Update DistributionDownloader to fetch snapshots and staging bundles. (#904)
This change updates DistributionDownloader by registering a second
repository containing release-candidate bundles.  This repository will
only be checked if the release repository does not contain the requested
version.

This change also updates the snapshot repository to point to a new
snapshot repository.

Signed-off-by: Marc Handalian <handalm@amazon.com>
2021-06-30 10:29:07 -07:00
Nick Knize ef1cbd9e3d
Version bump for 1.1 release (#772)
Bumps the minor to 1.1 for the 1.0 release.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-06-28 09:58:27 -05:00
Nick Knize 0317b604f6
[BUG] Fix versioning issues discovered through version bump (#884)
This commit fixes up the version framework to ensure a smooth version bump
across releases.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-06-25 16:53:06 -05:00
Marc Handalian c7e60b87f9
Add URL for lucene snapshots (#858)
Signed-off-by: Marc Handalian <handalm@amazon.com>
2021-06-22 11:40:53 -05:00
Tianli Feng 110cef7882
Fix resource leak issues suggested by Amazon CodeGuru (#816)
* Address a kind of issue suggested by Amazon CodeGuru Reviewer:

* Add try-with-resources block to automatically close the resources after using to avoid resource leak, in `SymbolicLinkPreservingTarIT`, `LicenseAnalyzer`, `SymbolicLinkPreservingUntarTransform`, `ConcurrentSeqNoVersioningIT` in `VersionProperties`, `GeoFilterIT`, `XContentHelper`,  `Json` and `IndexShard` class

* Add try-finally block to close the resources after using to avoid resource leak, in `ServerChannelContext` class.

* Add try-catch block to close the resources when exception occurs in `FsBlobContainer` class (when XContentFactory.xContentType throws an exception).

* Close resources when assertion error occurs, in `ServerChannelContext` class.
2021-06-15 09:31:36 -07:00
Peter Nied 74f29c7727
Fix arm architecture translation issue (#809)
Found when attempting to build on an `arm64` machine when I recieved an error message below.  Root cause is that string equality in java cannot be done with the `==` sign.

```
unknown architecture [arm64] for jdk [provisioned_runtime], must be one of [aarch64, x64]
```

Signed-off-by: Peter Nied <petern@amazon.com>
2021-06-04 12:26:46 -04:00
Gregor Zurowski 0f9060761c
Fix Javadoc errors in `client/sniffer` (#802)
* Adds a gradle plugin to validate missing javadocs

Use `./gradlew missingJavadoc` to validate missing javadocs.
Currently this task fails because several modules are missing
appropriate javadocs. Once added, this should pass.
Also, precommit PomValidation check currently fails with missing Javadoc
plugin, that needs to be fixed -
https://github.com/opensearch-project/OpenSearch/issues/449
Thus keeping this in a separate feature branch.

Signed-off-by: Himanshu Setia <setiah@amazon.com>

* Fix Javadoc errors in client/sniffer module

Signed-off-by: Gregor Zurowski <gregor@zurowski.org>

* Add package info to client/sniffer module

Signed-off-by: Gregor Zurowski <gregor@zurowski.org>

Co-authored-by: Himanshu Setia <setiah@amazon.com>
2021-06-03 17:08:16 -07:00
Daniel Doubrovkine (dB.) bd9ca7cce5
Version checks are incorrectly returning versions < 1.0.0. (#797)
* Version checks are incorrectly returning versions < 1.0.0.

Signed-off-by: dblock <dblock@amazon.com>

* Removed V_7_10_3 which has not been released as of time of the fork.

Signed-off-by: dblock <dblock@amazon.com>

* Update check for current version to get unreleased versions.

- no unreleased version if the current version is "1.0.0"
- add unit tests for OpenSearch 1.0.0 with legacy ES versions.
- update VersionUtils to include all legacy ES versions as released.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>

Co-authored-by: Rabi Panda <adnapibar@gmail.com>
2021-06-02 16:03:50 -04:00
Rabi Panda 59563fbf72
Support running elasticsearch-oss distribution in test cluster for BWC (#764)
We need to install elasticsearch in the test cluster nodes to be able to run the BWC tests. This commit makes the following changes to enable that,
- updates DistributionDownloadPlugin to download elasticsearch-oss distributions.
- updates OpenSearchNode to be able to run both OpenSearch and Elasticsearch distributions.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-05-25 11:16:12 -07:00
Rabi Panda 9c95cf424b
[Bug] Fix gradle build on Windows failing from a recent change (#758)
A recent change as part of the commit c2e816ec introduced a bug where the build is failing on Windows. The change was made to include the NOTICE.txt file as read-only in the distributions. The code fails on Windows as it's not a POSIX-compliant. This commit adds a check on the current operating system.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-05-24 11:15:06 -07:00
Himanshu Setia 6f893ed1cd
Merging javadoc feature branch changes to main (#715)
* Adds a gradle plugin to validate missing javadocs

Use `./gradlew missingJavadoc` to validate missing javadocs.
Currently this task fails because several modules are missing
appropriate javadocs. Once added, this should pass.
Also, precommit PomValidation check currently fails with missing Javadoc
plugin, that needs to be fixed -
https://github.com/opensearch-project/OpenSearch/issues/449
Thus keeping this in a separate feature branch.

Signed-off-by: Himanshu Setia <setiah@amazon.com>

* Fix Javadoc errors in module `client/rest` (#685)

* Fix Javadoc errors in client/rest module

Signed-off-by: Gregor Zurowski <gregor@zurowski.org>

* Add package info file in client/rest module

Signed-off-by: Gregor Zurowski <gregor@zurowski.org>

* Fix typos

Signed-off-by: Gregor Zurowski <gregor@zurowski.org>

* Add exception documentation to Javadoc

Signed-off-by: Gregor Zurowski <gregor@zurowski.org>

* Fixes precommit task configuration failures due to newly added missin… (#707)

* Fixes precommit task configuration failures due to newly added missingJavadoc task

Signed-off-by: Himanshu Setia <setiah@amazon.com>

* Fixes javadoc task errors due to PR#685

Signed-off-by: Himanshu Setia <setiah@amazon.com>

* Updated CONTRIBUTING.md for info on javadocs

Signed-off-by: Himanshu Setia <setiah@amazon.com>

* Correcting licenses and naming

Signed-off-by: Himanshu Setia <setiah@amazon.com>

* Correcting version info

Signed-off-by: Himanshu Setia <setiah@amazon.com>

Co-authored-by: Gregor Zurowski <gregor@zurowski.org>
2021-05-18 13:21:41 -07:00
Rabi Panda 50abf6d066
[CVE] Upgrade dependencies to mitigate CVEs (#657)
This PR upgrade the following dependencies to fix CVEs.

- commons-codec:1.12 (->1.13) apache/commons-codec@48b6157
- ant:1.10.8 (->1.10.9) https://ant.apache.org/security.html
- jackson-databind:2.10.4 (->2.11.0) FasterXML/jackson-databind#2589
- jackson-dataformat-cbor:2.10.4 (->2.11.0) https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-28491
- apache-httpclient:4.5.10 (->4.5.13) https://bugzilla.redhat.com/show_bug.cgi?id=CVE-2020-13956
- checkstyle:8.20 (->8.29) https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-10782
- junit:4.12 (->4.13.1) https://github.com/junit-team/junit4/security/advisories/GHSA-269g-pwp5-87pp
- netty:4.1.49.Final (->4.1.59) https://github.com/netty/netty/security/advisories/GHSA-5mcr-gq6c-3hq2

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-05-18 11:37:24 -07:00
Rabi Panda c2e816ecf6
[TEST] Fix failing distro tests for linux packages (#569)
Changes to fix the failing OpenSearch distribution tests for packages (linux-archive, linux-archive-aarch64, debian, rpm, docker) on supported linux distros. 

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-05-15 21:45:48 -07:00
Daniel Doubrovkine (dB.) d9cb7daa65
Make -Dtests.output=always actually work. (#648)
Signed-off-by: dblock <dblock@amazon.com>
2021-05-04 19:45:04 -04:00
Sarat Vemulapalli 0e9f74e35f
Renaming CPU architecture to have consistent naming (#612)
* Renaming CPU architecture from x86_64, aarch64 to x64, arm64

Signed-off-by: Sarat Vemulapalli <vemulapallisarat@gmail.com>
2021-04-26 09:40:09 -07:00
Nick Knize c5a3c3cb41
Update lucene version to 8.8.2 (#557)
This commit updates the codebase to the latest released version of Lucene.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-04-23 09:48:41 -05:00
Rabi Panda 3fede8be3c
Rename the distribution used in test clusters. (#603)
For test clusters, we are using the archive(zip, tar), so we rename the distribution accordingly.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-04-22 14:21:32 -07:00
Sarat Vemulapalli 9701b226d8
Updating the Ivy repository to point to real url for Releases (#602)
Signed-off-by: Sarat Vemulapalli <vemulapallisarat@gmail.com>
2021-04-22 10:22:50 -07:00
Rabi Panda 163a51fc5a
Remove the oss string from OpenSearch distributions (#575)
This commit removes the 'oss' string which was a remnant of the predecessor distribution flavors. As OpenSearch has no flavors for distributions, we are removing this tag from all the distribution names.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-04-20 09:27:40 -07:00
Robert Muir 5474e8d094
Speed ups to test suite and precommit tasks. (#580)
Use C1 compiler only for short-lived tasks and unit test execution. Tone
down some of the slowest unit tests.

Signed-off-by: Robert Muir <rmuir@apache.org>
2021-04-20 09:02:45 -05:00
Nick Knize 0ba0e7cc26
[Versioning] Rebase to OpenSearch version 1.0.0 (#555)
This commit rebases the versioning to OpenSearch 1.0.0

Co-authored-by: Rabi Panda <adnapibar@gmail.com>

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-04-15 17:06:47 -05:00
Robert Muir 6aa3675e9d
Prevent setting maxParallelForks=0 on single-cpu machines (#558)
The maxParallelForks is currently numJavaCpus / 2, but this evaluates to
zero (illegal value) if numJavaCpus is 1 and breaks the build.

Remove large chunk of complex dead code above this calculation, as it is
always overwritten by the simple calculation anyway.

Closes #554

Signed-off-by: Robert Muir <rmuir@apache.org>
2021-04-14 19:17:58 -07:00
Nick Knize ee6d15e26a
[License] Add SPDX License Header to security policies (#531)
This commit adds the SPDX license header and modifications copyright to security
policy files.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-04-12 22:59:36 -05:00
Nick Knize 9168f1fb43
[License] Add SPDX and OpenSearch Modification license header (#509)
This commit adds the SPDX Apache-2.0 license header along with an additional
copyright header for all modifications.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-04-09 14:28:18 -05:00
Nick Knize 8f4b2d58d1
[License] Update SPDX License Header (#510)
This commit updates the SPDX License Header for all new files created by
OpenSearch contributors.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-04-09 13:49:56 -05:00
Rabi Panda 2dca3462f2
Fix stragglers from renaming to OpenSearch work. (#483)
This commit fixes more instances where we missed renaming to OpenSearch.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-04-05 11:51:20 -07:00
Rabi Panda 247772033a
Update remote repo for BWC checks. (#482)
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-04-02 13:05:47 -07:00
Rabi Panda 70cad29af3
Perform more renaming to OpenSearch. (#470)
This commit performs more renaming to OpenSearch which were missed in the earlier commits.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-03-30 11:05:50 -07:00
Rabi Panda f7a7b80fba
Fix build-tools integ test failures. (#465)
A few build-tools integTests were failing as a result of the renaming to OpenSearch. This was due to some un-renamed files in the fake archives used for testing. This commit renames those.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-03-26 12:30:13 -07:00
Rabi Panda 3460a8c213
Fix a few more renaming issues. (#464)
This commit fixes some more missed instances where we can perform the renaming to OpenSearch.

Signed-off-by: Rabi Panda <adnapibar@gmail.com>
2021-03-26 12:05:16 -07:00
Himanshu Setia f9b0b9fe6d
[GradleCheck] Fixes failing tests for ExportOpenSearchBuildResourcesTaskIT (#442)
Signed-off-by: Himanshu Setia <setiah@amazon.com>
2021-03-23 12:23:11 -07:00
Nick Knize 49907793f3
[Rename] Change license header and copyright notice to SPDX (#437)
This commit changes the oss-license-header.txt file from the legacy ASLv2 header
to the new SPDX format. It also updates the ide gradle file and
LicenseHeaderTask of precommit to accept the new SPDX license format.

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
2021-03-22 12:36:34 -05:00