This change fixes the issue where the sources and javadoc artifacts were not built and included with the publish.
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
* Lower build requirement from Java 14+ to Java 11+
Avoid use of -Werror -Xlint:all, which may change significantly across
java releases (new warnings could be added). Instead, just list the
warnings individually.
Workaround JDK 11 compiler bug (JDK-8209058) that only impacts test fixture
code in the build itself.
Signed-off-by: Robert Muir <rmuir@apache.org>
* Disable warning around -source 7 -release 7 for java version checker
The java version checker triggers some default warnings because it
targets java7:
```
> Task :distribution:tools:java-version-checker:compileJava FAILED
warning: [options] source value 7 is obsolete and will be removed in a future release
warning: [options] target value 7 is obsolete and will be removed in a future release
warning: [options] To suppress warnings about obsolete options, use -Xlint:-options.
error: warnings found and -Werror specified
```
Suppress this warning explicitly for this module.
Signed-off-by: Robert Muir <rmuir@apache.org>
* more java14 -> java11 cleanup
Signed-off-by: Robert Muir <rmuir@apache.org>
Co-authored-by: Robert Muir <rmuir@apache.org>
Snapshots are published with a -latest flag into s3, this updates
the pattern to correctly point to -latest.
Signed-off-by: Marc Handalian <handalm@amazon.com>
This change updates DistributionDownloader by registering a second
repository containing release-candidate bundles. This repository will
only be checked if the release repository does not contain the requested
version.
This change also updates the snapshot repository to point to a new
snapshot repository.
Signed-off-by: Marc Handalian <handalm@amazon.com>
* Address a kind of issue suggested by Amazon CodeGuru Reviewer:
* Add try-with-resources block to automatically close the resources after using to avoid resource leak, in `SymbolicLinkPreservingTarIT`, `LicenseAnalyzer`, `SymbolicLinkPreservingUntarTransform`, `ConcurrentSeqNoVersioningIT` in `VersionProperties`, `GeoFilterIT`, `XContentHelper`, `Json` and `IndexShard` class
* Add try-finally block to close the resources after using to avoid resource leak, in `ServerChannelContext` class.
* Add try-catch block to close the resources when exception occurs in `FsBlobContainer` class (when XContentFactory.xContentType throws an exception).
* Close resources when assertion error occurs, in `ServerChannelContext` class.
Found when attempting to build on an `arm64` machine when I recieved an error message below. Root cause is that string equality in java cannot be done with the `==` sign.
```
unknown architecture [arm64] for jdk [provisioned_runtime], must be one of [aarch64, x64]
```
Signed-off-by: Peter Nied <petern@amazon.com>
* Adds a gradle plugin to validate missing javadocs
Use `./gradlew missingJavadoc` to validate missing javadocs.
Currently this task fails because several modules are missing
appropriate javadocs. Once added, this should pass.
Also, precommit PomValidation check currently fails with missing Javadoc
plugin, that needs to be fixed -
https://github.com/opensearch-project/OpenSearch/issues/449
Thus keeping this in a separate feature branch.
Signed-off-by: Himanshu Setia <setiah@amazon.com>
* Fix Javadoc errors in client/sniffer module
Signed-off-by: Gregor Zurowski <gregor@zurowski.org>
* Add package info to client/sniffer module
Signed-off-by: Gregor Zurowski <gregor@zurowski.org>
Co-authored-by: Himanshu Setia <setiah@amazon.com>
* Version checks are incorrectly returning versions < 1.0.0.
Signed-off-by: dblock <dblock@amazon.com>
* Removed V_7_10_3 which has not been released as of time of the fork.
Signed-off-by: dblock <dblock@amazon.com>
* Update check for current version to get unreleased versions.
- no unreleased version if the current version is "1.0.0"
- add unit tests for OpenSearch 1.0.0 with legacy ES versions.
- update VersionUtils to include all legacy ES versions as released.
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
Co-authored-by: Rabi Panda <adnapibar@gmail.com>
We need to install elasticsearch in the test cluster nodes to be able to run the BWC tests. This commit makes the following changes to enable that,
- updates DistributionDownloadPlugin to download elasticsearch-oss distributions.
- updates OpenSearchNode to be able to run both OpenSearch and Elasticsearch distributions.
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
A recent change as part of the commit c2e816ec introduced a bug where the build is failing on Windows. The change was made to include the NOTICE.txt file as read-only in the distributions. The code fails on Windows as it's not a POSIX-compliant. This commit adds a check on the current operating system.
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
* Adds a gradle plugin to validate missing javadocs
Use `./gradlew missingJavadoc` to validate missing javadocs.
Currently this task fails because several modules are missing
appropriate javadocs. Once added, this should pass.
Also, precommit PomValidation check currently fails with missing Javadoc
plugin, that needs to be fixed -
https://github.com/opensearch-project/OpenSearch/issues/449
Thus keeping this in a separate feature branch.
Signed-off-by: Himanshu Setia <setiah@amazon.com>
* Fix Javadoc errors in module `client/rest` (#685)
* Fix Javadoc errors in client/rest module
Signed-off-by: Gregor Zurowski <gregor@zurowski.org>
* Add package info file in client/rest module
Signed-off-by: Gregor Zurowski <gregor@zurowski.org>
* Fix typos
Signed-off-by: Gregor Zurowski <gregor@zurowski.org>
* Add exception documentation to Javadoc
Signed-off-by: Gregor Zurowski <gregor@zurowski.org>
* Fixes precommit task configuration failures due to newly added missin… (#707)
* Fixes precommit task configuration failures due to newly added missingJavadoc task
Signed-off-by: Himanshu Setia <setiah@amazon.com>
* Fixes javadoc task errors due to PR#685
Signed-off-by: Himanshu Setia <setiah@amazon.com>
* Updated CONTRIBUTING.md for info on javadocs
Signed-off-by: Himanshu Setia <setiah@amazon.com>
* Correcting licenses and naming
Signed-off-by: Himanshu Setia <setiah@amazon.com>
* Correcting version info
Signed-off-by: Himanshu Setia <setiah@amazon.com>
Co-authored-by: Gregor Zurowski <gregor@zurowski.org>
Changes to fix the failing OpenSearch distribution tests for packages (linux-archive, linux-archive-aarch64, debian, rpm, docker) on supported linux distros.
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
This commit removes the 'oss' string which was a remnant of the predecessor distribution flavors. As OpenSearch has no flavors for distributions, we are removing this tag from all the distribution names.
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
Use C1 compiler only for short-lived tasks and unit test execution. Tone
down some of the slowest unit tests.
Signed-off-by: Robert Muir <rmuir@apache.org>
This commit rebases the versioning to OpenSearch 1.0.0
Co-authored-by: Rabi Panda <adnapibar@gmail.com>
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
The maxParallelForks is currently numJavaCpus / 2, but this evaluates to
zero (illegal value) if numJavaCpus is 1 and breaks the build.
Remove large chunk of complex dead code above this calculation, as it is
always overwritten by the simple calculation anyway.
Closes#554
Signed-off-by: Robert Muir <rmuir@apache.org>
This commit adds the SPDX license header and modifications copyright to security
policy files.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
This commit adds the SPDX Apache-2.0 license header along with an additional
copyright header for all modifications.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
This commit updates the SPDX License Header for all new files created by
OpenSearch contributors.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
A few build-tools integTests were failing as a result of the renaming to OpenSearch. This was due to some un-renamed files in the fake archives used for testing. This commit renames those.
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
This commit changes the oss-license-header.txt file from the legacy ASLv2 header
to the new SPDX format. It also updates the ide gradle file and
LicenseHeaderTask of precommit to accept the new SPDX license format.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
This commit updates the NOTICE.txt file to include the OpenSearch copyright
notice.
Signed-off-by: Nicholas Knize <nknize@amazon.com>
Signed-off-by: Peter Nied <petern@amazon.com>
This commit refactors instances of 'elasticsearch' with opensearch everywhere
except references to issues, and other places needed to test compatibility with
old elasticsearch clusters.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
Fix miscellaneous issues identified during `gradle precommit`. These issues are the side effects of the renaming to OpenSearch work.
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
This commit fixes the currently broken gradle build resulted from the renaming work. It reverts a few dependencies and comments out the `opensearch_distibutions` task which is currently failing for some builds. We will address these separately in the future once we have a working build.
Signed-off-by: Rabi Panda <adnapibar@gmail.com>
This commit refactors the remaining o.e.index and o.e.test packages in the
test/fixtures module. References throughout the codebase are also refactored.
Signed-off-by: Nicholas Walter Knize <nknize@apache.org>
This commit updates the NOTICE.txt file to include the OpenSearch copyright
notice.
Signed-off-by: Nicholas Knize <nknize@amazon.com>
Signed-off-by: Peter Nied <petern@amazon.com>
This commit changes the building, packaging, and testing framework to only support OSS on different distributions.
Next steps:
completely remove -oss flag dependencies in package and build tests
move 6.x bwc testing to be an explicit option
remove any references to elastic.co download site (or replace with downloads from the OSS website)
Co-authored-by: Himanshu Setia <setiah@amazon.com>
Co-authored-by: Rabi Panda <pandarab@amazon.com>
Co-authored-by: Himanshu Setia <58999915+setiah@users.noreply.github.com>
Co-authored-by: Sarat Vemulapalli <vemsarat@amazon.com>
Signed-off-by: Peter Nied <petern@amazon.com>
This commit cleans up the following:
* Remove unused imports
* Remove ILM settings in hlrc testCluster formation
* Comment out security users settings in ElasticsearchNode creation for build-tools tests
Signed-off-by: Peter Nied <petern@amazon.com>
This commit removes all trace of the security high level rest client and other reference to x-pack security
Co-authored-by: Rabi Panda <rabipanda@icloud.com>
Signed-off-by: Peter Nied <petern@amazon.com>
Closes#63869. Perform `docker pull` explicitly instead of as part of
`docker build`, and wrap it in a retry loop. This is an attempt to make
the build more resilient to transient errors.
This commit converts build code that downloads distributions or other
artifacts to use the new no-kpi subdomain, and removes the formerly used
no-kpi header.
This commit updates the bundled jdk to 15.0.1, and at the same time once
again switches the bundled jdk back to adoptopenjdk, which has fixed
their build problem and regained support for glibc 2.12.
closes#64026
* Fix concurrent modification on task realization
* Use taskprovider instead of relying on tasks in distribution setup
* Port more task references in :distribution to task provider
* Fix nullpointer in distribution setup
The ML cpp notice only exists with default distributions, but the check
task exists on all archive distributions. This commit avoids creating
the task for distributions that don't have ML.
closes#63128
* Wire local unreleased bwc versions more efficient for tests (#62473)
For testing against the local distribution we already avoid the packaging/unpackaging
cycle of es distributions when setting up test clusters. This PR adopts the usage of the
expanded created distributions for unreleased bwc versions (versions that are checkout
from a branch and build from source in the :distribution:bwc:minor / :distribution:bwc:bugfix).
This makes the setup of bwc based cross version tests a bit faster by avoiding
the unpackaging overhead. We still assemble both in the bwcBuild tasks atm
which will be addressed in a later issue.
This reworks the :distribution:bwc project:
- Convert all the custom logic from build script logic (groovy) into gradle binary plugins (java)
- Tried to make the bwc setup logic a bit more readable
- Add basic functional test coverage for the bwc logic this PR tweaked.
- Extracted a general internal BWC Git plugin out of the bwc setup plugin to improve maintenance
and testability
- Changed the InternalDistributionPlugin to resolve the extracted distro instead on relying
on unpacking the distribution archive
* Fix java8 incompatibility
* Fix extension calculation for 6.8.* distribution
We switched to adoptopenjdk from oracle jdk to rely on the notarization
found in adoptopnejdk on MacOS. However, that notarization still had
issues, and we currently do our own notarization of the entire
distribution, including the jdk. The recent bump to jdk 15 has revealed
openjdk to be lax in maintaining support for older systems. Since the
notarization is no longer an issue, this PR moves the bundled jdk back
to Oracle, in order to continue supporting those older systems affected
by adoptopenjdk 15.
relates #62709
Referencing a project instance during task execution is discouraged by
Gradle and should be avoided. E.g. It is incompatible with Gradles
incubating configuration cache. Instead there are services available to handle
archive and filesystem operations in task actions.
Brings us one step closer to #57918
Backport for #62985 that includes the related changes, but
not the actual plugin for yamlRestCompatTest. The plugin
is not necessary in 7.x, and back porting relevant changes to
help keep 7.x code inline with master.
Bouncy Castle's BC-FJA-1.0.2 has been certified for a while now
but we had noticed that it seems to be rather entropy hungry and
ES would start very slowly ( and tests would take forever )
because of blocking calls to /dev/random.
We verified that this is resolved when enabling hw RNG or a
software one like haveged. While rng-tools should be suggested for
production uses, our ephemeral workers have haveged installed
which should work just fine for CI.
Backport of 63099
We support `"""` in `console` snippets to emulate kibana's CONSOLE.
CONSOLE also spits out `"""` when a json field contains a new line or a
double quote. This adds support for those sorts of responses to the
handling of `console-response` snippets.
JAVA_HOME is set as necessary in packaging tests, depending on whether
it is needed for no-jdk distributions or testing override behavior. We
currently rely on gradle finding java through PATH. However, JAVA_HOME
can sometimes be set by the system itself, which then leaks through to
the packaging test. This commit reworks our handling of JAVA_HOME to
pass it through for gradle, and then explicitly clear it whenever
running shell commands in packaging tests.
We use the bundled jdk for unit, integ and packaging tests. Since
upgrading to jdk 15, centos-6 and oracle enterprise linux 6 have failed
due to versions of glibc no longer supported by the jdk. This commit
adds detection of the old glibc versions to gradle, and utilizes that
when deciding which jdk to use for tests.
relates #62709closes#62635
The distro tests rely on two jdks, pulled in by the jdk download plugin.
The move the artifact transforms result in the path to the extracted
jdks existing under the gradle cache dir, which is outside the vagrant
mount of the elasticsearch project. This commit creates a local copy
within the `qa:os` project that the packaging tests use.
closes#61138
This commit changes the yamlRestTest and javaRestTest tasks to be lazily created.
This change requires pro-actively creating the testClusters container so that the
configuration can be applied without any changes to the build.gradle files.
related: #60261
related: #47804
This commit address some build failures from the perspective of Intellij.
These changes include:
* changing an order of a dependency definition that seems to can cause Intellij build to fail.
* introduction of an abstract class out of the test source set (seems to be an issue sharing
classes cross projects with non-standard source sets.
* a couple of missing dependency definitions (not sure how the command line worked prior to this)
This new snapshot contains the following JIRAs that we're interested in:
- [LUCENE-9525](https://issues.apache.org/jira/browse/LUCENE-9525)
Better handling of small documents. This should improve retrieval times
when documents are less than ~1kB.
- [LUCENE-9510](https://issues.apache.org/jira/browse/LUCENE-9510)
Faster flushes when index sorting is enabled by not compressing the
temporary files that store stored fields and term vectors.
- Extract distribution archives defaults into plugin
- Added basic test coverage
- Avoid packaging/unpackaging cycle when relying on locally build distributions
- Provide DSL for setting up distribution archives
- Cleanup archives build script
This commit removes `integTest` task from all es-plugins.
Most relevant projects have been converted to use yamlRestTest, javaRestTest,
or internalClusterTest in prior PRs.
A few projects needed to be adjusted to allow complete removal of this task
* x-pack/plugin - converted to use yamlRestTest and javaRestTest
* plugins/repository-hdfs - kept the integTest task, but use `rest-test` plugin to define the task
* qa/die-with-dignity - convert to javaRestTest
* x-pack/qa/security-example-spi-extension - convert to javaRestTest
* multiple projects - remove the integTest.enabled = false (yay!)
related: #61802
related: #60630
related: #59444
related: #59089
related: #56841
related: #59939
related: #55896
This commit adds external test modules. These are modules meant for
external systems to test edge cases in elasticsearch, but only within
snapshots. They are not meant to be used in production, so protections
are also added from their accidental inclusion in release builds.
Note that this commit does not actually add any new modules, it only
adds the infrastructure for the new modules, under
`test/external-modules`.
As we figured out in
https://github.com/elastic/elasticsearch/issues/61316#issuecomment-685482708
Azul brings back a lot of changes from JDK 11 to their Zulu8 build
and this means that we can't run this with SunJSSE in FIPS 140 mode.
This change ensures that we configure Zulu8 JDK JVMs in FIPS 140
mode, using the bouncy castle JSSE FIPS provider, instead of the
SunJSSE one ( as we do for the rest of the java 8 JVMs )
Resolves: #61316
There are currently half a dozen ways to add plugins and modules for
test clusters to use. All of them require the calling project to peek
into the plugin or module they want to use to grab its bundlePlugin
task, and then both depend on that task, as well as extract the archive
path the task will produce. This creates cross project dependencies that
are difficult to detect, and if the dependent plugin/module has not yet
been configured, the build will fail because the task does not yet
exist.
This commit makes the plugin and module methods for testclusters
symmetetric, and simply adding a file provider directly, or a project
path that will produce the plugin/module zip. Internally this new
variant uses normal configuration/dependencies across projects to get
the zip artifact. It also has the added benefit of no longer needing the
caller to add to the test task a dependsOn for bundlePlugin task.
We leverage artifact transforms now when downloading and unpacking elasticsearch distributions.
This has the benefit of
- handcrafted extract tasks on the root project are not required.
- The general tight coupling to the root project has been removed.
- The overall required configurations required to handle a distribution have been reduced
- ElasticsearchDistribution has been simplified by making Extracted an ordinary Configuration
downloaded and unpacked external distributions are reused in later builds by been cached
in the gradle user home.
DistributionDownloadPlugin functional tests have been extended and ported
to DistributionDownloadPluginFuncTest.
* Fix ElasticsearchNode#getDistributionFiles (#61219)
Fixes#61647
- Added test coverage
- Removes build script cluttering
- Splits archive building and archive checking logic
- only rely on boost for now for ML licenses(tbd)
- Use Gradle build-in untar and unzip support
* Handle dynamic versions in func tests assertions
Driven by this issue
https://github.com/elastic/elasticsearch/pull/60969#issuecomment-674962158
we apply some rework on how we handle distributions in our test cluster setups:
- If no custom modules, plugins or extra jar files are declared we do not create a cluster
specific distro folder and use the origin distribution folder instead.
- If a custom distribution folder is required, we fallback to file copy when hard linking
is not supported
Access the common versions map is done in a lot of places. While it can
be access through an import of VersionProperties, the vast majority of
places use it through the provided convenience property added by
BuildPlugin. This commit moves that convenience property to the base
java plugin, so further reduce dependence on the BuildPlugin.
Backport of #60742.
This PR resurrects support for building Docker images based on one of
Red Hat's UBI images. It also adds support for running the existing
Docker tests against the image. The image is named
`elasticsearch-ubi8:<version>`.
I also changed the Docker build file uses enums instead strings in a lot
of places, for added rigour.
We leverage artifact transforms now when downloading and unpacking elasticsearch distributions.
This has the benefit of
- handcrafted extract tasks on the root project are not required.
- the general tight coupling to the root project has been removed.
- the overall required configurations required to handle a distribution have been reduced
- ElasticsearchDistribution has been simplified by making Extracted an ordinary Configuration
downloaded and unpacked external distributions are reused in later builds by been cached
in the gradle user home.
DistributionDownloadPlugin functional tests have been extended and ported
to DistributionDownloadPluginFuncTest.
* Fix java8 compliant Path calculation
With Gradle 6.6 we can now use the native support for the --release compile options.
As Gradle by default resolves the compatibility version from the release property we needed to workaround this in order to keep our current setup. An issue was raised at gradle to track this at https://github.com/gradle/gradle/issues/14141
The reaper is used only when spawned processes by the build like
testclusters or vagrant are left behind in a failed build. When this
occurs on Windows, the Taskkill command is used. However, there was a
typo in the command name. Since the reaper is for edgecases and mostly
protection for local builds (CI uses immutable hosts), this has been
broken since its inception.
closes#60503
The Elasticsearch `run` task in Gradle is backed by testclusters
spinning up a node and waiting indefinitely on output from that node.
However, this thread currently waits for an interrupt to exit. If
Elasticsearch fails to start, we hang indefinitely. This commit adds a
check after each poll of the process stdout to see if it is still alive,
and fails the build if it has died.
closes#60761
Though bats tests were recently removed, there remains a few unnecessary
setups needed for those tests in our vagrant files, as well as CI setup.
Additionally, we no longer rely on the vagrant images setting a
JAVA_HOME, instead relying on DistroTestPlugin to pull the appropriate
jdk when testing with no-jdk distributions.
* Simplify jdk downloads via artifact transforms
This reworks the jdk download plugin to use artifact transforms.
That allows us to simplify the jdk download plugin by a lot. The
benefits of using artifact transforms are:
- they transform an artifact to an upacked folder on the fly as part
of the dependenc resolution allowing us to remove all the custom created unpack tasks
and configurations
- Artifact transforms support gradle build cache. Requesting a jdk folder on a clean machine
will likely be resolved from the build cache
- The manual mingling about not extracting jdks multiple times by all jdks channeling through
root project configurations can be removed as they support up-to-date checking and build cache
which will ensure these archives are only resolved once per build at max.
Also the test coverage has been ported to Spock that supports data driven testing. This porting
includes an introduction to a wiremock fixture that can be used later on for mocking repository
urls in other integration tests
* Simplify artifact transform registration
* Change jdk finalization and repository setup
Jdk finalization is now done via `configuration.defaultDependencies`.
This has two benefits:
- we only configure and finalize the JDKs we actually use during the build
- afterEvaluate hooks are problematic in combination with task avoidance api.
Relying on task avoidance api in Gradle moves more configuration logic into the materialization
of tasks which is, with task avoidance api, is done ideally during the task graph calculation.
Anything created (e.g. jdks) created late in this task graph calculation would never be finalized
via afterEvaluate hooks as this has been fired before already with the jdk not there yet.
Furthermore we now only configure repositories in the projects where the jdk is declared
(aka the jdk-download plugin is applied) and only if the jdk is actually requested during the build.
* Fix javadoc
* Fix jdk download repo content filtering
* Minor cleanup
* Apply review feedback and cleanup
This commit removes the last of the bats tests, converting the rpm/deb
upgrade tests to java. It adds a new pattern of tasks, similar in nature
but separate from the existing distro tests, named `distroUpgradeTest`.
For each index compatible version, a `distroUpgradeTest.VERSION` task
exxists. Each distribution then has a task, named
`distroUpgradeTest.VERSION.DISTRO`.
One thing to note is these new tests do not cover no-jdk versions of
the rpm/deb packages, since the distribution/bwc project does not
currently build those.
closes#59145closes#46005
* Merge test runner task into RestIntegTest (#60261)
* Merge test runner task into RestIntegTest
* Reorganizing Standalone runner and RestIntegTest task
* Rework general test task configuration and extension
* Fix merge issues
* use former 7.x common test configuration
- Replace immediate task creations by using task avoidance api
- One step closer to #56610
- Still many tasks are created during configuration phase. Tackled in separate steps
* Split internal distribution handling into separate internal plugin (#60295)
* Provide proper failure if unexpected non jdk bundled bwc version is requested
Sven R
Rabi Panda
xuezhou25
Vacha
Daniel Doubrovkine (dB.)
Marc Handalian
Nick Knize
Tianli Feng
Peter Nied
Gregor Zurowski
Himanshu Setia
Sarat Vemulapalli
Robert Muir
Robert Muir
Nicholas Walter Knize
Rene Groeschke
Francisco Fernández Castaño
Rory Hunter
Mark Vieira
Andrei Dan
Ryan Ernst
Ignacio Vera
Mayya Sharipova
Nhat Nguyen
Jake Landis
Ioannis Kakavas
Nik Everett
Adrien Grand
Jason Tedor
Yannick Welsch