This new snapshot contains the following JIRAs that we're interested in:
- [LUCENE-9525](https://issues.apache.org/jira/browse/LUCENE-9525)
Better handling of small documents. This should improve retrieval times
when documents are less than ~1kB.
- [LUCENE-9510](https://issues.apache.org/jira/browse/LUCENE-9510)
Faster flushes when index sorting is enabled by not compressing the
temporary files that store stored fields and term vectors.
- Extract distribution archives defaults into plugin
- Added basic test coverage
- Avoid packaging/unpackaging cycle when relying on locally build distributions
- Provide DSL for setting up distribution archives
- Cleanup archives build script
This commit removes `integTest` task from all es-plugins.
Most relevant projects have been converted to use yamlRestTest, javaRestTest,
or internalClusterTest in prior PRs.
A few projects needed to be adjusted to allow complete removal of this task
* x-pack/plugin - converted to use yamlRestTest and javaRestTest
* plugins/repository-hdfs - kept the integTest task, but use `rest-test` plugin to define the task
* qa/die-with-dignity - convert to javaRestTest
* x-pack/qa/security-example-spi-extension - convert to javaRestTest
* multiple projects - remove the integTest.enabled = false (yay!)
related: #61802
related: #60630
related: #59444
related: #59089
related: #56841
related: #59939
related: #55896
This commit adds external test modules. These are modules meant for
external systems to test edge cases in elasticsearch, but only within
snapshots. They are not meant to be used in production, so protections
are also added from their accidental inclusion in release builds.
Note that this commit does not actually add any new modules, it only
adds the infrastructure for the new modules, under
`test/external-modules`.
As we figured out in
https://github.com/elastic/elasticsearch/issues/61316#issuecomment-685482708
Azul brings back a lot of changes from JDK 11 to their Zulu8 build
and this means that we can't run this with SunJSSE in FIPS 140 mode.
This change ensures that we configure Zulu8 JDK JVMs in FIPS 140
mode, using the bouncy castle JSSE FIPS provider, instead of the
SunJSSE one ( as we do for the rest of the java 8 JVMs )
Resolves: #61316
There are currently half a dozen ways to add plugins and modules for
test clusters to use. All of them require the calling project to peek
into the plugin or module they want to use to grab its bundlePlugin
task, and then both depend on that task, as well as extract the archive
path the task will produce. This creates cross project dependencies that
are difficult to detect, and if the dependent plugin/module has not yet
been configured, the build will fail because the task does not yet
exist.
This commit makes the plugin and module methods for testclusters
symmetetric, and simply adding a file provider directly, or a project
path that will produce the plugin/module zip. Internally this new
variant uses normal configuration/dependencies across projects to get
the zip artifact. It also has the added benefit of no longer needing the
caller to add to the test task a dependsOn for bundlePlugin task.
We leverage artifact transforms now when downloading and unpacking elasticsearch distributions.
This has the benefit of
- handcrafted extract tasks on the root project are not required.
- The general tight coupling to the root project has been removed.
- The overall required configurations required to handle a distribution have been reduced
- ElasticsearchDistribution has been simplified by making Extracted an ordinary Configuration
downloaded and unpacked external distributions are reused in later builds by been cached
in the gradle user home.
DistributionDownloadPlugin functional tests have been extended and ported
to DistributionDownloadPluginFuncTest.
* Fix ElasticsearchNode#getDistributionFiles (#61219)
Fixes#61647
- Added test coverage
- Removes build script cluttering
- Splits archive building and archive checking logic
- only rely on boost for now for ML licenses(tbd)
- Use Gradle build-in untar and unzip support
* Handle dynamic versions in func tests assertions
Driven by this issue
https://github.com/elastic/elasticsearch/pull/60969#issuecomment-674962158
we apply some rework on how we handle distributions in our test cluster setups:
- If no custom modules, plugins or extra jar files are declared we do not create a cluster
specific distro folder and use the origin distribution folder instead.
- If a custom distribution folder is required, we fallback to file copy when hard linking
is not supported
Access the common versions map is done in a lot of places. While it can
be access through an import of VersionProperties, the vast majority of
places use it through the provided convenience property added by
BuildPlugin. This commit moves that convenience property to the base
java plugin, so further reduce dependence on the BuildPlugin.
Backport of #60742.
This PR resurrects support for building Docker images based on one of
Red Hat's UBI images. It also adds support for running the existing
Docker tests against the image. The image is named
`elasticsearch-ubi8:<version>`.
I also changed the Docker build file uses enums instead strings in a lot
of places, for added rigour.
We leverage artifact transforms now when downloading and unpacking elasticsearch distributions.
This has the benefit of
- handcrafted extract tasks on the root project are not required.
- the general tight coupling to the root project has been removed.
- the overall required configurations required to handle a distribution have been reduced
- ElasticsearchDistribution has been simplified by making Extracted an ordinary Configuration
downloaded and unpacked external distributions are reused in later builds by been cached
in the gradle user home.
DistributionDownloadPlugin functional tests have been extended and ported
to DistributionDownloadPluginFuncTest.
* Fix java8 compliant Path calculation
With Gradle 6.6 we can now use the native support for the --release compile options.
As Gradle by default resolves the compatibility version from the release property we needed to workaround this in order to keep our current setup. An issue was raised at gradle to track this at https://github.com/gradle/gradle/issues/14141
The reaper is used only when spawned processes by the build like
testclusters or vagrant are left behind in a failed build. When this
occurs on Windows, the Taskkill command is used. However, there was a
typo in the command name. Since the reaper is for edgecases and mostly
protection for local builds (CI uses immutable hosts), this has been
broken since its inception.
closes#60503
The Elasticsearch `run` task in Gradle is backed by testclusters
spinning up a node and waiting indefinitely on output from that node.
However, this thread currently waits for an interrupt to exit. If
Elasticsearch fails to start, we hang indefinitely. This commit adds a
check after each poll of the process stdout to see if it is still alive,
and fails the build if it has died.
closes#60761
Though bats tests were recently removed, there remains a few unnecessary
setups needed for those tests in our vagrant files, as well as CI setup.
Additionally, we no longer rely on the vagrant images setting a
JAVA_HOME, instead relying on DistroTestPlugin to pull the appropriate
jdk when testing with no-jdk distributions.
* Simplify jdk downloads via artifact transforms
This reworks the jdk download plugin to use artifact transforms.
That allows us to simplify the jdk download plugin by a lot. The
benefits of using artifact transforms are:
- they transform an artifact to an upacked folder on the fly as part
of the dependenc resolution allowing us to remove all the custom created unpack tasks
and configurations
- Artifact transforms support gradle build cache. Requesting a jdk folder on a clean machine
will likely be resolved from the build cache
- The manual mingling about not extracting jdks multiple times by all jdks channeling through
root project configurations can be removed as they support up-to-date checking and build cache
which will ensure these archives are only resolved once per build at max.
Also the test coverage has been ported to Spock that supports data driven testing. This porting
includes an introduction to a wiremock fixture that can be used later on for mocking repository
urls in other integration tests
* Simplify artifact transform registration
* Change jdk finalization and repository setup
Jdk finalization is now done via `configuration.defaultDependencies`.
This has two benefits:
- we only configure and finalize the JDKs we actually use during the build
- afterEvaluate hooks are problematic in combination with task avoidance api.
Relying on task avoidance api in Gradle moves more configuration logic into the materialization
of tasks which is, with task avoidance api, is done ideally during the task graph calculation.
Anything created (e.g. jdks) created late in this task graph calculation would never be finalized
via afterEvaluate hooks as this has been fired before already with the jdk not there yet.
Furthermore we now only configure repositories in the projects where the jdk is declared
(aka the jdk-download plugin is applied) and only if the jdk is actually requested during the build.
* Fix javadoc
* Fix jdk download repo content filtering
* Minor cleanup
* Apply review feedback and cleanup
This commit removes the last of the bats tests, converting the rpm/deb
upgrade tests to java. It adds a new pattern of tasks, similar in nature
but separate from the existing distro tests, named `distroUpgradeTest`.
For each index compatible version, a `distroUpgradeTest.VERSION` task
exxists. Each distribution then has a task, named
`distroUpgradeTest.VERSION.DISTRO`.
One thing to note is these new tests do not cover no-jdk versions of
the rpm/deb packages, since the distribution/bwc project does not
currently build those.
closes#59145closes#46005
* Merge test runner task into RestIntegTest (#60261)
* Merge test runner task into RestIntegTest
* Reorganizing Standalone runner and RestIntegTest task
* Rework general test task configuration and extension
* Fix merge issues
* use former 7.x common test configuration
- Replace immediate task creations by using task avoidance api
- One step closer to #56610
- Still many tasks are created during configuration phase. Tackled in separate steps
* Split internal distribution handling into separate internal plugin (#60295)
* Provide proper failure if unexpected non jdk bundled bwc version is requested
For all OSS plugins (except repository-* and discovery-*) integTest
task is now a no-op and all of the tests are now executed via a test,
yamlRestTest, javaRestTest, or internalClusterTest.
This commit does NOT convert the discovery-* and repository-* since they
are bit more complex then the rest of tests and this PR is large enough.
Those plugins will be addressed in a future PR(s).
This commit also fixes a minor issue that did not copy the rest api
for projects that only had YAML TEST tests.
related: #56841
This commit adds compatibility testing of our JDBC driver against
different Elasticsearch versions. Although we are really testing the
forwards compatibility nature of the JDBC driver we model the testing
the same as we do existing BWC tests, that is, with the current branch
fetching the earlier versions of the artifact that is to be tested. In
this case, that's the JDBC driver itself.
Because the tests include the JDBC driver jar on it's classpath we had
to change the packaging of the driver jar in order to avoid jarhell and
other conflicting dependency issues when using an old JDBC driver with
later branches. For this we simply relocate all driver dependencies in
the shadow jar under a "shadowed" package. This allows the JDBC driver
to use the correct version of Elasticsearch libs classes, while the
tests themselves use their versions. Since this required a change to the
driver jar compatibility testing can only go back as far as that version
which at the time of this commit is 7.8.1.
Introduce a javaRestTest source set and task to compliment the yamlRestTest.
javaRestTest differs such that the code is sourced from Java and may have
different dependencies and setup requirements for the test clusters. This also
allows the tests to run in parallel in different cluster instances to prevent any
cross test contamination between the two types of tests.
Included in this PR is all :modules no longer use the integTest task. The tests
are now driven by test, yamlRestTest, javaRestTest, and internalClusterTest.
Since only :modules (and :rest-api-spec) have been converted to yamlRestTest
we can now disable the integTest task if either yamlRestTest or javaRestTest have
been applied. Once all projects are converted, we can delete the integTest task.
related: #56841
related: #59444
Adrien Grand
Ignacio Vera
Rene Groeschke
Jake Landis
Ryan Ernst
Ioannis Kakavas
Jason Tedor
Rory Hunter
Mark Vieira
Yannick Welsch