This commit migrates the RestIntegTestTask from groovy to Java.
No changes to logic should be included, however the following changes
are needed:
* Move Fixture interface to Java (Java can not depend on Groovy classes)
* Support lazy evaluation of non-input System parameters (can not use Groovy strings)
* Use constants for system property names
* Remove dead System property pass through code (the build plugin does this already)
Drop a nasty regex in our checkstyle config that I wrote a long time ago
in favor of a checkstyle extension. This is better because:
* It is faster. It saves a little more than a minute across the entire
build.
* It is easier to read. Who knew 100 lines of Java would be easier to
read than a regex, but it is.
* It has tests.
Backport of #54276.
Move and rename formatter config file, so that it is easier for
Eclipse users to import.
Also switch to an opt-out list for formatting. Instead of explcitly
listing projects that should be formatted, instead list projects
that should not be formatted. This means that any new projects will
automatically be formatted and checked.
Xpack license state contains a helper method to determine whether
security is disabled due to license level defaults. Most code needs to
know whether security is enabled, not disabled, but this method exists
so that the security being explicitly disabled can be distinguished from
licence level defaulting to disabled. However, in the case that security
is explicitly disabled, the handlers in question are never registered,
so security is implicitly not disabled explicitly, and thus we can share
a single method to know whether licensing is enabled.
This commit removes the configuration time vs execution time distinction
with regards to certain BuildParms properties. Because of the cost of
determining Java versions for configuration JDK locations we deferred
this until execution time. This had two main downsides. First, we had
to implement all this build logic in tasks, which required a bunch of
additional plumbing and complexity. Second, because some information
wasn't known during configuration time, we had to nest any build logic
that depended on this in awkward callbacks.
We now defer to the JavaInstallationRegistry recently added in Gradle.
This utility uses a much more efficient method for probing Java
installations vs our jrunscript implementation. This, combined with some
optimizations to avoid probing the current JVM as well as deferring
some evaluation via Providers when probing installations for BWC builds
we can maintain effectively the same configuration time performance
while removing a bunch of complexity and runtime cost (snapshotting
inputs for the GenerateGlobalBuildInfoTask was very expensive). The end
result should be a much more responsive build execution in almost all
scenarios.
(cherry picked from commit ecdbd37f2e0f0447ed574b306adb64c19adc3ce1)
This commit introduces aarch64 packaging, including bundling an aarch64
JDK distribution. We had to make some interesting choices here:
- ML binaries are not compiled for aarch64, so for now we disable ML on
aarch64
- depending on underlying page sizes, we have to disable class data
sharing
When depending on lucene snapshots we point maven at our own s3 backed
repository. However, in this case lucene packages should only be
retrieved from this location, and no other packages should ever be found
in that repo. This commit makes the maven repository exclusive to lucene
packages.
Prior to this commit Watcher explicitly copied test between two
projects with a copy task. This commit removes the explicit copy in favor
of adding the Watcher tests to the available restResources that may be
copied between projects.
This is how inter-project dependencies should be modeled. However, only
Watcher is included here since it is (currently) the only project with
inter-project test dependencies.
Re-applies the change from #53523 along with test fixes.
closes#53626closes#53624closes#53622closes#53625
Co-authored-by: Nik Everett <nik9000@gmail.com>
Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>
Co-authored-by: Jake Landis <jake.landis@elastic.co>
The multiline regex rule used to detect docs code snippets greater than
76 characters in length has considerable cost to. For the high level
rest client project alone, this was taking upwards of 3 minutes. This
commit updates the rule regex pattern to use non-greedy matching when
appropriate which results in a lot fewer backtracks and about a 70%
reduction in execution time on the high level rest client module.
This commit upgrades the jackson-databind depdendency to
2.8.11.6. Additionally, we revert a previous change that put
ingest-geoip on the version of jackson-databind from the version
properties file. This is because upgrading ingest-geoip to a later
version of jackson-databind also requires an upgrade to the geoip2
dependency which is currently blocked. Therefore, if we can get to a
point where we otherwise upgrade our Jackson dependencies, we do not
want ingest-geoip to automatically come along with it.
The jdk and distribution download plugins create fake ivy repositories,
and use group based repository filtering to ensure no other artifacts
try to resolve against the fake repos. Currently this works by adding a
blanket exclude to all repositories for the given group name. This
commit changes to using the new exclusiveContent feature in
Gradle to do the exclusion.
Lucene 8.5.0 release candidates are imminent. This commit upgrades master to use
the latest snapshot to check that there are no last-minute bugs or regressions.
This commit fixes ensures that for external builds
(e.g. plugin development) that the REST tests that are
copied are properly filtered to only include the API
by default.
The code prior to this change resulted in including both
the API and tests since the copy.include resulted as an
empty list by default since the stream is empty unless
explicitly configured.
related #52114fixes#53183
A recent PR #52114 introduced two new tasks to copy the REST api and tests.
A couple bugs were found in that initial PR that prevents the incremental
build from working as expected.
The pattern match of empty string is equivalent to match all and it was coded
as match none. Fixed with explicit checks against empty patterns.
The fileCollection.plus return value was ignored. Fixed by changing how the
input's fileTree is constructed.
If a project has an src/test/resources directory, and tests are being copied
without a rest-api-spec/test directory could result no-op. Masked by the other
bugs and fixed by minor changes to logic to determine if a project has tests.
We embed the :reaper project jar in the build-tools jar so we can spawn
a reaper process at build runtime. Due to this, the jar technically
isn't part of the test runtime classpath, but for input snapshotting
purposes, we should be treating it as such. Instead, because it lives
in META-INF, Gradle treats it as a normal file, which in practice means
its hash changes on every build (timestamps, etc).
This commit changes our input snapshotting strategy such that instead
we explicitly add the jar as an input to any test tasks using Gradle's
runtime classpath normalization strategy (ignore timestamps, jar entry
order, etc) and ignore the file in META-INF. This ensures that we can
properly cache test results for build-tools, why still ensuring that
changes to the :reaper project trigger reexecution of tests.
* Smarter copying of the rest specs and tests (#52114)
This PR addresses the unnecessary copying of the rest specs and allows
for better semantics for which specs and tests are copied. By default
the rest specs will get copied if the project applies
`elasticsearch.standalone-rest-test` or `esplugin` and the project
has rest tests or you configure the custom extension `restResources`.
This PR also removes the need for dozens of places where the x-pack
specs were copied by supporting copying of the x-pack rest specs too.
The plugin/task introduced here can also copy the rest tests to the
local project through a similar configuration.
The new plugin/task allows a user to minimize the surface area of
which rest specs are copied. Per project can be configured to include
only a subset of the specs (or tests). Configuring a project to only
copy the specs when actually needed should help with build cache hit
rates since we can better define what is actually in use.
However, project level optimizations for build cache hit rates are
not included with this PR.
Also, with this PR you can no longer use the includePackaged flag on
integTest task.
The following items are included in this PR:
* new plugin: `elasticsearch.rest-resources`
* new tasks: CopyRestApiTask and CopyRestTestsTask - performs the copy
* new extension 'restResources'
```
restResources {
restApi {
includeCore 'foo' , 'bar' //will include the core specs that start with foo and bar
includeXpack 'baz' //will include x-pack specs that start with baz
}
restTests {
includeCore 'foo', 'bar' //will include the core tests that start with foo and bar
includeXpack 'baz' //will include the x-pack tests that start with baz
}
}
```
We explicitly set the path for the temporary directory to use in test
tasks, but today this path is a relative path, relative to the current
working directory of the test task. The fact that we are using a
relative path here appears to be legacy, simply leftover from the days
of the Maven build. An absolute path is preferred here, since it's
explicit and we do not have to rely on everyone resolving the path
properly relative to the working directory.
Today we we set the test temporary directory explicitly by controling
java.io.tmpdir. Yet, we do not guarantee this directory exists, instead
relying on a test base class (LuceneTestCase) to create this directory
when it initializes. However, some of our tests do not rely on our test
framework, and thus do not have access to LuceneTestCase, instead
relying on RandomizedRunner directly. We should not be relying on the
temporary directory being implicitly created, instead guaranteeing that
it exists before test execution starts. This commit does that by
creating the test temporary directory before the test task executes (via
a doFirst).
This commit converts the sysv init tests from bats tests into the java
packaging tests. Since it is the last oss specific test, the bats oss
test task is also removed.
relates #46005
When docker-compose is required for a test fixture but is not
available, we warn log a message to this effect. This ends up being
noise during configuration, especially when working locally. This
commit changes the logging level of these messages to debug.
Jake Landis
Mark Vieira
Rory Hunter
Ryan Ernst
Bogdan Pintea
Alan Woodward
Jason Tedor
Armin Braun
Luca Cavanna
Nhat Nguyen
Ignacio Vera
JaeGeun
markharwood
Tim Brooks
Ioannis Kakavas
Ignacio Vera