The "old jdk" tests are just testing support for downloading from oracle
prior to java 12.0.1, when oracle added a hash to the url. This commit
moves these tests into the openjdk tests (ie oracle download tests),
since adoptopenjdk does not have any change in behavior that needs to be
tested.
The pom files for our published artifacts are sent to maven central
during Elastic's release process, but we may not found out until then
that we have inadvertently broken the pom structure, as has happened
several times before. This commit adds validation of the pom file
specifically for the rules required by maven central.
Currently forbidden apis accounts for 800+ tasks in the build. These
tasks are aggressively created by the plugin. In forbidden apis 3.0, we
will get task avoidance
(https://github.com/policeman-tools/forbidden-apis/pull/162), but we
need to ourselves use the same task avoidance mechanisms to not trigger
these task creations. This commit does that for our foribdden apis
usages, in preparation for upgrading to 3.0 when it is released.
Upgrade to lucene 8.5.1 release that contains a bug fix for a bug that might introduce index corruption when deleting data from an index that was previously shrunk.
Backport of #55073.
We added tasks to build an ARM distribution and Docker image, but didn't
provide any way to run packaging tests against them. Add extra loops on
the possible Architecture values, and skip tasks that can't be run on
the current Architecture.
This change converts the module and plugin parameters
for testClusters to be lazy. Meaning that the values
are not resolved until they are actually used. This
removes the requirement to use project.afterEvaluate to
be able to resolve the bundle artifact.
Note - this does not completely remove the need for afterEvaluate
since it is still needed for the custom resource extension.
The secure_settings_password was never taken into consideration in
the ReloadSecureSettings API. This commit fixes that and adds
necessary REST layer testing. Doing so, it also:
- Allows TestClusters to have a password protected keystore
so that it can be set for tests.
- Adds a parameter to the run task so that elastisearch can
be run with a password protected keystore from source.
This commit includes a number of changes to reduce overall build
configuration time. These optimizations include:
- Removing the usage of the 'nebula.info-scm' plugin. This plugin
leverages jgit to load read various pieces of VCS information. This
is mostly overkill and we have our own minimal implementation for
determining the current commit id.
- Removing unnecessary build dependencies such as perforce and jgit
now that we don't need them. This reduces our classpath considerably.
- Expanding the usage lazy task creation, particularly in our
distribution projects. The archives and packages projects create
lots of tasks with very complex configuration. Avoiding the creation
of these tasks at configuration time gives us a nice boost.
ForbiddenApis task via the precommit task currently makes an assumption
that only the test and main source sets are present for any given project.
This commit removes that assumption and allows for any project source set's
compileClasspath class path to be added to the forbiddenApis classpath
configuration.
Guava was removed from Elasticsearch many years ago, but remnants of it
remain due to transitive dependencies. When a dependency pulls guava
into the compile classpath, devs can inadvertently begin using methods
from guava without realizing it. This commit moves guava to a runtime
dependency in the modules that it is needed.
Note that one special case is the html sanitizer in watcher. The third
party dep uses guava in the PolicyFactory class signature. However, only
calling a method on the PolicyFactory actually causes the class to be
loaded, a reference alone does not trigger compilation to look at the
class implementation. There we utilize a MethodHandle for invoking the
relevant method at runtime, where guava will continue to exist.
Change how we format exceptions to only wrap them as necessary. While
the config's overall philosophy is to put items one-per-line when
wrapping, in practice this is a little cumbersome for exception lists.
This commit migrates the RestIntegTestTask from groovy to Java.
No changes to logic should be included, however the following changes
are needed:
* Move Fixture interface to Java (Java can not depend on Groovy classes)
* Support lazy evaluation of non-input System parameters (can not use Groovy strings)
* Use constants for system property names
* Remove dead System property pass through code (the build plugin does this already)
Drop a nasty regex in our checkstyle config that I wrote a long time ago
in favor of a checkstyle extension. This is better because:
* It is faster. It saves a little more than a minute across the entire
build.
* It is easier to read. Who knew 100 lines of Java would be easier to
read than a regex, but it is.
* It has tests.
Backport of #54276.
Move and rename formatter config file, so that it is easier for
Eclipse users to import.
Also switch to an opt-out list for formatting. Instead of explcitly
listing projects that should be formatted, instead list projects
that should not be formatted. This means that any new projects will
automatically be formatted and checked.
Xpack license state contains a helper method to determine whether
security is disabled due to license level defaults. Most code needs to
know whether security is enabled, not disabled, but this method exists
so that the security being explicitly disabled can be distinguished from
licence level defaulting to disabled. However, in the case that security
is explicitly disabled, the handlers in question are never registered,
so security is implicitly not disabled explicitly, and thus we can share
a single method to know whether licensing is enabled.
This commit removes the configuration time vs execution time distinction
with regards to certain BuildParms properties. Because of the cost of
determining Java versions for configuration JDK locations we deferred
this until execution time. This had two main downsides. First, we had
to implement all this build logic in tasks, which required a bunch of
additional plumbing and complexity. Second, because some information
wasn't known during configuration time, we had to nest any build logic
that depended on this in awkward callbacks.
We now defer to the JavaInstallationRegistry recently added in Gradle.
This utility uses a much more efficient method for probing Java
installations vs our jrunscript implementation. This, combined with some
optimizations to avoid probing the current JVM as well as deferring
some evaluation via Providers when probing installations for BWC builds
we can maintain effectively the same configuration time performance
while removing a bunch of complexity and runtime cost (snapshotting
inputs for the GenerateGlobalBuildInfoTask was very expensive). The end
result should be a much more responsive build execution in almost all
scenarios.
(cherry picked from commit ecdbd37f2e0f0447ed574b306adb64c19adc3ce1)
This commit introduces aarch64 packaging, including bundling an aarch64
JDK distribution. We had to make some interesting choices here:
- ML binaries are not compiled for aarch64, so for now we disable ML on
aarch64
- depending on underlying page sizes, we have to disable class data
sharing
When depending on lucene snapshots we point maven at our own s3 backed
repository. However, in this case lucene packages should only be
retrieved from this location, and no other packages should ever be found
in that repo. This commit makes the maven repository exclusive to lucene
packages.
Prior to this commit Watcher explicitly copied test between two
projects with a copy task. This commit removes the explicit copy in favor
of adding the Watcher tests to the available restResources that may be
copied between projects.
This is how inter-project dependencies should be modeled. However, only
Watcher is included here since it is (currently) the only project with
inter-project test dependencies.
Re-applies the change from #53523 along with test fixes.
closes#53626closes#53624closes#53622closes#53625
Co-authored-by: Nik Everett <nik9000@gmail.com>
Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>
Co-authored-by: Jake Landis <jake.landis@elastic.co>
The multiline regex rule used to detect docs code snippets greater than
76 characters in length has considerable cost to. For the high level
rest client project alone, this was taking upwards of 3 minutes. This
commit updates the rule regex pattern to use non-greedy matching when
appropriate which results in a lot fewer backtracks and about a 70%
reduction in execution time on the high level rest client module.
This commit upgrades the jackson-databind depdendency to
2.8.11.6. Additionally, we revert a previous change that put
ingest-geoip on the version of jackson-databind from the version
properties file. This is because upgrading ingest-geoip to a later
version of jackson-databind also requires an upgrade to the geoip2
dependency which is currently blocked. Therefore, if we can get to a
point where we otherwise upgrade our Jackson dependencies, we do not
want ingest-geoip to automatically come along with it.
The jdk and distribution download plugins create fake ivy repositories,
and use group based repository filtering to ensure no other artifacts
try to resolve against the fake repos. Currently this works by adding a
blanket exclude to all repositories for the given group name. This
commit changes to using the new exclusiveContent feature in
Gradle to do the exclusion.
Lucene 8.5.0 release candidates are imminent. This commit upgrades master to use
the latest snapshot to check that there are no last-minute bugs or regressions.
This commit fixes ensures that for external builds
(e.g. plugin development) that the REST tests that are
copied are properly filtered to only include the API
by default.
The code prior to this change resulted in including both
the API and tests since the copy.include resulted as an
empty list by default since the stream is empty unless
explicitly configured.
related #52114fixes#53183
A recent PR #52114 introduced two new tasks to copy the REST api and tests.
A couple bugs were found in that initial PR that prevents the incremental
build from working as expected.
The pattern match of empty string is equivalent to match all and it was coded
as match none. Fixed with explicit checks against empty patterns.
The fileCollection.plus return value was ignored. Fixed by changing how the
input's fileTree is constructed.
If a project has an src/test/resources directory, and tests are being copied
without a rest-api-spec/test directory could result no-op. Masked by the other
bugs and fixed by minor changes to logic to determine if a project has tests.
We embed the :reaper project jar in the build-tools jar so we can spawn
a reaper process at build runtime. Due to this, the jar technically
isn't part of the test runtime classpath, but for input snapshotting
purposes, we should be treating it as such. Instead, because it lives
in META-INF, Gradle treats it as a normal file, which in practice means
its hash changes on every build (timestamps, etc).
This commit changes our input snapshotting strategy such that instead
we explicitly add the jar as an input to any test tasks using Gradle's
runtime classpath normalization strategy (ignore timestamps, jar entry
order, etc) and ignore the file in META-INF. This ensures that we can
properly cache test results for build-tools, why still ensuring that
changes to the :reaper project trigger reexecution of tests.
Ryan Ernst
Ignacio Vera
Rory Hunter
Mark Vieira
Jake Landis
Ioannis Kakavas
Jason Tedor
James Baiera
Bogdan Pintea
Alan Woodward
Armin Braun
Luca Cavanna
Nhat Nguyen
JaeGeun