Commit Graph

724 Commits

Author SHA1 Message Date
Mark Vieira 84eab4eba1
Omit JDK sources archive from bundled JDK (#42821)
(cherry picked from commit 71d1454fe5ecc222801731a5f0e0e1053dc8997e)
2019-06-05 10:09:25 -07:00
Przemyslaw Gomulka cfdb1b771e
Enable console audit logs for docker backport#42671 #42887
Enable audit logs in docker by creating console appenders for audit loggers.
also rename field @timestamp to timestamp and add field type with value audit

The docker build contains now two log4j configuration for oss or default versions. The build now allows override the default configuration.

Also changed the format of a timestamp from ISO8601 to include time zone as per this discussion #36833 (comment)

closes #42666
backport#42671
2019-06-05 17:15:37 +02:00
Mark Vieira e44b8b1e2e
[Backport] Remove dependency substitutions 7.x (#42866)
* Remove unnecessary usage of Gradle dependency substitution rules (#42773)

(cherry picked from commit 12d583dbf6f7d44f00aa365e34fc7e937c3c61f7)
2019-06-04 13:50:23 -07:00
Mark Vieira c1816354ed
[Backport] Improve build configuration time (#42674) 2019-05-30 10:29:42 -07:00
Alpar Torok 4dbf6c0df9 Make packer cache branches explicit (#41990)
Before this change we would recurse to cache bwc versions.
This proved to be problematic due to  the number of steps it was
generating taking too long.
Also this required tricky maintenance to break the recursion for old
branches we don't really care about.

With this change we now cache specific branches only.
2019-05-27 09:46:43 +03:00
Ryan Ernst 8681dd9cba Hide bwc build output on success (#42102)
Previously we used LoggedExec for running the internal bwc builds.
However, this had bad performance implications as all the output was
buffered into memory, thus we changed back to normal Exec. This commit
adds a `spoolOutput` setting to LoggedExec which can be used for
commands with large amounts of output, and switches the bwc builds to
use this flag.
2019-05-16 09:49:23 -04:00
Ryan Ernst 9944fdf237 Don't create tempdir for cli scripts (#41913)
The elasticsearch-cli helper script does not use the tempdir created by
elasticsearch-env, yet the env script still creates it. This can lead to
lots of temp directories being created when running cli scripts in an
automated fashion. This commit passes a fake tmpdir to the env script to
avoid creation.

closes #34445
2019-05-10 11:17:12 -07:00
Ryan Ernst 69824ed908 Cleanup plugin bin directories (#41907)
This commit adds deletion of the bin directory to postrm cleanup. While
the package's bin files are cleaned up by the package manager, plugins
may have created subdirectories under bin. We already cleanup plugins,
but not the extra bin dirs their installation created.

closes #18109
2019-05-10 11:00:41 -07:00
Jason Tedor 598e0962ed
Fix compilation in JVMErgonomicsTests
This issue arose after a cherry-pick from the wrong branch. Sorry. This
commit addresses the issue.
2019-05-09 10:24:16 -04:00
Jason Tedor 970a2254c3
Limit max direct memory size to half of heap size (#42006)
This commit adds an ergonomic choice ot the max direct memory size such
that if it is not set, we default it to half of the heap size.
2019-05-09 10:10:56 -04:00
Jason Tedor 7ee541546a
Account for Java 8 in JVM options parsing
Java 8 presents the JVM options slightly differently when displaying via
-XX:+PrintFlagsFinal. This commit adapts the JVM options parser for this
possibility.

Relates #42009
2019-05-09 09:20:45 -04:00
Jason Tedor 37771502ae
Remove manual parsing of JVM options (#41962)
This commit removes manual parsing of JVM options when calculating
ergonomics. This is to avoid a situation that we parse values
differently than the JVM would. In fact, we already have a bug along
these lines today. It is possible to start the JVM with the same flag
multiple times on the command line. In this case, the last value
wins. For example, -Xmx1g -Xmx2g would start the JVM with a heap size of
two gigabytes. Our JVM ergonomics ignores this possibility and instead
the first value is winning!

Our strategy to avoid manual parsing of the JVM options is to start the
Java command line parser (without actually starting a JVM) by invoking
java with the same command line flags as presented and request that the
JVM tell us what values it would start with. This ensures that we have
the correct values when making ergonomic decisions.

Moreover, our strategy also is ignoring ES_JAVA_OPTS which could
override the heap size as well leading to incorrect ergonomic
choices. This commit address this issue too.
2019-05-09 06:44:04 -04:00
Ryan Ernst 6343ec3d3e
Update lintian overrides (#41561) (#41953)
The deb package has been updated several times in the past to contain
overrides in order to pass lintian inspection. However, there have never
been any tests to ensure we do not fallback to failure. This commit
updates the overrides file given things that have changed since 2.x like
adding ML and bundling the jdk.

closes #17185
2019-05-08 12:09:02 -07:00
Ryan Ernst 374ce3e6a8 Add gradle plugin for downloading jdk (#41461)
We currently download 3 variants of the same version of the jdk for
bundling into the distributions. Additionally, the vagrant images do
their own downloading. This commit moves the jdk downloading into a
utility gradle plugin. This will be used in a future PR by the packaging
tests.

The new plugin exposes a "jdks" project extension which allows creating
named jdks. Once the jdk version and platform are set for a named jdk,
the jdk object may be used as a lazy String for the jdk home path, or a
file collection for copying.
2019-05-08 07:16:44 -07:00
Alpar Torok 711ace0533 Testclusters: support for security and convert example plugins (#41864)
testclusters detect from settings that security is enabled
if a user is not specified using the DSL introduced in this PR, a default one is created
the appropriate wait conditions are used authenticating with the first user defined in the DSL ( or the default user ).
an example DSL to create a user is user username:"test_user" password:"x-pack-test-password" role: "superuser" all keys are optional and default to the values shown in this example
2019-05-08 14:04:00 +03:00
Jason Tedor d7fd51a84e
Provide names for all artifact repositories (#41857)
This commit adds a name for each Maven and Ivy repository used in the
build.
2019-05-07 06:35:28 -04:00
Jason Tedor c808badb23
Add tasks to build Docker build context artifacts (#41819)
This commit adds some tasks to generate dedicated Docker build context
artifacts.
2019-05-06 21:04:57 -04:00
Jason Tedor ff317d0d7f
Set metadata sources for Ivy repositories (#41818)
We have faked some Ivy repositories on a few artifact locations. Today
when Gradle attempts to resolve these artifacts, it follows its default
strategy to search for Gradle metadata, then Maven POM files, then Ivy
descriptors, and finally will fallback to looking directly for the
artifact. This wastes times on remote network calls that will 404 anyway
since these metadata resources will not exist for these fake Ivy
repositories. This commit overrides the Gradle strategy to look directly
for artifacts.
2019-05-05 11:16:52 -04:00
Jason Tedor 03c959f188
Upgrade keystore on package install (#41755)
When Elasticsearch is run from a package installation, the running
process does not have permissions to write to the keystore. This is
because of the root:root ownership of /etc/elasticsearch. This is why we
create the keystore if it does not exist during package installation. If
the keystore needs to be upgraded, that is currently done by the running
Elasticsearch process. Yet, as just mentioned, the Elasticsearch process
would not have permissions to do that during runtime. Instead, this
needs to be done during package upgrade. This commit adds an upgrade
command to the keystore CLI for this purpose, and that is invoked during
package upgrade if the keystore already exists. This ensures that we are
always on the latest keystore format before the Elasticsearch process is
invoked, and therefore no upgrade would be needed then. While this bug
has always existed, we have not heard of reports of it in practice. Yet,
this bug becomes a lot more likely with a recent change to the format of
the keystore to remove the distinction between file and string entries.
2019-05-03 10:34:30 -04:00
Yogesh Gaikwad 719e4452af Revert "Suppress illegal access in plugin install (#41620)"
This reverts commit fb9f729426.
2019-04-29 13:56:31 +10:00
Jason Tedor fb9f729426
Suppress illegal access in plugin install (#41620)
We use Bouncy Castle to verify signatures when installing official
plugins. This leads to illegal access warnings because Bouncy Castle
accesses the Sun security provider constructor. This commit adds an
add-opens flag to suppress this illegal access.
2019-04-28 23:04:32 -04:00
Jason Tedor 3b3f7b7e34
Bump the bundled JDK to 12.0.1 (#41627)
This commit bumps the bundled JDK to version 12.0.1. Note that we had to
add a new pattern here as Oracle has changed the source of the
builds. This commit will be backported to 6.7 in a different form to
bump the bundled JDK in the Docker images too.
2019-04-28 21:04:42 -04:00
Jason Tedor 7a82277bc3
Change JDK distribution source (#41626)
We had been obtaining JDK distributions from download.java.net. This
site is now presenting a certificate that does not list
download.java.net as a SAN. Therefore with host verification, the build
can not use this site. This commit switches to using download.oracle.com
which appears to be an alternative name for the same CNAME
download.oracle.com.edgekey.net. This allows our builds to resume.
2019-04-28 20:16:06 -04:00
Ryan Ernst 7e3875d781 Upgrade hamcrest to 2.1 (#41464)
hamcrest has some improvements in newer versions, like FileMatchers
that make assertions regarding file exists cleaner. This commit upgrades
to the latest version of hamcrest so we can start using new and improved
matchers.
2019-04-24 23:40:03 -07:00
Alpar Torok 58dd88329f Dix the packer cache script 2019-04-24 11:35:27 +03:00
Ryan Ernst cc867e8a5a
Remove /var/run/elasticsearch from packages (#41102)
The pid dir for both systemd and init.d is already managed by those
respective systems (tmpfiles.d and the init script, respectively). Since
the /var/run dir is often mounted as tmpfs, it does not make sense to
have the elasticsearch pid dir added by the package installation. This
commit removes that empty dir from deb and rpm.
2019-04-22 09:57:51 -07:00
Ryan Ernst c0164cbb63 Only include relevant platform files from modules (#41089)
This commit adds a filter to the files include from modules to only
include platform specific files relevant to the distribution being
built. For example, the deb files on linux would now only include linux
ML binaries, and not windows or macos files.
2019-04-19 11:34:50 -07:00
Alpar Torok 45c151ff8b Disable composePull only if it exists (#41306)
The task will not be created when docker is not available.
2019-04-18 09:54:01 +03:00
Alpar Torok 84e2f9d8ec fix the packer cache script (#41183)
* fix the packer cache script

This PR disabled the explicit pull since it seems this always tries to
work with a registry.
Functionality will not be affected since we will still build the images
on pull.
2019-04-17 09:04:24 +03:00
Alpar Torok bee892006a Use the built image in docker tests (#40314)
Instead of allowing docker-compose to rebuild it.
With this change we tag the image with a test label, and use that
in the testing as this is simpler that dealing with a dynamically
generated docker-compose file.
2019-04-12 12:34:55 +03:00
Ryan Ernst 884c3fd7ca
Use separate tasks per bwc artifact (#40703) (#40826)
This commit changes the bwc builds from a single task for a branch to a
task for each bwc artifact. This reduces the bwc build time when only
needing a specific artifact, for example when running cluster restart
tests on a mac, the windows artifacts or rpm/debs are not needed.
2019-04-10 10:43:01 -07:00
Jason Tedor c0f715b337
Fix Docker build when sourced from artifacts
This commit fixes an issue when the artifact used to build the Docker
image is sourced from artifacts.elastic.co. In particular, the artifact
was not downloaded to the proper location.
2019-04-09 20:33:15 -04:00
Mark Vieira 1287c7d91f
[Backport] Replace usages RandomizedTestingTask with built-in Gradle Test (#40978) (#40993)
* Replace usages RandomizedTestingTask with built-in Gradle Test (#40978)

This commit replaces the existing RandomizedTestingTask and supporting code with Gradle's built-in JUnit support via the Test task type. Additionally, the previous workaround to disable all tasks named "test" and create new unit testing tasks named "unitTest" has been removed such that the "test" task now runs unit tests as per the normal Gradle Java plugin conventions.

(cherry picked from commit 323f312bbc829a63056a79ebe45adced5099f6e6)

* Fix forking JVM runner

* Don't bump shadow plugin version
2019-04-09 11:52:50 -07:00
Ioannis Kakavas aea4e6596f
Source additional files correctly in elasticsearch-cli (#40890)
Since we only source additional sources from the same dir as our
cli scripts, resolve the path relevant to $ES_HOME
2019-04-05 17:42:34 -04:00
Jason Tedor 2fd1689341
Allow AVX-512 on JDK 11+ (#40828)
We previously found a bug in the JVM where AVX-512 instructions could
crash the JVM to crash with a segmentation fault. This bug impacted JDK
9 and JDK 10, but was most prominent on JDK 10 because AVX-512 was
enabled there by default. In JDK 11, this bug is reported fixed so this
commit restricts the disabling of AVX-512 to JDK 10 only. Since we no
longer support JDK 10 for any versions that this commit will be
integrated into (7.1, 8.0), we simply remove the disabling of this flag
from the JVM options.
2019-04-05 16:40:31 -04:00
Przemyslaw Gomulka 65d25186d3
Bat scripts to work with JAVA_HOME with parentheses Backports (#40832)
backports :
Bat scripts to work with JAVA_HOME with parentheses (#39712)
Link to SYSTEM_JAVA_HOME on windows (#40806)
2019-04-04 18:42:05 +02:00
Jason Tedor df65e46d10
Deprecate versions of Java prior to Java 11 (#40756)
This commit deprecates versions of Java prior to Java 11. This commit
will cause a warning to be printed to standard error when any command
line tool is invoked, or when Elasticsearch is started. Additionally, we
log a deprecation message when Elasticsearch is started.
2019-04-03 06:39:40 -04:00
Ryan Ernst 803ff3bc4c
Add notice for bundled jdk (#40576)
* Add notice for bundled jdk

This commit adds the license/notice for the bundled openjdk.

* First draft

* iteration

* Fix package notices

* Iteration

* One more iteration
2019-03-29 17:21:21 -04:00
Jason Tedor 585f38787c
Add usage indicators for the bundled JDK (#40616)
This commit adds indications whether or not a distribution is from the
bundled JDK, and whether or not we are using the bundled JDK.
2019-03-29 08:25:32 -04:00
Dimitrios Liappis 856789f791
Wrap Dockerfile yum operations in retries (#40460)
While yum does retry retrieving files 10 times by default [1], slow
network fetches, governed by `minrate` cause immediate aborts without
getting retried.

Wrap yum commands in a 10 iteration retry loop.

[1] http://man7.org/linux/man-pages/man5/yum.conf.5.html

Backport of #40349
2019-03-27 14:14:47 +02:00
Ryan Ernst aa24669e95
Resolve JAVA_HOME at windows service install time (#39714)
On windows, JAVA_HOME is currently resolved when the windows service is started. However, this is contrary to what our documentation states. This commit moves resolution to service install. This has the side effect of making java existence checking optional in elasticsearch-env.bat, since the rest of the service commands do not require java.

closes #30720
2019-03-26 19:48:02 -07:00
Ryan Ernst 562ebaefa6 Delete jdk extract dir before extracting (#40471)
This commit first deletes the directory we extract the bundled jdk into in order to make gradle's copy task happy.
2019-03-26 11:41:37 -07:00
Jay Modi 9bd8600c2e
Use ephemeral ports for idp-fixture (#40333)
This change removes the use of hardcoded port values for the
idp-fixture in favor of the mapped ephemeral ports. This should prevent
failures due to port conflicts in CI.
2019-03-26 08:44:53 -06:00
Alpar Torok b5a8de9a7f Revert tmpdir (#40312)
* Revert "Configure TMP for test nodes on Windows (#39959)"
This reverts commit 97562a874fcb1f29fb05272ab860a0307e97d1aa.

* Configure a tmp dir without spaces
* Pass on TMP instead of changing it
2019-03-26 13:31:18 +02:00
Alpar Torok b52d44cc98 Fix building bwc versions (#40361)
Account for openjdk when figuring out what java version to
use for building bwc versions.
2019-03-22 11:27:53 -06:00
Jason Tedor 68d98c85c9
Use bundled JDK in Docker images (#40238)
Now that we have the bundled JDK in the Docker images, we should use
them as opposed to procuring a JDK ourselves. This commit replaces the
JDK in the Docker image with the bundled JDK.
2019-03-20 08:53:38 -04:00
Jason Tedor c2a566a64e
Upgrade bundled JDK and Docker images to JDK 12 (#40229)
This commit bumps the JDK used the bundled JDK and the Docker images to
JDK 12.
2019-03-19 23:45:07 -04:00
Andrey Ershov d35d9d1886 cd ES_HOME in elasticsearch-env (#39937)
This commit adds cd $ES_HOME to elasticsearch-env and removes it from
elasticsearch. This way, both elasticsearch and elasticsearch-cli are
executed with the working directory set to $ES_HOME. The need for the
fix arose from the following bug:
1. Explicitly set path.data to relative to ES_HOME path in
elasticsearch.yml.
2. Run elasticsearch from any directory. Elasticsearch is able to
correctly start.
3. Stop elasticsearch.
4. Run elasticsearch-node unsafe-bootstrap, not from ES_HOME directory.
It will fail with an exception.
This commit fixes the issue and adds a new test.

This PR fixes the issue and adds a new test.
Also tests >=100 are renamed because alphabetic order does not work for
them.

(cherry picked from commit 2ffc29306ff7366efc598e7b4dd2ce528895cd3a
with fixes by #40083 and #40118)
2019-03-18 16:26:27 +01:00
Ryan Ernst 8f09c77777 Add no-jdk distributions (#39882)
This commit adds a variant for every official distribution that omits
the bundled jdk. The "no-jdk" naming is conveyed through the package
classifier, alongside the platform. Package tests are also added for
each new distribution.
2019-03-15 00:55:57 -07:00
Jason Tedor d02bca1314
Upgrade the bouncycastle dependency to 1.61 (#40017)
This commit upgrades the bouncycastle dependency from 1.59 to 1.61.
2019-03-14 08:54:47 -04:00