Commit Graph

1698 Commits

Author SHA1 Message Date
Jason Tedor 3b3f7b7e34
Bump the bundled JDK to 12.0.1 (#41627)
This commit bumps the bundled JDK to version 12.0.1. Note that we had to
add a new pattern here as Oracle has changed the source of the
builds. This commit will be backported to 6.7 in a different form to
bump the bundled JDK in the Docker images too.
2019-04-28 21:04:42 -04:00
Alpar Torok 335f2bf102 Testclsuters: convert plugins qa projects (#41496)
Add testclusters support for files in keystore and convert qa subprojects within plugins.
2019-04-26 08:57:52 -07:00
David Kyle 31d5a5aa36 Revert "Clean up clusters between tests (#41187)"
This reverts commit 44a0c468cf.
2019-04-25 12:08:14 +01:00
Ryan Ernst 7e3875d781 Upgrade hamcrest to 2.1 (#41464)
hamcrest has some improvements in newer versions, like FileMatchers
that make assertions regarding file exists cleaner. This commit upgrades
to the latest version of hamcrest so we can start using new and improved
matchers.
2019-04-24 23:40:03 -07:00
Jason Tedor 38b43c4a56
Limit the number of forks getting Java versions (#41386)
To reduce configuration time, we fork some threads to compute the Java
version for the various configured Javas. However, as the number of
JAVA${N}_HOME variable increases, the current implementation creates as
many threads as there are such variables, which could be more than the
number of physical cores on the machine. It is not likely that we would
see benefits to trying to execute all of these once beyond the number of
physical cores (maybe simultaneous multi-threading helps though, who
knows. Therefore, this commit limits the parallelization here to the
number number of physical cores.
2019-04-19 17:16:19 -04:00
Jason Tedor 154d40494f
Fix build issue if no specific Java version are set (#41379)
If no Java versions are set then when we size the executor thread pool
we end up with zero threads, which is illegal. This commit avoids that
problem by only starting the executor when needed.
2019-04-19 14:09:10 -04:00
Alpar Torok e1e2568fa3 Add FIPS specific testclusters configuration (#41199)
ClusterFormationTasks auto configured these properties for clusters.
This PR adds FIPS specific configuration across all test clusters from
the main build script to prevent coupling betwwen testclusters and the
build plugin.

Closes #40904
2019-04-19 10:36:54 +03:00
Alpar Torok 2f8bbce85d Clean up build tool dependencies (#41336)
We are no longer using these dependencies.

Relates to #41061 since the class that seems to be leaking is both part
of Gradle and the logging jar.
2019-04-19 09:22:38 +03:00
Alpar Torok 1f91759bc7 Improove the configuration time if the build (#41251)
This will help with reproduction lines and running tests form IDEs and
other operations that are quick and executed often enough for the
configuration time to matter.

Running Gradle with a FIPS JVM is not supproted, so if the runtime JVM
is the same one, no need to spend time checking for fips support.

Verification of the JAVA<version>_HOME env vars is now async and
parallel so it doesn't block configuration.
2019-04-19 08:58:43 +03:00
Mark Vieira 0227ac5690
Fix issue with subproject test task dependencies (#41321) (#41351) 2019-04-18 11:15:34 -07:00
Alpar Torok 44a0c468cf Clean up clusters between tests (#41187)
This PR adds additional cleanup when stopping the node.
The data dir is excepted because it gets reused in some tests.
Without this cleanup the number of working dir copies could grew to
exhaust all available disk space.
2019-04-18 08:58:47 +03:00
Tim Brooks 56c00eecbc
Remove string usages of old transport settings (#41207)
This is related to #36652. We intend to deprecate a number of transport
settings in 7.x and remove them in 8.0. This commit removes the string
usages of these settings.
2019-04-15 16:54:24 -06:00
Alpar Torok 636b9338a6 Fix the rpm and deb names for version starting with 7.0.0 (#41198)
With the 7.0.0 release, we switched to download the packages instead of
using locally built ones.
This PR fixes the artifact names to include the architecture as
introduced in the 7.0.0 release.
2019-04-15 18:52:19 +03:00
Alpar Torok ff64314b2b Simplify testclusters, don't allow cross project clusters (#40972)
* Simplify testclusters, don't allow cross project clusters
2019-04-12 12:41:52 +03:00
Mark Vieira 1287c7d91f
[Backport] Replace usages RandomizedTestingTask with built-in Gradle Test (#40978) (#40993)
* Replace usages RandomizedTestingTask with built-in Gradle Test (#40978)

This commit replaces the existing RandomizedTestingTask and supporting code with Gradle's built-in JUnit support via the Test task type. Additionally, the previous workaround to disable all tasks named "test" and create new unit testing tasks named "unitTest" has been removed such that the "test" task now runs unit tests as per the normal Gradle Java plugin conventions.

(cherry picked from commit 323f312bbc829a63056a79ebe45adced5099f6e6)

* Fix forking JVM runner

* Don't bump shadow plugin version
2019-04-09 11:52:50 -07:00
Jason Tedor 3e078b2026
Use bundled Java for all eligible versions in tests (#40928)
This commit sets the version to ensure that we use the bundled Java when
running integration tests for all eligible versions. In particular,
since we started bundling Java with 7.0.0, this commits sets said
version to 7.0.0.
2019-04-08 08:15:59 -04:00
Jay Modi f34663282c
Update apache httpclient to version 4.5.8 (#40875)
This change updates our version of httpclient to version 4.5.8, which
contains the fix for HTTPCLIENT-1968, which is a bug where the client
started re-writing paths that contained encoded reserved characters
with their unreserved form.
2019-04-05 13:48:10 -06:00
Martijn van Groningen 809a5f13a4
Make -try xlint warning disabled by default. (#40833)
Many gradle projects specifically use the -try exclude flag, because
there are many cases where auto-closeable resource ignore is never
referenced in body of corresponding try statement. Suppressing this
warning specifically in each case that it happens using
`@SuppressWarnings("try")` would be very verbose.

This change removes `-try` from any gradle project and adds it to the
build plugin. Also this change removes exclude flags from gradle projects
that is already specified in build plugin (for example -deprecation).

Relates to #40366
2019-04-05 08:02:26 +02:00
Tim Vernum 9d34164ce6
Add build utility to check cluster health over ssl (#40713)
By default, in integ tests we wait for the standalone cluster to start
by using the ant Get task to retrieve the cluster health endpoint.
However the ant task has no facilities for customising the trusted
CAs for a https resource, so if the integ test cluster has TLS enabled
on the http interface (using a custom CA) we need a separate utility
for that purpose.

Backport of: #40573
2019-04-04 21:44:03 +11:00
Alpar Torok 25944c4317 convert modules to use testclusters (#40804)
* convert modules to use testclusters
* Eliminate PluginPropertiesTask and move logic in plugin where it belongs
2019-04-04 11:45:40 +03:00
Ryan Ernst 9d785e2b69 Don't replace forbidden pattern failures when found (#40710)
This commit fixes a bug in forbidden patterns where the failures for a
file replace the failures from the previous files instead of extending
them.
2019-04-03 16:55:25 -07:00
Alpar Torok 95395938c8 Testclusters support for multi node clusters (#40699)
* Testclusters: introduce support for a multi node cluster
2019-04-03 19:35:36 +03:00
Jason Tedor 60000400db
Drop support for Ubuntu 14.04 (#40709)
With the 7.0.0 release, we are dropping support for Ubuntu 14.04. This
commit removes this OS from our testing infrastructure.
2019-04-02 08:51:40 -04:00
Ricky(道离) 74bf4b8292 If not set JAVA_HOME then use the JDK that Gradle was run with instead (for master) (#40622)
Extending the IDE behavior to the cli
2019-04-01 09:03:44 +03:00
Alpar Torok e8c0b53796 Add ability to mute and mute flaky fixture (#40630) 2019-03-29 12:10:04 +02:00
Alpar Torok 49f18811c3 Run the build integ test in parallel (#39788)
* Run the build integ test in parallel

Because the randomized runner lives in buildSrc, we run these tests with
the Gradle runner, and had no parallelism configured so far.

* Handle Windows and "auto" better
2019-03-29 08:11:20 +02:00
Alpar Torok d791e08932 Test fixtures krb5 (#40297)
Replaces the vagrant based kerberos fixtures with docker based test fixtures plugin.
The configuration is now entirely static on the docker side and no longer driven by Gradle,
also two different services are being configured since there are two different consumers of the fixture that can run in parallel and require different configurations.
2019-03-28 17:26:58 +02:00
Alpar Torok 524e0273ae Testclusters: convert plugin repository-s3 (#40399)
* Add support for setting and keystore settings
* system properties and env var config
* use testclusters for repository-s3
* Some cleanup of the build.gradle file for plugin-s3
* add runner {} to rest integ test task
2019-03-27 08:40:16 +02:00
Ryan Ernst f82f9423b7 Remove platform test command (#40280)
The platformTest gradle task was a packaging test meant to ensure unit
tests run on the various supported operating systems without relying on
CI to maintain a full matrix of platforms. Howevever, it never really
worked out as intended and is now additional code in our vagrant setup
to maintain. This commit removes the platformTest task.
2019-03-26 19:47:33 -07:00
Ryan Ernst e49ba6a9d9 Remove unused variable (#40468)
This was leftover from previous iterations of test clusters refactorings
2019-03-26 11:26:31 -07:00
Alpar Torok b5a8de9a7f Revert tmpdir (#40312)
* Revert "Configure TMP for test nodes on Windows (#39959)"
This reverts commit 97562a874fcb1f29fb05272ab860a0307e97d1aa.

* Configure a tmp dir without spaces
* Pass on TMP instead of changing it
2019-03-26 13:31:18 +02:00
Alpar Torok d50305bbee Upgrade to Gradle 5.3! (#40346)
Here are the highlights of this release:
 - Feature variants AKA "optional dependencies"
 - Type-safe accessors in Kotlin precompiled script plugins
 - Gradle Module Metadata 1.0

For more details see https://docs.gradle.org/5.3/release-notes.html
2019-03-26 13:23:40 +02:00
Alpar Torok cd4eac5fa4 Testclusters links (#39831)
* Create hardlinks instead of sync to save on disk space
2019-03-26 13:10:13 +02:00
Jason Tedor 3212557da4
Require JDK 12 for compilation (#40231)
This commit switches from requiring JDK 11 for compilation to requiring
JDK 12 for compilation.
2019-03-22 08:33:23 -04:00
Jason Tedor c2a566a64e
Upgrade bundled JDK and Docker images to JDK 12 (#40229)
This commit bumps the JDK used the bundled JDK and the Docker images to
JDK 12.
2019-03-19 23:45:07 -04:00
Alpar Torok cb29d3d8d0 Fix build-tools unit test on windows (#39986) 2019-03-19 08:07:33 +02:00
Ryan Ernst 8f09c77777 Add no-jdk distributions (#39882)
This commit adds a variant for every official distribution that omits
the bundled jdk. The "no-jdk" naming is conveyed through the package
classifier, alongside the platform. Package tests are also added for
each new distribution.
2019-03-15 00:55:57 -07:00
Jason Tedor d02bca1314
Upgrade the bouncycastle dependency to 1.61 (#40017)
This commit upgrades the bouncycastle dependency from 1.59 to 1.61.
2019-03-14 08:54:47 -04:00
Alpar Torok 9b8b47114e Configure TMP for test nodes on Windows (#39959)
This breaks on windows where TMP dir default to C:\Windows and startup
fails with a permission error.
I tried to create a tmp dir and pass in `TMP` env, but it lead to a
class not found error, and since testclusers is already independent of
the calling environment I stopped there.
2019-03-13 19:21:13 +02:00
Jim Ferenczi 7a7658707a
Upgrade to Lucene release 8.0.0 (#39998)
This commit upgrades to the GA release of Lucene 8

Closes #39640
2019-03-13 18:11:50 +01:00
Daniel Mitterdorfer 34006105ae
Allow IntegTestClusters to run with FIPS 140 JVMs (#39917) (#39923)
The changes in #39732 mean that nodes in the IntegTest clusters will
now run with whichever java version is defined as `runtime.java` and
not JAVA_HOME anymore.
This means that these nodes will also run in JVM with fips approved
mode enabled and as such, need to have access to the password for the
BCFKS keystore that is used as the default keystore/truststore.

This change sets the two necessary system properties.

Resolves #39855
2019-03-12 06:46:51 +01:00
Ryan Ernst 35dd52461f Make integ tests always use runtime java (#39732)
This commit ensures cluster formation always uses runtime java, never
attempting to use the bundled jdk, since the integ test zip does not
contain it.
2019-03-08 11:08:26 -08:00
Ryan Ernst 465343f12a
Bundle java in distributions (#38013)
* Bundle java in distributions

Setting up a jdk is currently a required external step when installing
elasticsearch. This is particularly problematic for the rpm/deb packages
as installing a jdk in the same package installation command does not
guarantee any order, so must be done in separate steps. Additionally,
JAVA_HOME must be set and often causes problems in selecting a correct
jdk when, for example, the system java is an older unsupported version.

This commit bundles platform specific openjdks into each distribution.
In addition to eliminating the issues above, it also presents future
possible improvements like using jlink to build jdk images only
containing modules that elasticsearch uses.

closes #31845
2019-03-08 11:04:18 -08:00
Jason Tedor 0250d554b6
Introduce forget follower API (#39718)
This commit introduces the forget follower API. This API is needed in cases that
unfollowing a following index fails to remove the shard history retention leases
on the leader index. This can happen explicitly through user action, or
implicitly through an index managed by ILM. When this occurs, history will be
retained longer than necessary. While the retention lease will eventually
expire, it can be expensive to allow history to persist for that long, and also
prevent ILM from performing actions like shrink on the leader index. As such, we
introduce an API to allow for manual removal of the shard history retention
leases in this case.
2019-03-07 11:08:45 -05:00
Alpar Torok 6c75a2f2b0 Testclusters: start using it for testing some plugins (#39693)
* enable testclusters for some plugins
2019-03-07 17:52:50 +02:00
Alpar Torok 7dcc191aa8 Fix verify versions (#39624)
closes #38708
2019-03-07 17:40:25 +02:00
Alpar Torok 0f89427eb6
Back port build changes from same version bwc tests (#39744)
* Back port build changes from #39102

This back-ports how versions are determined and bwc test are set up from
 #39102 without enabling the bwc from current version tests so it's
 easier/possible to backmerge future buld changes.
It's expected that the tets are lacking many of the required fixes in
this version to enable them.
2019-03-07 17:25:09 +02:00
Akshesh Doshi 4df26a2a90 Implement ConcatFilesTask from Groovy to Java issues#34459 (#37497)
relates  #344597
2019-03-07 15:30:54 +02:00
Alpar Torok 34ea84948c
Fix bwc tests failure to extract (#39619)
* Set correct packaging for older versions

Continue using zip packages for pre 7
Some other bwc fixes that hid behind this one.
Closes #39441  #39751
2019-03-07 09:07:11 +02:00
Alpar Torok 0f85182abe Testclusters: implement support to install plugins (#39116)
* methods to run bin script
* Add support for specifying and installing plugins
* Add OS specific distirbution support
* Add test to verify plugin installed
* Remove use of Gradle internal OperatingSystem
2019-03-04 09:15:23 +02:00
Alpar Torok 813351fe26 Un-mute and fix BuildExamplePluginsIT (#38899)
* Un-mute and fix BuildExamplePluginsIT

There doesn't seem to be anything wrong with the test iteself.
I think the failure were CI performance related, but while it was muted,
some failures managed to sneak in.

Closes #38784

* PR review
2019-03-04 08:50:55 +02:00
James Baiera 2a20e8e3fe
Remove missing variable from error message (#39321)
When test clusters are stood up, one of the steps in the wait task is to wait for 
ports files to appear. An exception throw was added if this were to time out
instead of failing with no information, but the exception text uses a missing 
variable which further obfuscates the problem.

Backports #39321
2019-03-01 13:27:57 -05:00
Jason Tedor 6d72d45e33
Use https to obtain Lucene snapshots (#39458)
This commit changes the protocol used to download Lucene snapshots.
2019-02-27 11:41:06 -05:00
Alan Woodward 71b8494181
Upgrade to lucene 8.0.0-snapshot-ff9509a8df (#39444)
Backport of #39350

Contains the following:

* LUCENE-8635: Move terms dictionary off-heap for non-primary-key fields in `MMapDirectory`
* LUCENE-8292: `TermsEnum` is fully abstract
* LUCENE-8679: Return WITHIN in `EdgeTree#relateTriangle` only when polygon and triangle share one edge
* LUCENE-8676: Nori tokenizer deals correctly with large buffers
* LUCENE-8697: `GraphTokenStreamFiniteStrings` better handles side paths with gaps
* LUCENE-8664: Add `equals` and `hashCode` to `TotalHits`
* LUCENE-8660: `TopDocsCollector` returns accurate hit counts if the total equals the threshold
* LUCENE-8654: `Polygon2D#relateTriangle` fix for when the polygon is inside the triangle
* LUCENE-8645: `Intervals#fixField` can merge intervals from different fields
* LUCENE-8585: Create jump-tables for DocValues at index time
2019-02-27 14:36:08 +00:00
Alpar Torok d955375e9c Don't do BWC packaing tests if disabled (#39397)
Packaging tests did not honor bwc tests being off.
This was also the reason for which we were building  the BWC versinons
even if the tests are off, so this closes #35347.
2019-02-27 09:25:29 +02:00
James Baiera e6a124c118
[Backport 7.x] Fix the OS sensing code in ClusterFormationTasks (#38457)
This fixes a bug in the sensing of the current OS family in the test cluster
formation code. Previously all builds would assume every environment 
was windows and would jump to using the windows zip build. This fixes 
the OS sensing code as well as updates some tests to account for 
different build flavors.

Backport of #38457
2019-02-25 14:39:34 -05:00
Mark Vieira 0f68814e04
Ensure global test seed is used for all random testing tasks (#38991) (#39195)
This commit fixes a bug which resulted in every RandomizedTestingTask
generating its own unique test seed rather than using the global test
seed generated by the the RandomizedTestingPlugin.

The issue didn't affect build invocations which explicitly ran with
-Dtest.seed. In those cases, the Ant junit task correctly uses the
global seed.

Since the Ant random testing target looks for an Ant project property
for determining the existing seed we now explicitly set this inside
RandomizedTestingPlugin. It just so happens that, incidentally, Ant will
inherit system properties, thus wy the -Dtest.seed mechanism continued
to work.
2019-02-21 08:52:14 -08:00
Marios Trivyzas 1316825f52
Replace superfluous usage of Counter with Supplier (#39048) (#39225)
`Counter` was used as a means of a functional argument to pass
the relative cached time before `Supplier` iface was introduced.
2019-02-21 12:42:54 +02:00
Jason Tedor 751c05eff9
Bump jackson-databind version for ingest-geoip (#39182)
This commit bumps the jackson-databind version for ingest-geoip to
2.8.11.3.
2019-02-20 11:40:31 -05:00
James Baiera fe9b89ef97
Add an exception throw if waiting on transport port file fails (#37574)
In the ClusterConfiguration class of the build source, there is an Ant waitfor block 
that runs to ensure that the seed node's transport ports file is created before 
trying to read it. If the wait times out, the file read fails with not much helpful info. 
This just adds a timeout property to the waitfor block and throws a descriptive 
exception instead.

Backports #37574
2019-02-20 11:09:43 -05:00
Alpar Torok 1ecdef707d Make specifying java home mandatory for testclusters (#37864)
Since we build and test with multiple versions it's better to be
explicit about it.
2019-02-19 11:19:58 +02:00
Alpar Torok af7b89b80c Force kill testcluster nodes (#37353)
* Force kill testcluster nodes
2019-02-18 14:43:23 +02:00
Mark Vieira 63bfaac16d
Improve testcluster distribution artifact handling (#38933) (#38981)
This commit moves validation logic for ensuring our testclusters
configuration doesn't contain unexpected artifacts into the plugin
itself. This change allows us to remove the custom copy task
implementation altogether.

Additionally, the error message has been improved to display component
ids in addition to the artifacts to make it easier to figure out what
actual dependency is at fault.
2019-02-15 13:31:05 -08:00
Alpar Torok eb7da4b90d Upgrade to Gradle 5.2.1 (#38880) 2019-02-15 15:15:43 +02:00
Marios Trivyzas 6181d2f4f4
Build: Fix issue with test status logging (#38799) (#38943)
Handle the case of `Description` being null which is a valid case as
described in the `HeartBeatEvent`'s javadoc, which previously resulted
in exceptions that "pollute" the build output.

Follows: #28563
Backport: #38799
2019-02-15 15:06:49 +02:00
Alpar Torok 745d0c1bba Add packaging tests for Fedora 29, remove 27 (#38726) 2019-02-12 14:31:17 +02:00
Jason Tedor 5ab5a0a529
Avoid polluting download stats on builds (#38660)
Recently we changed where we source released artifacts for usage in
backwards compatibility tests. We now source these from
artifacts.elastic.co. To avoid polluting the download stats from builds,
we want to add the X-Elastic-No-KPI header to requests from
artifacts.elastic.co. To do this, we hack the Ivy feature of custom HTTP
header credentials and specify our desired headers.
2019-02-09 09:23:33 -05:00
Tim Vernum 84483b26cf
Fix version logic when bumping major version (#38593)
When we are preparing to release a major version the rules around
"unreleased" versions and branches get a bit more complex.

This change implements the following rules:

- If the tip version on the previous major is a .0 (e.g. 6.7.0) then
  the tip of the minor before that (e.g. 6.6.1) must be unreleased.
  (This is because 6.7.0 would be "staged" in preparation for release,
  but 6.6.1 would be open for bug fixes on the release 6.6.x line)
  (in VersionCollection & VersionUtils)

- The "major.x" branch (if it exists) will always point to the latest
  minor in that series. Anything that is not the latest minor, must
  therefore be on a the "major.minor" branch
  For example, if v7.1.0 exists then the "7.x" branch must be 7.1.0,
  and 7.0.0 must be on the "7.0" branch
  (in VersionCollection)
2019-02-08 18:00:03 +11:00
Jason Tedor fdf6b3f23f
Add 7.1 version constant to 7.x branch (#38513)
This commit adds the 7.1 version constant to the 7.x branch.

Co-authored-by: Andy Bristol <andy.bristol@elastic.co>
Co-authored-by: Tim Brooks <tim@uncontended.net>
Co-authored-by: Christoph Büscher <cbuescher@posteo.de>
Co-authored-by: Luca Cavanna <javanna@users.noreply.github.com>
Co-authored-by: markharwood <markharwood@gmail.com>
Co-authored-by: Ioannis Kakavas <ioannis@elastic.co>
Co-authored-by: Nhat Nguyen <nhat.nguyen@elastic.co>
Co-authored-by: David Roberts <dave.roberts@elastic.co>
Co-authored-by: Jason Tedor <jason@tedor.me>
Co-authored-by: Alpar Torok <torokalpar@gmail.com>
Co-authored-by: David Turner <david.turner@elastic.co>
Co-authored-by: Martijn van Groningen <martijn.v.groningen@gmail.com>
Co-authored-by: Tim Vernum <tim@adjective.org>
Co-authored-by: Albert Zaharovits <albert.zaharovits@gmail.com>
2019-02-07 16:32:27 -05:00
David Turner 2d114a02ff
Rename static Zen1 settings (#38333)
Renames the following settings to remove the mention of `zen` in their names:

- `discovery.zen.hosts_provider` -> `discovery.seed_providers`
- `discovery.zen.ping.unicast.concurrent_connects` -> `discovery.seed_resolver.max_concurrent_resolvers`
- `discovery.zen.ping.unicast.hosts.resolve_timeout` -> `discovery.seed_resolver.timeout`
- `discovery.zen.ping.unicast.hosts` -> `discovery.seed_addresses`
2019-02-05 08:46:52 +00:00
Josh Soref 0154052335 spelling: java script -- not JavaScript (#37057) 2019-01-31 14:09:36 +02:00
Jay Modi 54dbf9469c
Update httpclient for JDK 11 TLS engine (#37994)
The apache commons http client implementations recently released
versions that solve TLS compatibility issues with the new TLS engine
that supports TLSv1.3 with JDK 11. This change updates our code to
use these versions since JDK 11 is a supported JDK and we should
allow the use of TLSv1.3.
2019-01-30 14:24:29 -07:00
David Turner 81c443c9de
Deprecate minimum_master_nodes (#37868)
Today we pass `discovery.zen.minimum_master_nodes` to nodes started up in
tests, but for 7.x nodes this setting is not required as it has no effect.
This commit removes this setting so that nodes are started with more realistic
configurations, and deprecates it.
2019-01-30 20:09:15 +00:00
Alpar Torok f3379940c6
Fix the packer cache script (#38023)
The script is used to create a cache on ephemeral CI workers.
Changes:
    - create and use a `pullFixture` task that always exists regardless
    of docker support
    - wire dependencies correctly so any pre fixture setup runs for pull
    as well
    - set up java env vars so bwc versions can build
2019-01-30 14:41:35 +02:00
Ryan Ernst 8e5f9c4b14
Add OS/architecture classifier to distributions (#37881)
This commit adds classifiers to the distributions indicating the
OS (for archives) and platform. The current OSes are for windows, darwin (ie
macos) and linux. This change will allow future OS/architecture specific
changes to the distributions. Note the docs using distribution links
have been updated, but will be reworked in a followup to make OS
specific instructions for the archives.
2019-01-29 11:18:30 -08:00
Ryan Ernst 6325e55dbf
Use quotes in reproduce line task for vagrant failure (#37884)
This commit wraps the packaging test task for reproducing a vagrant test
failure in quotes. The task names sometimes contain "#", which confuses
bash.
2019-01-28 11:56:36 -08:00
Julie Tibshirani 24ff23812e Mute TestClustersPluginIT and BuildExamplePluginsIT
Tracked in #37889.
2019-01-25 15:41:56 -08:00
Ryan Ernst 980c13d85c
Pass distribution type through to docs tests (#37885)
This commit fixes the distribution flavor passed to the docs tests to be
the same as the distribution. These two values are now in sync (either
oss or default) for the docs tests.
2019-01-25 13:49:16 -08:00
Alpar Torok e7f0adb0c5
Add path sensitivity annotations (#37762)
The plugin builder plugin generates warnings for these.
There's no immediate impact as we don't have a shared build cache.
2019-01-25 14:15:09 +02:00
Alpar Torok 97e89ad0bc
Testclusters logging improovements (#37354)
- Cluster logs are only indented as  node name is already in the logs
- silence logging on shutdown
- have fully qualified name as node and cluster name
2019-01-25 14:13:34 +02:00
Alpar Torok f6b6f927ec
Fail randomized testing tasks if no tests ran (#37764)
Reverts #36259 in part to make randomized test fail if no tests are ran.
This is useful when filtering tests as it's easy to make a typo and
think the test ran trough successfully.
2019-01-24 17:33:08 +02:00
Alpar Torok 37768b7eac
Testing conventions now checks for tests in main (#37321)
* Testing conventions now checks for tests in main

This is the last outstanding feature of the old NamingConventionsTask,
so time to remove it.

* PR review
2019-01-24 17:30:50 +02:00
Alpar Torok e7aa7e909a
Switch testclusters to use nio (#37365) 2019-01-24 17:30:24 +02:00
Alpar Torok 4e08cca6bc
Ground work to start up the docker image in the build (#37754)
This change adds a docker compose configuration that's used with
the `elasticsearch.test.fixtures` plugin to start up the image
and check that the TCP ports are up.

We can build on this to add other checks for culster health,
run REST tests, etc.

We can add multiple containers and configurations to the compose
file (e.x. test different env vars) and form clusters.
2019-01-24 17:26:42 +02:00
Alpar Torok bc20142791
Consolidate testclusters tests into a single project (#37362) 2019-01-24 13:47:21 +02:00
Ryan Ernst d9d13f3414
Use project dependency instead of substitutions for distributions (#37730)
Currently integration tests which use either bwc snapshot versions or
the current version of elasticsearch depend on project substitutions to
link to the build of those artifacts. Likewise, vagrant tests use
dependency substitutions to get to bwc snapshots of rpm and debs.
This commit changes those to depend on the relevant project/configuration
and removes the dependency substitutions for distributions we do not
publish.
2019-01-23 23:41:21 -08:00
Ryan Ernst 49b8b07758
Use explicit version for build-tools in example plugin integ tests (#37792)
The example plugins are currently built within the build-tools integ
tests as a means to ensure the gradle plugin works for external plugin
builds. These tests generate a dummy build.gradle, and a dummy local
maven repository to find the local builds dependencies in. Currently
that build-tools dependency uses "+" as the version. However, this
allows gradle to find the "latest" version, and unfortunately gradle has
its own plugin repository which is apparently connected to jcenter. This
recently triggered a flood of CI failures when jcenter suddenly pulled
alpha2, and all builds started trying to use that instead of the locally
built build-tools. This commit uses the explicit version of build-tools
that was build locally, which will cause resolution to stop when the
local repo is first checked.
2019-01-23 17:39:13 -08:00
Alpar Torok f54a3b5f67
Don't use Groovy's `withDefault` (#37726)
Closes #37061
2019-01-23 12:09:40 +02:00
olcbean c28479819e Fix a typo in a warning message in TestFixturesPlugin (#37631) 2019-01-22 22:36:42 +02:00
Adrien Grand e9fcb25a28
Upgrade to lucene-8.0.0-snapshot-83f9835. (#37668)
This snapshot uses a new file format for doc-values which is expected to make
advance/advanceExact perform faster on sparse fields:
https://issues.apache.org/jira/browse/LUCENE-8585
2019-01-22 11:44:29 +01:00
Ryan Ernst 9a34b20233
Simplify integ test distribution types (#37618)
The integ tests currently use the raw zip project name as the
distribution type. This commit simplifies this specification to be
"default" or "oss". Whether zip or tar is used should be an internal
implementation detail of the integ test setup, which can (in the future)
be platform specific.
2019-01-21 12:37:17 -08:00
Alpar Torok 3a2b616579
Use zen2 with testclusters (#37352) 2019-01-21 12:52:25 +02:00
Ryan Ernst c9c088ff02
Use ivy pattern for elasticsearch artifacts download (#37557)
The rpm, deb and tar distributions were removed some time ago from maven
central. The zip distribution still exists there, but it does not need
to. Instead, this commit sets up an ivy repository with pattern pointing
to the elasticsearch artifacts download service. Note that the
integ-test-zip remains in maven central, since it is not present in the
download service.
2019-01-20 23:33:11 -08:00
Ryan Ernst 614950ef1c
Handle requiring versioned java home during execution (#37599)
Currently bwc builds require different java home environment variables
depending on the version of elasticsearch being built. The java home
version checks are run at the end of gradle configuration, when the task
graph is ready. However, we do not know which versions are needed for
bwc builds until execution time, when we have finished checking out the
version of elasticsearch to be built. This commit accounts for late java
home checks, checking immediately instead of delaying the check.

closes #37586
2019-01-20 22:47:43 -08:00
Yannick Welsch 6d64a2a901
Propagate Errors in executors to uncaught exception handler (#36137)
This is a continuation of #28667 and has as goal to convert all executors to propagate errors to the
uncaught exception handler. Notable missing ones were the direct executor and the scheduler. This
commit also makes it the property of the executor, not the runnable, to ensure this property. A big
part of this commit also consists of vastly improving the test coverage in this area.
2019-01-17 17:46:35 +01:00
Dimitrios Liappis 347cbaf0ed
Fix line length for aliases and remove suppression (#37455)
Relates #34884
2019-01-16 13:06:29 +02:00
Alpar Torok e7511753a9
Upgrade to Gradle 5.1.1 (#37410)
Fixes mem leak in the daemon.
2019-01-15 21:20:19 +02:00
Dimitrios Liappis 63793499bd
Fix line length for `node` and remove suppresion (#37454)
Relates #34884
2019-01-15 19:57:24 +02:00
Dimitrios Liappis 19fc59f089
Fix line length for monitor and remove suppressions (#37456)
Relates #34884
2019-01-15 14:18:15 +02:00
Alpar Torok 100ef2cc0e
Fix artifactId in plugin poms (#37315) 2019-01-11 11:23:54 +02:00
Przemyslaw Gomulka c812e6aea6
Fix line length in org.elasticsearch.routing (#37253)
Remove the line length suppression for this package and fix offending
lines

relates: #34884
2019-01-10 15:23:34 +01:00
Ryan Ernst 29c895b55c
Build: Quiet naming convention logging (#37244)
This commit moves log statements related to classification of naming
convention checks for tests to debug level. At info level they emit an
enormous amount of output in CI, while these are not generally useful
for debugging normal build failures.
2019-01-09 10:56:30 -08:00
Jack Conradson 95eef77ad4
[Style] Fix line length violations for threadpool, indexing, and script packages (#37205) 2019-01-09 10:55:52 -08:00
Alpar Torok d68d6aa00c
Compatability mode for forbidden APIs for Java 12 (#37199) 2019-01-09 18:26:08 +02:00
Ryan Ernst 86b71930f6
Remove more checkstyle suppressions (#36735)
This commit removes several more line length violations from checkstyle
suppression.

relates #34884
2019-01-08 11:04:54 -08:00
Alpar Torok 20c2c439e7
Upgrade to Gradle 5.1 (#36767) 2019-01-08 13:43:10 +02:00
Alpar Torok 6344e9a3ce
Testing conventions: add support for checking base classes (#36650) 2019-01-08 13:39:03 +02:00
Alpar Torok 0c68eead51 Mute failing test clusters test
Tracking issue: #37218
2019-01-08 13:00:55 +02:00
Alpar Torok a7c3d5842a
Split third party audit exclusions by type (#36763) 2019-01-07 17:24:19 +02:00
Armin Braun 31c33fdb9b
MINOR: Remove some Deadcode in Gradle (#37160) 2019-01-07 09:21:25 +01:00
Nick Knize e613bcae43
Remove XLatLonShape classes (#37094)
This commit removes local XLatLonShape classes and replaces with current LatLonShape classes in latest lucene snapshot
2019-01-03 12:48:36 -06:00
Nick Knize b2aa655f46
Upgrade master to lucene-8.0.0-snapshot-a1c6e642aa (#37091)
Updates the master branch to the latest snapshot of Lucene 8.0.
2019-01-02 20:18:19 -06:00
Josh Soref 5d5d5c26bc Spelling: replace interruptable with interruptible (#37049) 2019-01-02 17:35:53 +01:00
Przemyslaw Gomulka 15c4d5b184
Fix line length in org.elasticsearch.get (#37071)
Remove the line suppression for this package and fix offedning lines
relates #34844
2019-01-02 17:22:15 +01:00
Alpar Torok 9f75b6ea58 Fix build on sles with new docker version (#36707)
The build hash for some Docker versions is
longer than 7 characters.

Closes #36414
2018-12-20 10:15:26 +00:00
Przemyslaw Gomulka 1345dff507
Fix line length in org.elasticsearch.snapshots (#36646)
Remove the line length suppression for this package and fix offending
lines in both main and test

relates #34884
2018-12-19 11:29:21 +01:00
Alpar Torok e9ef5bdce8
Converting randomized testing to create a separate unitTest task instead of replacing the builtin test task (#36311)
- Create a separate unitTest task instead of Gradle's built in 
- convert all configuration to use the new task 
- the  built in task is now disabled
2018-12-19 08:25:20 +02:00
Alpar Torok 8015560bd4
Remove line length supressions from plugins (#36708) 2018-12-19 08:06:48 +02:00
Tim Brooks aaf466ff5e
Revert transport.port change for tests (#36809)
Commit #36786 updated docs and strings to reference transport.port instead of
transport.tcp.port. However, this breaks backwards compatibility tests
as the tests rely on string configurations and transport.port does not
exist prior to 6.6. This commit reverts the places were we reference
transport.tcp.port for tests. This work will need to be reintroduced in
a backwards compatible way.
2018-12-18 19:01:13 -07:00
Nick Knize 20b58f0b0f
[GEO] Fork Lucene's LatLonShape Classes to local lucene package (#36794)
Lucene 7.6 uses a smaller encoding for LatLonShape. This commit forks the LatLonShape classes to Elasticsearch's local lucene package. These classes will be removed on the release of Lucene 7.6.
2018-12-18 16:48:30 -06:00
Tim Brooks 47a9a8de49
Update transport docs and settings for changes (#36786)
This is related to #36652. In 7.0 we plan to deprecate a number of
settings that make reference to the concept of a tcp transport. We
mostly just have a single transport type now (based on tcp). Settings
should only reference tcp if they are referring to socket options. This
commit updates the settings in the docs. And removes string usages of
the old settings. Additionally it adds a missing remote compress setting
to the docs.
2018-12-18 13:09:58 -07:00
Evgenia Badyanova 27ad733dce
Fixing line length for EnvironmentTests and RecoveryTests (#36657)
Relates #34884
2018-12-17 11:59:45 -05:00
Alan Woodward c7ac9ef826
Upgrade to lucene snapshot 774e9aefbc (#36637)
Includes LUCENE-8607: improvement to MatchAllDocsQuery
2018-12-14 20:30:07 +00:00
Alpar Torok b5725ed482
Move inputs and outputs before action (#36539) 2018-12-14 18:31:52 +02:00
Daniel Mitterdorfer b7ba2fa74b
Fix line length for VersionTests
Relates #34884
Relates #36585
2018-12-14 10:07:27 +01:00
Daniel Mitterdorfer 42f094952b
Fix line length for IndexLifecycleActionIT
Relates #34884
Relates #36586
2018-12-14 10:00:37 +01:00
Alan Woodward 9ac7359643
Update lucene to snapshot-7e4555a2fd (#36563)
Includes the following:

* Reversion of doc-values changes in LUCENE-8374; we are interested in seeing if this 
  has an effect on benchmarks for node-stats and index-stats
* More improvements to docvalues updates
2018-12-12 20:18:32 +00:00
Alpar Torok f920f571d1
Fix possible missing class error (#36491)
- Use the `testRuntime` classpath when loading the tests instead of
compile
- simplify task action
- improove error message when classes are missing
2018-12-12 12:24:17 +02:00
Alpar Torok 09a9e2236c
Third party audit improovements (#36167)
- fix up to date checks to ignore elasticsearch jars. We were not scanning them but these still triggered a rebuild.
- add tests to assert correct behavior and up to date checks.
- make the task less verbose with `-i` and include the output only on errors.
2018-12-12 12:04:36 +02:00
Alpar Torok c00d0fc814
Test fixtures improovements (#36037)
* Upgrae plugin to latest and expose udp
* Explicit check for windows
* Rename the properties for the port numbers
* Tasks for pre and pos container actions
2018-12-12 12:00:47 +02:00
Nhat Nguyen 3fb5a12b30 Upgrade to Lucene-8.0.0-snapshot-61e448666d (#36518)
Includes:
- LUCENE-8602: Share TermsEnum if possible while applying DV updates
2018-12-12 06:47:40 +01:00
Jason Tedor f2e18a6bf6
Add check for minimum required Docker version (#36497)
Our Docker build uses a multi-stage Docker build. This requires Docker
version 17.05 or greater. Without an explicit check here, the build
fails in a mysterious way such as "invalid reference format" that is
hard to track down (Google searches for "Docker invalid reference
format" do not turn up anything useful). This commit refactors our
existing Docker checks, and adds a new one for the minimum Docker
version.
2018-12-11 14:33:53 -05:00
James Baiera a07fb312fc
Move LoggedExec to minimumRuntime source set (#36453)
This moves LoggedExec to the minimumRuntime source set so that it correctly 
compiles for earlier Java versions, namely Java 8.
2018-12-11 13:49:24 -05:00
Christian Schneider 00eadd93eb converting ForbiddenPatternsTask to .java (#36194)
* converting ForbiddenPatternsTask to java impl & unit tests
2018-12-11 14:15:44 +02:00
Jason Tedor b030125e23
Build Docker images if the binary exists (#36459)
This commit changes the Docker assemble tasks so that they attempt to
build Docker if the Docker binaries exist, of if build.docker is set to
true. If the Docker binaries do not exist, or if build.docker is set to
false, then no attempt is made to build the Docker images.
2018-12-10 18:36:55 -05:00
Nhat Nguyen 2a7edca59f
Upgrade to Lucene-8.0.0-snapshot-ef61b547b1 (#36450)
Includes:

- LUCENE-8598: Improve field updates packed values
- LUCENE-8599: Use sparse bitset to store docs in SingleValueDocValuesFieldUpdates
2018-12-10 16:33:49 -05:00
Yannick Welsch 6e6e63d01d
Zen2: Move all mixed-version REST tests to Zen2 (#36398)
Moves all remaining (rolling-upgrade and mixed-version) REST tests to use Zen2. To avoid adding
extra configuration, it relies on Zen2 being set as the default discovery type. This required a few
smaller changes in other tests. I've removed AzureMinimumMasterNodesTests which tests Zen1
functionality and dates from a time where host providers were not configurable and each cloud
plugin had its own discovery.type, subclassing the ZenDiscovery class. I've also adapted a few tests
which were unnecessarily adding addTestZenDiscovery = false for the same legacy reasons. Finally,
this also moves the unconfigured-node-name REST test to Zen2, testing the auto-bootstrapping
functionality in development mode when no discovery configuration is provided.
2018-12-10 11:00:57 +01:00
Alpar Torok bf2c61d2d0
Forbid empty testing tasks (#36259)
Closes #34820

With this change we allow for no tests being ran in randomized testing
task, and forbid empty testing tasks from the testing conventions task.
We will no longer have to disable the task if all tests are muted.
2018-12-10 08:32:19 +02:00
Alpar Torok e160346b9a
Improve logged exec output readability (#36217)
* Improve logged exec output readability

- Split error and out streams and log them separately
- Log everything in a single call to prevent interference from
  other log messages
2018-12-10 08:23:03 +02:00
David Roberts 9e8cfbb40d
[ML] Deprecate X-Pack centric ML endpoints (#36315)
This commit is part of our plan to deprecate and
ultimately remove the use of _xpack in the REST APIs.

Relates #35958
2018-12-07 20:34:11 +00:00
Alpar Torok 03c577b477
Move VersionProperties and Version to min runtime (#36378)
Allow elasticsearch-hadoop to access these classes when running on
java 8.
2018-12-07 20:07:36 +02:00
Armin Braun e6d190613f
[ZEN2] Use Zen2 in REST Tests (#36300) 2018-12-07 09:15:11 +01:00
Nhat Nguyen 10feb75eb7
Upgrade to Lucene-8.0.0-snapshot-aaa64d70159 (#36335)
Includes:

LUCENE-8594: DV update are broken for updates on new field
LUCENE-8590: Optimize DocValues update datastructures
LUCENE-8593: Specialize single value numeric DV updates

Relates #36286
2018-12-06 20:33:25 -05:00
Ryan Ernst 37b3fc383f
Build: Use explicit deps on test tasks for check (#36325)
This commit moves back to use explicit dependsOn for test tasks on
check. Not all tasks extending RandomizedTestingTask should be run by
check directly.
2018-12-06 14:13:49 -08:00
Jason Tedor ed45acf221
Fix name of Docker binaries
This commit fixes an oops when pushing a change to add the building of
the Docker images. A change that was made for testing was accidentally
left behind. This commit addresses that.
2018-12-06 17:10:48 -05:00
Jason Tedor 11dd412ec6
Introduce Docker images build (#36246)
This commit introduces the building of the Docker images as bonafide
packaging formats alongside our existing archive and packaging
distributions. This build is migrated from a dedicated repository, and
converted to Gradle in the process.
2018-12-06 17:06:13 -05:00
Adrien Grand b08cffad48
Fix line length offenders in the o.e.search package. (#36223)
Relates #34884
2018-12-06 16:31:12 +01:00
Alpar Torok 9ae5ee2497 Fix typo in comment 2018-12-06 09:14:50 +02:00
Christoph Büscher 75635251ab
Make sure test don't use Math.random for reproducability (#36241)
Currently we use Math.random() in a few places in the tests which makes these
tests not reproducable with the random seed mechanism that comes with
ESTestCase. The change removes those instances.
2018-12-06 00:07:15 +01:00
Alpar Torok d531fab4b3 Run Build tests with proper java version 2018-12-05 14:47:15 +02:00
Alan Woodward 73ceaad03a
Update to lucene-8.0.0-snapshot-c78429a554 (#36212)
Includes:

* A fix for a bug in Intervals.or() (https://issues.apache.org/jira/browse/LUCENE-8586)
* The ability to disable offset mangling in WordDelimiterGraphFilter
        (https://issues.apache.org/jira/browse/LUCENE-8509)
* BM25Similarity no longer multiplies scores by k1 + 1
2018-12-05 12:43:56 +00:00
Alpar Torok 60e45cd81d
Testing conventions task part 2 (#36107)
Closes #35435

- make it easier to add additional testing tasks with the proper configuration and add some where they were missing.
- mute or fix failing tests
- add a check as part of testing conventions to find classes not included in any testing task.
2018-12-05 14:20:01 +02:00
Alpar Torok 59b0900174
Upgrade to Gradle 5.0 (#34263) 2018-12-05 14:06:11 +02:00
Alpar Torok d036e0ca89
Testclusters: implement starting, waiting for and stopping single cluster nodes (#35599) 2018-12-04 10:16:51 +02:00
Jay Modi 7b999bdc88
Fix logic in dockerComposeSupported (#36125)
The logic in the dockerComposeSupported method currently returns false
even when docker and docker compose are available on the build machine.
This change updates the check to see if docker compose is available in
one of the two paths and allows the `tests.fixture.enabled` property to
disable the tests even if docker compose is available.
2018-11-30 14:38:10 -07:00
Tim Brooks 370472b6d1
Upgrade Netty 4.3.32.Final (#36102)
This commit upgrades netty. This will close #35360. Netty started
throwing an IllegalArgumentException if a CompositeByteBuf is
created with < 2 components. Netty4Utils was updated to reflect this
change.
2018-11-30 09:02:10 -07:00
Alpar Torok d128b38116
Introducing testing conventions task (#35861) 2018-11-30 09:34:53 +02:00
Ryan Ernst 14eeb2aa30
Build: Remove padding from java version in build info (#36088)
This commit removes padding from the java version in the output of
compiler/runtime/gradle java versions. This value is only ever 1 or 2
digits, and the later information (hotspot/jvm vendor info) is separated
by the java home path from the other versions, so there isn't a visual
reason for needing exact vertical alignment.
2018-11-29 18:58:00 -08:00
Alpar Torok 9709282735
Don't disable tasks based on the plugin (#36045)
Some times the test fixtures plugin did not correctly disable tasks
from the build plugin as it should.
The plugin manager and tasks both use domain name collections so
the previus conde should have worked.
I did not have trime to track it down, but suspect there's some race
condition in Gradle causing this. The plugin manager is still incubating.
Since the tasks are on the cp even if the plugin is not applyed, we
don't really  need to involve the plugin at all.

Closes #36041
2018-11-29 15:54:34 +02:00
Nik Everett 4b0b479d1b
Build: Fix reproduce info for methods with ( or ) (#35712)
Our `REPRODUCE WITH` line wasn't working for tests with `(` or `)` in
the method name. This isn't super common, but happens sometimes for our
yml based rest tests. This is because randomized runner's globbing
mechanism dind't escape `(` or `)`. I filed this upstream at
https://github.com/randomizedtesting/randomizedtesting/issues/271
and Dawid kindly fixed the issue. This upgrades to pick up the fix.

Closes #35692
2018-11-29 08:38:09 -05:00
Alpar Torok 45db829039
Replace fixtures with docker-compose (#35651)
Creates a new plugin to manage docker-compose based test fixtures. 
Convert the smb-fixture as a first example.
2018-11-29 09:43:16 +02:00
Hendrik Muhs 8e86340d1d
set line lengths for comments equal to code (#35984)
eclipse (default) wraps code comments at 80 characters, this change sets line length for code comments equal to code == 140.
2018-11-28 13:34:20 +01:00
Marios Trivyzas af9233b067
Build: Fix gradle build for Mac OS (#35968)
Remove trailing line feed from the output of the command to get
the number of CPUs.

Follow up to: #35789
2018-11-27 21:31:43 +01:00
Nik Everett 38725bd3a1
Build: Pick default test jvms for macOS (#35789)
In #35259 we switched the default number of VMs to fork for unit tests to
the number of physical CPU cores. But because we could only get an accurate
count on machines with a normal `/proc` filesystem, macOS machine did not
pick up the new default. Given that macOS is a huge portion of developer
machines, we'd like to get the right default there. This does that.

It also moves the default-finding process from happening once per testing
task to happening once at startup. This seems like a good choice in general,
but a very good choice for macOS because we have to run a command to list
the count.
2018-11-27 09:59:44 -05:00
Armin Braun 04a48430aa
NETWORKING: Upgrade Netty 4.3.31.Final (#35877) 2018-11-25 10:17:54 +01:00
Jim Ferenczi e37a0ef844
Upgrade to lucene-8.0.0-snapshot-67cdd21996 (#35816) 2018-11-22 15:42:59 +01:00
Gordon Brown 4bda469861
Fix line lengths in misc other files (#35650) 2018-11-20 12:29:48 -07:00
Alpar Torok 8659af68e0
Auto skip license headers on no source (#35640)
* Unmute BuildExamplePluginsIT

* Skip licenseHeaders when there are no sources
2018-11-20 13:02:33 +02:00
Jim Ferenczi a5f5ceb869
Remove remaining line length violations for o.e.index (#35652)
This commit removes line length violations in the classes under
org.elasticsearch.index.
2018-11-20 08:09:14 +01:00
Alpar Torok fda59ff969
The default of tests.jvms should not be capped (#35259)
Default to the number of physical cores across all sockets instead.
2018-11-19 14:01:32 +02:00
Martijn van Groningen c468d928b8
Remove line length violations in o.e.indices package (#35647)
Relates to #34884
2018-11-19 07:36:24 +01:00
Lee Hinman 22933ebcd5
Fix line length for org.elasticsearch.action.* files (#35607)
This removes the checkstyle suppressions for things in the `action` package.

Relates to #34884
2018-11-16 14:11:39 -07:00
Alpar Torok 6a8fe49f0c Unmute BuildExamplePluginsIT 2018-11-16 17:16:00 +02:00
Tom Callahan 76b77dbaa4
Remove remaining line length violations for o.e.action.admin.cluster (#35156)
This inserts newlines in order to reduce line lengths in the
o.e.action.admin.cluster package to 140 characters or less.  This
also remves the checkstyle suppressions for affected files.

Relates #34884, #34923
2018-11-16 09:37:42 -05:00
Armin Braun 68d9b56fcb
MINOR: Remove SNAPSHOT Suffixes in Gradle (#35577)
* MINOR: Remove SNAPSHOT Suffixes in Gradle

* The snapshot qualifiers here are redundant and 6.5.0 is released anyway
2018-11-15 14:24:04 +01:00
Armin Braun e6f4425e01
REST-Tests: Use Dedicated AntBuilders (#35576)
* REST-Tests: Use Dedicated AntBuilders

* Use dedicated AntBuilder everywhere since AntBuilder is not threadsafe
* Closes #33778
2018-11-15 14:23:28 +01:00
Alpar Torok a3cb74190c Fix flaky gradle integration test 2018-11-14 17:21:25 +02:00
Alpar Torok 212c202709 Remove hard coded version string in tests
- The current version was hard coded in the test, causing it to fail as
we removed the qualifier
- also applied the base plugin to the root project to get the `clean`
task to work as expected. This was preventing the failure from
reproducing locally.
2018-11-14 14:14:03 +02:00
Alpar Torok 0ce4649e88
Manage dependencies for test clusters (#35304)
* Manage dependencies for test clusters

Create a configuration and add the distribution to it automatically.
A task is created and added as a dependency to any task that uses a test
cluster.
The task extracts all the zip archives ( only zip support for now )
in the configuration.
We do this only once because most tests mostly use the same distribution
and thus we can avoid extracting it multiple times.
With this we will be able to start the node from the same files which
will most of the time live in OS caches or COW if the
configuration requires it.
2018-11-14 11:22:00 +02:00
Alpar Torok dc0e05f35a
Fix Eclipse configuration (#35452) 2018-11-14 11:11:46 +02:00
Alpar Torok b49f461bf1
Swithc to jcenter and hope to have fewer network failures in builds (#35427)
* Swithc to jcenter

Jcenter suposedly operates on a CDN.
It should be faster and more reliable.
It's the default in Andorid Studio currently
( it switched from mavenCentral ).

This change is safe because jcenter is a superset of mavenCentral.

* update comment
2018-11-14 10:50:08 +02:00
Armin Braun 3db671e12a
MINOR: Remove Hack in TestCluster Task (#35485)
*  The logic that splits by `","` isn't necessary anymore since #35375 removed the hack that set the comma separated list for the unicast host
2018-11-13 20:26:42 +02:00
Armin Braun 7919b66e67
DISCOVERY: 0s Initial State Timeout in Tests (#35481)
* DISCOVERY: 0s Initial State Timeout in Tests

* Don't wait for initial state even with a single node, otherwise the loop writing the discovery file causes that single node to wait
for its own transport.ports file for 30s.
* Closes #35456
2018-11-13 14:19:05 +02:00
Lukáš Vlček 76cefb06e3 Correct typo in BuildPlugin exception message (#35458) 2018-11-13 10:57:12 +01:00
Nick Knize 2591f66a33
upgrade to lucene-8.0.0-snapshot-6d9c714052 (#35428) 2018-11-12 10:48:27 -06:00
Alpar Torok ca24d9a07e
Switch to default to no build qualifier (#35415) 2018-11-12 14:43:42 +02:00
Christoph Büscher 09cac321e7
Upgrade to Joda 2.10.1 (#35410)
This version contains a bugfix that allows us to reenable one of our muted tests
in DateTimeUnitTests.

Closes #33749
2018-11-12 10:02:41 +01:00
Armin Braun 529910a43c
DISCOVERY: Fix RollingUpgradeTests (#35375)
* DISCOVERY: Fix RollingUpgradeTests

* Don't manually manage min master nodes if not necessary
* Remove some dead code
* Allow for manually supplying list of seed nodes
* Closes #35178
2018-11-09 10:47:30 +01:00
Alpar Torok 5ae03195d3
Make version field names more meaningful (#35334)
* Consolidate the name of the qualified build version

* Field name in response should not be redundant
2018-11-07 18:36:02 +02:00
Alpar Torok 8a85b2eada
Remove build qualifier from server's Version (#35172)
With this change, `Version` no longer carries information about the qualifier,
we still need a way to show the "display version" that does have both
qualifier and snapshot. This is now stored  by the build and red from `META-INF`.
2018-11-07 14:01:05 +02:00
Alpar Torok 366f79cef2
Check for java 11 in buildSrc (#35260) 2018-11-06 14:43:48 +02:00
Alpar Torok 58f3e81555
Add missing up-to-date configuration (#35255)
* Add missing up-to-date configuration

The source properties file and the dynamic elasticsearch version
(set based on properties ) were missing from task outputs leading
to the task being incorrectly considered up to date.

Closes #35204
2018-11-06 14:00:40 +02:00
Nick Knize a5e1f4d3a2 Upgrade to lucene-8.0.0-snapshot-31d7dfe6b1 (#35224) 2018-11-06 11:55:23 +01:00
Alpar Torok f22700812e
Introduce build qualifier parameter (#35155)
* Introduce property to set version qualifier

- VersionProperties.elasticsearch is now a string which can have qualifier
and snapshot too
- The Version class in the build no longer cares about snapshot and
qualifier.
2018-11-02 05:27:40 +02:00
Alpar Torok 0bdf6a768a Make start/stop metods package protected
As requested in #34818
2018-11-01 19:46:58 +02:00
Alpar Torok ffacd6f0e4
Testclusters: Refactor and fix for newly added tests (#34818)
* Testclusters: Refactor and fix for newly added tests
* Move state management outside of the config object
2018-11-01 19:41:41 +02:00
Alpar Torok f8378d91a6
Implement VersionCollection in Java (#34050) 2018-11-01 17:43:57 +02:00
Vladimir Dolzhenko be75b40a29
Fix LineLength Check Suppressions: index.mapper (#35087)
Relates #34884
2018-10-30 18:00:14 +01:00
Ryan Ernst 5dda2b0c7a
Remove remaining line length violations in o.e.cluster (#34941)
relates #34923, #34884
2018-10-29 19:45:35 -07:00
Alpar Torok c9a8c58039
Fix test failure on Windows (#35047) 2018-10-29 18:13:17 +02:00
Nik Everett 6fd7ea71f1
HLRC: DRY up remaining CRUD docs (#34925)
This further applies the pattern set in #34125 to reduce copy-and-paste
in the multi-document CRUD portion of the High Level REST Client docs.
It also adds line wraps to snippets that are too wide to fit into the box
when rendered in the docs, following up on the work started in #34163.
2018-10-29 10:32:17 -04:00
Mark Tozzi 329a94be0c
Cleanup suppressed overlength line for action.support package (#34889)
Clean up lines over 140 characters in the `org.elasticsearch.action.support.*` packages

Relates to #34884
2018-10-29 09:22:20 -04:00
vboulaye 3ef1fa566b convert FilePermissionsTask.groovy to .java (#34674) 2018-10-29 09:06:34 +02:00
Igor Motov 01c62fc06b
Fix line length for bootstrap/client/discovery/gateway files (#34905)
Removes the checkstyle suppressions for files in
org.elasticsearch.bootstrap/client/discovery/gateway packages.

Relates to #34884
2018-10-26 18:13:09 -04:00
Jake Landis 11fa8d3744
Enforce 140 char line lengths for packages action.bulk/delete/explain/get/index (#34885)
part of #34884
2018-10-26 16:14:04 -05:00
Armin Braun 5acabc47ed
DISCOVERY: Use Realistic Num. of Min Master Nodes (#34854)
* DISCOVERY: Use Realistic Num. of Min Master Nodes

* With all 3 nodes starting in parallel 2 nodes can win a master election and
start waiting for nodes to join when started in parallel. We wait for 30s on the
rest tests for the cluster health endpoint to show 3 nodes which breaks if
one of the master nodes itself waits for 30s for joining nodes, waiting for 5 seconds
fixes the issue and allows us to run with `minimum master nodes < node count`
* relates #33675
2018-10-26 22:35:15 +02:00
Jack Conradson aefe2909c4
[Style] Remove line length violations from ingest actions (#34886) 2018-10-26 09:15:35 -07:00
Lee Hinman af28d1f648
Fix line length for org.elasticsearch.common.* files (#34888)
This removes the checkstyle suppressions for things in the `common` package.

Relates to #34884
2018-10-26 08:47:39 -06:00
Gordon Brown 5c2c1f44c8
[Style] Fix line lengths in action.admin.indices (#34890)
Clean up lines over 140 characters in the the
`org.elasticsearch.action.admin.indices` packages
2018-10-26 08:01:38 -06:00
Armin Braun db12005674
Fix LineLength Check Suppressions: index.fielddata (#34891)
* Fix linelength suppressions in index.fielddata
  * Some lines that were too long were dead code => Removed them and all code that became dead because of it
* Relates #34884
2018-10-26 12:56:19 +02:00
Alpar Torok 13ca942152
Switch build-tools to latest target version (#34746)
- we already require Java 11 to build, yet we target the minimum
supported version in build-tools ( currently 8 )
- this is because we have some checks that are executed in a new JVM
which could be running the minimum version.
- For everything else it would be nice to be able to use new features,
like the new process API.

With this change, we selectively compile the few classes that need an
older target version and move everything over to Java 10.
Unfortunately the current Gradle version does not support 11 as a target
version yet.
2018-10-25 10:03:23 +03:00
Armin Braun 0a85997cb0
TESTS: Use File Based Discovery in REST Tests (#34560)
* For `6.5+` use file based discovery in REST tests
* Relates #33675
2018-10-24 17:18:52 +02:00
Alpar Torok 4370b44ef0
Rename testcluster extension to align with plugin name (#34751) 2018-10-24 18:15:25 +03:00
Alpar Torok 795d57b4f9
Auto configure all test tasks (#34666)
With this change, we apply the common test config automatically to all
newly created tasks instead of opting in specifically.

For plugin authors using the plugin externally this means that the
configuration will be applied to their RandomizedTestingTasks as well.

The purpose of the task is to simplify setup and make it easier to
change projects that use the `test` task but actually run integration
tests to use a task called `integTest` for clarity, but also because
we may want to configure and run them differently.
E.x. using different levels of concurrency.
2018-10-24 16:05:50 +03:00
Zachary Tong 299d044bfc
Collapse pipeline aggs into single package (#34658)
- Restrict visibility of Aggregators and Factories
- Move PipelineAggregatorBuilders up a level so it is consistent with
AggregatorBuilders
- Checkstyle line length fixes for a few classes
- Minor odds/ends (swapping to method references, formatting, etc)
2018-10-23 16:01:01 -04:00
Alpar Torok 5fb4f32c91
[Build] Simplify testclusters configuration (#34334)
Remove interface, prefer straught implementaton instead.
2018-10-23 12:13:49 +03:00
Alpar Torok 0536635c44
Upgrade forbiddenapis to 2.6 (#33809)
* Upgrade forbiddenapis to 2.6

Closes #33759

* Switch forbiddenApis back to official plugin

* Remove CLI based task

* Fix forbiddenApisJava9
2018-10-23 12:06:46 +03:00
Jason Tedor f55843c9f0
Enable docs tests to have tear down snippet (#34716)
This commit adds the ability for docs tests to add a tear down
snippet. This snippet will be converted to a tear down section of the
generated REST tests.
2018-10-22 19:41:12 -04:00
Jason Tedor 66ebcdc112
Enable using defined setups from test setup in docs (#34681)
In the docs tests, we have pre-defined setups in the build.gradle file,
and we can also define test setup sections within the doc page
itself. Alas, these two are incompatible in that if you try to use a
pre-defined setup alongside a test setup section, the pre-defined setup
will be silently ignored. This commit enables pre-defined setup sections
to be used together with test setup sections. The ordering here is that
pre-defined setup sections will be executed first, followed by the test
setup section.
2018-10-22 11:14:39 -04:00
Jason Tedor 19cc4439d7
Fix cluster formation BWC versions
We support JDK 10 through the 6.3 and 6.4 series. This commit adjusts
the BWC logic when starting nodes during tests to account for this.
2018-10-20 15:58:38 -04:00
Jason Tedor 48ab6d7c92
Switch old reindex tests to use use JDK 8 (#34646)
This commit switches the old reindex tests to use JDK 8. This enables us
to not require that JDK 7 be installed, easing the burden on developers.
2018-10-19 15:11:48 -04:00
Jim Ferenczi 7b49beb9b0
Fix threshold frequency computation in Suggesters (#34312)
The `term` and `phrase` suggesters have different options to filter candidates
based on their frequencies. The `popular` mode for instance filters candidate
terms that occur in less docs than the original term. However when we compute this threshold
we use the total term frequency of a term instead of the document frequency. This is not inline
with the actual filtering which is always based on the document frequency. This change fixes
this discrepancy and clarifies the meaning of the different frequencies in use in the suggesters.
It also ensures that the threshold doesn't overflow the maximum allowed value (Integer.MAX_VALUE).

Closes #34282
2018-10-19 13:33:19 +02:00
Nik Everett 6c07d105f3
Amazon: Wrap at 140 columns (#34495)
Applies our standard column wrapping to the `discovery-ec2` and
`repository-s3` plugins.
2018-10-18 09:09:09 -04:00
Nik Everett a45626deb5
Analysis: Wrap at 140 columns (#34494)
Applies our standard column width to all analysis plugins.
2018-10-17 16:17:25 -04:00
Nik Everett 1452d55155
Expressions: Wrap at 140 columns (#34493)
Apply our standard line length to all of the java files in the
`lang-expression` plugin.
2018-10-17 16:12:21 -04:00
Nik Everett 5d74f8085b
SMB: Wrap at 140 columns (#34492)
Applies our max column width to the store-smb project.
2018-10-17 16:04:41 -04:00
Nik Everett b6aa42777a
Search: Wrap lucene classes at 140 columns (#34491)
Applies our line length guidance for all classes in the server in `lucene`
directories *except* `XMoreLikeThis`. The only long line in
`XMoreLikeThis` says "remove this when we upgrade to Lucene 5. Given
that we're on Lucene 8, this is a little terrifying and deserves another
look.
2018-10-17 15:54:35 -04:00
Nik Everett 2cccd4c669
LLREST: Wrap at 140 columns (#34489)
Wraps a single line in a test to fit on 140 characters, removing the
only checkstyle suppression for the clients directory.
2018-10-17 09:17:02 -04:00
Nik Everett 2cc3caf5e2
Build: Drop missing checkstyle suppressions (#34490)
This drops checkstyle suppressions that refer to files that don't exist
since those suppressions don't do anything other than make us feel bad.

It also updates some suppressions to more closely match the path to the
file that they suppress. These suppressions are still needed but didn't
pass the "the file exists" test because they weren't precise. It is just
easier on future-me if they are precise.
2018-10-16 11:14:11 -04:00
Armin Braun 7ba320d5b2
NETWORKING: Upgrade Netty to 4.1.30 (#34417)
* closes #34411
2018-10-12 16:13:28 +01:00
Alpar Torok c031322c18
[Build] randomizedtesting: Allow property values to be closures (#34319)
Makes it easier to have property values that are evaluated at build time
rather than configuration time in a Gradle idiomatic way.
2018-10-05 18:57:23 +03:00
Alpar Torok 7df842c78d
Rename clsuterformation -> testclusters (#34299) 2018-10-05 10:51:04 +03:00
Alpar Torok 0065e72be6
[Build] make sure there are no duplicate classes in third party audit (#34213)
* make sure there are no duplicate classes in third party audit
2018-10-05 10:47:15 +03:00
Nik Everett dc2cf28fde
Docs: Allow skipping response assertions (#34240)
We generate tests from our documentation, including assertions about the
responses returned by a particular API. But sometimes we *can't* assert
that the response is correct because of some defficiency in our tooling.
Previously we marked the response `// NOTCONSOLE` to skip it, but this
is kind of odd because `// NOTCONSOLE` is really to mark snippets that
are json but aren't requests or responses. This introduces a new
construct to skip response assertions:
```
// TESTRESPONSE[skip:reason we skipped this]
```
2018-10-04 08:03:38 -04:00
Kazuhiro Sera d45fe43a68 Fix a variety of typos and misspelled words (#32792) 2018-10-03 18:11:38 +01:00
Jason Tedor 86642d29e5
Require JDK 11 for compilation (#34103)
Now that JDK 11 is GA, we would switch our 6.x and master branches to
the JDK 11 compiler. This commit makes this change, as well as removes
JDK 10 from the CI configuration.
2018-10-03 08:59:00 -04:00
Nik Everett d3a4fe9a8e
Docs: Wrap expert script example to fit in docs (#34201)
This slightly reworks the expert script plugin example so it fits on the
page when the docs are rendered. The box in which it is rendered is not
very wide so it took a bit of twisting to make it readable.
2018-10-02 12:25:58 -04:00
David Turner 2aff005a69
Clean up TransportMasterNodeAction (#34076)
Mainly this fixes a warning by replacing the unchecked `new ActionListener`
with the checked `new ActionListener<Response>`, and it also fixes the line
length violations in this class.
2018-10-02 03:17:55 +01:00
Nik Everett ad3218b4ab
Checkstyle: Catch wide snippets (#34163)
We use wrap code in `// tag` and `//end` to include it in our docs. Our
current docs style wraps code snippets in a box that is only wide enough
for 76 characters and adds a horizontal scroll bar for wider snippets
which makes the snippet much harder to read. This adds a checkstyle check
that looks for java code that is included in the docs and is wider than
that 76 characters so all snippets fit into the box. It solves many of
the failures that this catches but suppresses many more. I will clean
those up in a follow up change.
2018-10-01 15:22:15 -04:00
Jason Tedor 128ea1b7b5
Add Ubuntu 18.04 to packaging tests (#34139)
We intend to start supporting Ubuntu 18.04 with the 6.5.0 release. This
commit adds Ubuntu 18.04 to the packaging tests.
2018-09-28 17:33:29 -04:00
Christoph Büscher b654d986d7
Add OneStatementPerLineCheck to Checkstyle rules (#33682)
This change adds the OneStatementPerLineCheck to our checkstyle precommit
checks. This rule restricts the number of statements per line to one. The
resoning behind this is that it is very difficult to read multiple statements on
one line. People seem to mostly use it in short lambdas and switch statements in
our code base, but just going through the changes already uncovered some actual
problems in randomization in test code, so I think its worth it.
2018-09-21 11:52:31 +02:00
Nik Everett 3ede13a454
Test framework fall cleaning (#33423)
Wraps all lines in our test framework at 140 characters because that is
our standard line length and removes all of the checkstyle suppressions
for the test framework.

Drops most of `ModuleTestCase` because it isn't used and we're moving
away from using guice in the way that it wants to test anyway. Also
switches a few classes that extend it but don't use it to extend
`ESTestCase` instead.
2018-09-19 14:34:02 -04:00
Ryan Ernst 99513b306e
Test: Relax jarhell gradle test (#33787)
Gradle can sometimes emit mixed log lines due to how we spawn things in
separate processes. This commit changes the jarhell integ test to only
look for the exception and message, instead of including the outer part
about the exception in thread main.

closes #33774
2018-09-18 13:11:46 -07:00
Jim Ferenczi 241c74efb2
upgrade to a new snapshot of Lucene 8 (7d0a7782fa) (#33812) 2018-09-18 18:16:40 +02:00
Nik Everett bc12a948b5
Checkstyle: Package declarations (#33784)
Make sure that all java files have a package declaration and that all of
the package declarations line up with the directory structure. This
would have caught the bug that I caused in
190ea9a6de and fixed in
b6d68bd805.
2018-09-18 12:04:35 -04:00
Armin Braun 87cedef3cf
NETWORKING:Def CName in Http Publish Addr to True (#33631)
* Follow up to #32806 setting the setting to true for 7.x
2018-09-18 10:29:02 +02:00
Ryan Ernst e686909768
Build: Change test task sanity check to be per project (#33544)
This commit changes the sanity check which ensures the test task was
properly replaced with randomized testing to have a per project check,
isntead of a global one. The previous global check assumed all test
tasks within the root project and below should be randomized testing,
but that is not the case for a multi project in which only one project
is an elasticsearch plugin. While the new check is not able to emit all
of the failed replacements in one error message, the efficacy of the
check remains.
2018-09-17 15:10:16 -07:00
Alpar Torok 5ca6f31205
Move precommit task implementation to java (#33407)
Replace precommit tasks that execute with Java implementations
2018-09-17 14:09:28 +03:00
Alpar Torok 896b3864a8
Favor running compile tasks before pre-commit (#33424) 2018-09-17 12:04:57 +03:00
Ryan Ernst 3046656ab1
Scripting: Rework joda time backcompat (#33486)
This commit switches the joda time backcompat in scripting to use
augmentation over ZonedDateTime. The augmentation methods provide
compatibility with the missing methods between joda's DateTime and
java's ZonedDateTime. Due to getDayOfWeek returning an enum in the java
API, ZonedDateTime is wrapped so that the method can return int like the
joda time does. The java time api version is renamed to
getDayOfWeekEnum, which will be kept through 7.x for compatibility while
users switch back to getDayOfWeek once joda compatibility is removed.
2018-09-16 19:18:00 -07:00
Vladimir Dolzhenko 141c6ef93e
upgrade randomizedrunner to 2.7.0 (#33623) 2018-09-12 17:18:34 +02:00
Armin Braun 94cdf0ceba
NETWORKING: http.publish_host Should Contain CNAME (#32806)
* NETWORKING: http.publish_host Should Contain CNAME

* Closes #22029
2018-09-12 06:15:36 +02:00
Alan Woodward 39c3234c2f
Upgrade to latest Lucene snapshot (#33505)
* LeafCollector.setScorer() now takes a Scorable
* Scorers may not have null Weights
* IndexWriter.getFlushingBytes() reports how much memory is being used by IW threads writing to disk
2018-09-10 20:51:55 +01:00
Tanguy Leroux 079d130d8c
[Test] Remove duplicate method in TestShardRouting (#32815) 2018-09-10 18:29:00 +02:00
Tanguy Leroux 80c4661d0c
Fix typos (#33499) 2018-09-10 14:20:04 +02:00
David Turner 284c45a6ff
Strengthen FilterRoutingTests (#33149)
Today the FilterRoutingTests take the belt-and-braces approach of excluding
some node attribute values and including some others. This means that we don't
really test that both inclusion and exclusion work correctly: as long as one of
them works as expected then the test will pass. This change improves these
tests by only using one approach at once, demonstrating that both do indeed
work, and adds tests for various other scenarios too.
2018-09-10 11:23:05 +02:00
Nik Everett 190ea9a6de
Logging: Configure the node name when we have it (#32983)
Change the logging infrastructure to handle when the node name isn't
available in `elasticsearch.yml`. In that case the node name is not
available until long after logging is configured. The biggest change is
that the node name logging no longer fixed at pattern build time.
Instead it is read from a `SetOnce` on every print. If it is unset it is
printed as `unknown` so we have something that fits in the pattern.
On normal startup we don't log anything until the node name is available
so we never see the `unknown`s.
2018-09-07 14:31:23 -04:00
Jim Ferenczi 79cd6385fe
Collapse package structure for metrics aggs (#33463)
This change collapses all metrics aggregations classes into a single package `org.elasticsearch.aggregations.metrics`.
It also restricts the visibility of some classes (aggregators and factories) that should not be used outside of the package.

Relates #22868
2018-09-07 10:58:06 +02:00
Jim Ferenczi 7ad71f906a
Upgrade to a Lucene 8 snapshot (#33310)
The main benefit of the upgrade for users is the search optimization for top scored documents when the total hit count is not needed. However this optimization is not activated in this change, there is another issue opened to discuss how it should be integrated smoothly.
Some comments about the change:
* Tests that can produce negative scores have been adapted but we need to forbid them completely: #33309

Closes #32899
2018-09-06 14:42:06 +02:00
Nik Everett 74b87989d9
Build: Merge xpack checkstyle config into core (#33399)
Solves all of the xpack line length suppressions and then merges the
remainder of the xpack checkstyle_suppressions.xml file into the core
checkstyle_suppressions.xml file. At this point that just means the
antlr generated files for sql.

It also adds an exclusion to the line length tests for javadocs that
are just a URL. We have one such javadoc and breaking up the line would
make the link difficult to use.
2018-09-05 09:17:02 -04:00
Alpar Torok 7f7e8fd733
Disable assemble task instead of removing it (#33348) 2018-09-04 07:32:14 +03:00
Jason Tedor 09bf4e5f00
Introduce private settings (#33327)
This commit introduces the formal notion of a private setting. This
enables us to register some settings that we had previously not
registered as fully-fledged settings to avoid them being exposed via
APIs such as the create index API. For example, we had hacks in the
codebase to allow index.version.created to be passed around inside of
settings objects, but was not registered as a setting so that if a user
tried to use the setting on any API then they would get an
exception. This prevented users from setting index.version.created on
index creation, or updating it via the index settings API. By
introducing private settings, we can continue to reject these attempts,
yet now we can represent these settings as actual settings. In this
change, we register index.version.created as an actual setting. We do
not cutover all settings that we had been treating as private in this
pull request, it is already quite large due to moving some tests around
to account for the fact that some tests need to be able to set the
index.version.created. This can be done in a follow-up change.
2018-09-03 19:17:57 -04:00
Alpar Torok 3c367a2c46
Fix extractjar task ci (#33272)
Remove tasks to check license and notice and add build integration test instead.

Closes #33201
2018-09-03 10:18:34 +03:00
Alpar Torok db3d32ce91
Fix pom for build-tools (#33300)
Looks like `java-gradle-plugin` reconfigures the pom.
Stop using it since we don't publish to Gradle plugin portal.
2018-08-31 16:48:00 +03:00
Alpar Torok a84a20844b
Lazy evaluate java9home (#33301) 2018-08-31 16:36:57 +03:00
Alpar Torok 44ed5f6306
Enable forbiddenapis server java9 (#33245) 2018-08-31 09:31:55 +03:00
Alpar Torok 5cf6e0d4bc
Ignore module-info in jar hell checks (#33011)
* Ignore module-info in JarHell checks
* Add unit test
* integration test to test that jarhell is ran with precommit
2018-08-30 11:41:39 +03:00
Alpar Torok 9c541b8f72
Upgrade to latest Gradle 4.10 (#32801)
Upgrade to Gradle 4.10
2018-08-30 10:20:38 +03:00
Alpar Torok f097446066
Fix/30904 cluster formation part2 (#32877)
Gradle integration for the Cluster formation plugin with ref counting
2018-08-30 10:17:42 +03:00
Alpar Torok f29f0af7bc
Consider multi release jars when running third party audit (#33206)
Exclude classes meant for newer versions than what we are auditing against, those classes won't be found. There's no reason to exclude JDK classes from newer versions, with this PR, we will not extract them in the first place.
2018-08-29 09:53:04 +03:00
Jonathan Little 9d92a87ae6 Remove support for deprecated params._agg/_aggs for scripted metric aggregations (#32979) 2018-08-28 09:27:43 +01:00
Alpar Torok 2cc611604f
Run Third party audit with forbidden APIs CLI (part3/3) (#33052)
The new implementation is functional equivalent with the old, ant based one.
It parses task standard error to get the missing classes and violations in the same way.
I considered re-using ForbiddenApisCliTask but Gradle makes it hard to build inheritance with tasks that have task actions , since the order of the task actions can't be controlled.
This inheritance isn't dully desired either as the third party audit task is much more opinionated and we don't want to expose some of the configuration.
We could probably extract a common base class without any task actions, but probably more trouble than it's worth.

Closes #31715
2018-08-28 10:03:30 +03:00
Alpar Torok 71d5c66fd3
Fix plugin build test on Windows (#33078)
Fix plugin build test on Windows
2018-08-28 10:00:11 +03:00
Jay Modi 309fb22181
Build: forked compiler max memory matches jvmArgs (#33138)
This commit removes the setting of the fork options maximum memory size
in our build plugin and instead adds the value in the gradle.properties
file to be alongside the value set in jvmArgs.

This change is necessary when using parallel compilation as 512m is not
sufficient for parallel compilation on some machines.
2018-08-27 10:26:25 -06:00
Alpar Torok 974f839093
Fix forbiddenapis on java 11 (#33116)
Cap forbiddenapis to java version 10
2018-08-27 08:47:42 +03:00
Jim Ferenczi f4e9729d64
Remove unsupported Version.V_5_* (#32937)
This change removes the es 5x version constants and their usages.
2018-08-24 09:51:21 +02:00
Alpar Torok edd477a15e Fix the default pom file name (#33063)
Before this change the default was fixed at compile time and not picking
up changes in the build script.
2018-08-22 14:53:42 -04:00
Alpar Torok 82d10b484a
Run forbidden api checks with runtimeJavaVersion (#32947)
Run forbidden APIs checks with runtime hava version
2018-08-22 09:05:22 +03:00
Nik Everett 2c81d7f77e
Build: Rework shadow plugin configuration (#32409)
This reworks how we configure the `shadow` plugin in the build. The major
change is that we no longer bundle dependencies in the `compile` configuration,
instead we bundle dependencies in the new `bundle` configuration. This feels
more right because it is a little more "opt in" rather than "opt out" and the
name of the `bundle` configuration is a little more obvious.

As an neat side effect of this, the `runtimeElements` configuration used when
one project depends on another now contains exactly the dependencies needed
to run the project so you no longer need to reference projects that use the
shadow plugin like this:

```
testCompile project(path: ':client:rest-high-level', configuration: 'shadow')
```

You can instead use the much more normal:

```
testCompile "org.elasticsearch.client:elasticsearch-rest-high-level-client:${version}"
```
2018-08-21 20:03:28 -04:00
Alpar Torok e3700a9b8d
Only configure publishing if it's applied externally (#32351)
Only configure publishing if it's applied externally, reconfigure for hasClientJar
2018-08-20 13:46:21 +03:00
Alpar Torok 4b34b3f4aa
Set forbidden APIs target compatibility to compiler java version (#32935)
Set forbidden apis target compatibility to compiler version

Fix outstanding deprecation
2018-08-20 09:27:02 +03:00
Ioannis Kakavas e3aa68b0a9
[TEST] Run pre 6.4 nodes in non-FIPS JVMs (#32901)
Elasticsearch versions earlier than 6.4.0 cannot properly run in a
FIPS 140 JVM. This commit ensures that we use a non-FIPS JVM for
nodes that we spin up in BWC tests even when we're testing FIPS.
2018-08-17 18:23:13 +03:00
Jonathan Little a08127c072 Scripted metric aggregations: add deprecation warning and system property to control legacy params (#31597)
* Scripted metric aggregations: add deprecation warning and system property to control legacy params

Scripted metric aggregation params._agg/_aggs are replaced by state/states context variables. By default the old params are still present, and a deprecation warning is emitted when Scripted Metric Aggregations are used. A new system property can be used to disable the legacy params. This functionality will be removed in a future revision.

* Fix minor style issue and docs test failure

* Disable deprecated params._agg/_aggs in tests and revise tests to use state/states instead

* Add integration test covering deprecated scripted metrics aggs params._agg/_aggs access

* Disable deprecated params._agg/_aggs in docs integration tests and revise stored scripts to use state/states instead

* Revert unnecessary migrations doc change

A relevant note should be added in the changes destined for 7.0; this PR is going to be backported to 6.x.

* Replace deprecated _agg param bwc integration test with a couple of unit tests

* Fix compatibility test after merge

* Rename backwards compatibility system property per code review feedback

* Tweak deprecation warning text per review feedback
2018-08-17 13:11:18 +01:00
Alpar Torok d16562eab5
RFC: Test that example plugins build stand-alone (#32235)
Add tests for build-tools to make sure example plugins build stand-alone using it.

This will catch issues such as referencing files from the buildSrc directly, breaking external uses of build-tools.
2018-08-17 09:41:39 +03:00
Alpar Torok bb3eacf384
Revert "cluster formation DSL - Gradle integration - part 2 (#32028)" (#32876)
This reverts commit e680a64434.
2018-08-15 19:17:46 +03:00
Alpar Torok e680a64434
cluster formation DSL - Gradle integration - part 2 (#32028)
* Implement Version in java

- This allows to move all  all .java files from .groovy.
- Will prevent eclipse from tangling up in this setup
- make it possible to use Version from Java

* PR review comments

* Cluster formation plugin with reference counting

```
> Task :plugins:ingest-user-agent:listElasticSearchClusters
Starting cluster: myTestCluster
   * myTestCluster: /home/alpar/work/elastic/elasticsearch/plugins/ingest-user-agent/foo
Asked to unClaimAndStop myTestCluster, since cluster still has 1 claim it will not be stopped

> Task :plugins:ingest-user-agent:testme UP-TO-DATE
Stopping myTestCluster, since no of claims is 0
```

- Meant to auto manage the clusters lifecycle
- Add integration test for cluster formation

* Fix rebase

* Change to `useCluster` method on task
2018-08-15 19:16:10 +03:00
Andy Bristol 0b601c6631 [test] disable packaging tests for suse boxes 2018-08-13 08:56:50 -07:00
Alpar Torok 02f2fad57b
Add a task to run forbiddenapis using cli (#32076)
* Add a task to run forbiddenapis using cli

Add a task that offers an equivalent check to the forbidden APIs plugin,
but runs it using the forbiddenAPIs CLI instead.

This isn't wired into precommit first, and doesn't work for projects
that require specific signatures, etc. It's meant to show how this can
be used. The next step is to make a custom task type and configure it
based on the project extension from the pugin and make some minor
adjustments to some build scripts as we can't  bee 100% compatible with
that at least due to how additional signatures are passed.

Notes:
- there's no `--target` for the CLI version so we have to pass in
specific bundled signature names
- the cli task already wires to `runtimeJavaHome`
- no equivalent for `failOnUnsupportedJava = false` but doesn't seem to
be a problem. Tested with Java 8 to 11
- there's no way to pass additional signatures as URL, these will have
to be on the file system, and can't be resources on the cp unless we
rely on how forbiddenapis is implemented and mimic them as boundled
signatures.
- the average of 3 runs is 4% slower using the CLI for :server.
( `./gradlew clean :server:forbiddenApis` vs `./gradlew clean
:server:forbiddenApisCli`)
- up-to-date checks don't work on the cli task yet, that will happen
with the custom task.

See also: #31715
2018-08-13 08:52:16 +03:00
Alpar Torok ef21ab8567
Access build tools resources (#32201)
Add a buildResources 

buildResources.take('resource') writes the resources out to the filesystem so the paths can be sent int other tools.
2018-08-10 15:03:17 +00:00
Alpar Torok de2897a190 Only require java<version>_home env var if needed
We accidentally started enforcing the env vars are always there.
This reverts to only do this if they are requierd.
2018-08-10 16:10:13 +03:00
David Roberts ae0c303dad
Move icu4j and super-csv version numbers to versions file (#32769)
The upcoming ML log structure finder functionality will use these
libraries, and it makes sense to use the same versions that are
being used elsewhere in Elasticsearch.  This is especially true
with icu4j, which is pretty big.
2018-08-10 12:19:06 +01:00
Alpar Torok af8c23eb40
Java version reproduction (#32715)
Enhance reproduction line with info about jdks

Provide the ability to control compiler and hava versions just by
passing a property. The actual java home comes from the
`JAVA<major>_HOME` env vars that we allready require.
This works better with the Gradle daemon as well.

Output is also changed a bit.

for `-Druntime.java=8 -Dcompiler.java=9`:
```
=======================================
Elasticsearch Build Hamster says Hello!
  Gradle Version        : 4.9
  OS Info               : Linux 4.17.8-1-ARCH (amd64)
  Compiler JDK Version  : 11 (Oracle Corporation 11-ea [OpenJDK 64-Bit Server VM 11-ea+22])
  Runtime JDK Version   : 11 (Oracle Corporation 11-ea [OpenJDK 64-Bit Server VM 11-ea+22])
  Gradle JDK Version    : 10 (Oracle Corporation 10.0.1 [OpenJDK 64-Bit Server VM 10.0.1+10])
  Compiler java.home    : /home/alpar/opt/jdk-11-ea22/
  Runtime java.home     : /home/alpar/opt/jdk-11-ea22/
  Gradle java.home      : /usr/lib/jvm/java-10-openjdk
  Random Testing Seed   : EA858533191E8DFB
=======================================
```

Without configuration:
```
=======================================
Elasticsearch Build Hamster says Hello!
=======================================
  Gradle Version        : 4.9
  OS Info               : Linux 4.17.8-1-ARCH (amd64)
  JDK Version           : 10 (Oracle Corporation 10.0.1 [OpenJDK 64-Bit Server VM 10.0.1+10])
  JAVA_HOME             : /usr/lib/jvm/java-10-openjdk
  Random Testing Seed   : 4BD5B2A839C8FCA1
=======================================
```

Here's how a reproduction line will look like (test made to fail):
```
./gradlew :modules:lang-painless:test -Dtests.seed=2DA2379065A4EEAB -Dtests.class=org.elasticsearch.painless.AdditionTests -Dtests.method="testInt" -Dtests.security.manager=true -Dtests.locale=es-PE -Dtests.timezone=WET -Dcompiler.java=10 -Druntime.java=10
```
2018-08-10 08:07:43 +00:00
Jack Conradson 293c8a2b24
Painless: Add an Ingest Script Processor Example (#32302)
This commit adds two pieces. The first is a small set of documentation providing 
instructions on how to get setup to run context examples. This will require a download 
similar to how Kibana works for some of the examples. The second is an ingest processor 
example using the downloaded data. More examples will follow as ideally one per PR. 
This also adds a set of tests to individually test each script as a unit test.
2018-08-09 14:24:55 -07:00
Nicholas Knize e162127ff3 Upgrade to Lucene-7.5.0-snapshot-13b9e28f9d
The main feature is the inclusion of bkd backed geo_shape with
INTERSECT, DISJOINT, WITHIN bounding box and polygon query support.
2018-08-09 11:15:02 -05:00
Armin Braun 79375d35bb
Scripting: Replace Update Context (#32096)
* SCRIPTING: Move Update Scripts to their own context
* Added system property for backwards compatibility of change to `ctx.params`
2018-08-09 14:32:36 +02:00
Armin Braun 580d59e2d7
CORE: Upgrade to Jackson 2.8.11 (#32670)
* closes #30352
2018-08-08 12:04:25 +02:00
Jack Conradson b46e13629f
Docs: Allow snippets to have line continuation (#32649)
Currently, snippets in lists cannot be rendered correctly as a console command because the console command requires a line continuation '+'. This allows snippets to have a line continuation between the snippet and the // CONSOLE.
2018-08-06 14:43:53 -07:00
Armin Braun 0a67cb4133
LOGGING: Upgrade to Log4J 2.11.1 (#32616)
* LOGGING: Upgrade to Log4J 2.11.1
* Upgrade to `2.11.1` to fix memory leaks in slow logger when logging large requests
   * This was caused by a bug in Log4J https://issues.apache.org/jira/browse/LOG4J2-2269 and is fixed in `2.11.1` via https://git-wip-us.apache.org/repos/asf?p=logging-log4j2.git;h=9496c0c
* Fixes #32537
* Fixes #27300
2018-08-06 14:56:21 +02:00
Ryan Ernst 781bb8efe5
Build: Add elastic maven to repos used by BuildPlugin (#32549)
This commit adds the elastic repo, alongside maven central, to any
plugin using BuildPlugin. This is necessary now because the default
distribution, which the test framework uses by default, is now only
hosted on elastic maven. While inside the elasticsearch build this does
not matter, those that build external plugins with our build-tools could
have tests fail to find the distribution dependency.
2018-08-01 13:48:13 -07:00
Andy Bristol 418540e630 Enable packaging tests on suse boxes
This reverts commit 0f7d2044c3.
2018-08-01 11:25:36 -07:00
Ryan Ernst 478f6d6cf1
Scripting: Conditionally use java time api in scripting (#31441)
This commit adds a boolean system property, `es.scripting.use_java_time`,
which controls the concrete return type used by doc values within
scripts. The return type of accessing doc values for a date field is
changed to Object, essentially duck typing the type to allow
co-existence during the transition from joda time to java time.
2018-08-01 08:58:49 -07:00
Jim Ferenczi 53ff06e621
Upgrade to Lucene-7.5.0-snapshot-608f0277b0 (#32390)
The main highlight is the removal of the reclaim_deletes_weight in the TieredMergePolicy.
The es setting index.merge.policy.reclaim_deletes_weight is deprecated in this commit and the value is ignored. The new merge policy setting setDeletesPctAllowed should be added in a follow up.
2018-07-27 08:28:51 +02:00
Nik Everett 0f783ce94b
Build: Stop double generating buildSrc pom (#32408)
When we added the `java-gradle-plugin` to `buildSrc` it added a second
task to generate the pom that duplicates the publishing work that we
configure in `BuildPlugin`. Not only does it dupliciate the pom, it
creates a pom that is missing things like `name` and `description` which
are required for publishing to maven central.

This change disables the duplicate pom generation.
2018-07-26 13:19:05 -04:00
Alpar Torok 6aea82956c
Number of utilities for writing gradle integration tests (#32282)
These are collected from a number of open PRs and are required to
improove existing and write more readable future tests.
I am extracting them to their own PR hoping to be able to merge and use
them sooner.
2018-07-25 06:22:54 +00:00
Alpar Torok b6c14935d7
Determine the minimum gradle version based on the wrapper (#32226)
* Determine the minimum gradle version based on the wrapper

This is restrictive and forces users of the plugin to move together with
us, but without integration tests it's close to impossible to make sure
that the claimed compatability is really there.
If we do want to offer more flexibility, we should add those tests
first.

* Track gradle version in individual file

* PR review
2018-07-25 06:21:03 +00:00
Nik Everett e6b9f59e4e
Build: Shadow x-pack:protocol into x-pack:plugin:core (#32240)
This bundles the x-pack:protocol project into the x-pack:plugin:core
project because we'd like folks to consider it an implementation detail
of our build rather than a separate artifact to be managed and depended
on. It is now bundled into both x-pack:plugin:core and
client:rest-high-level. To make this work I had to fix a few things.

Firstly, I had to make PluginBuildPlugin work with the shadow plugin.
In that case we have to bundle only the `shadow` dependencies and the
shadow jar.

Secondly, every reference to x-pack:plugin:core has to use the `shadow`
configuration. Without that the reference is missing all of the
un-shadowed dependencies. I tried to make it so that applying the shadow
plugin automatically redefines the `default` configuration to mirror the
`shadow` configuration which would allow us to use bare project references
to the x-pack:plugin:core project but I couldn't make it work. It'd *look*
like it works but then fail for transitive dependencies anyway. I think
it is still a good thing to do but I don't have the willpower to do it
now.

Finally, I had to fix an issue where Eclipse and IntelliJ didn't properly
reference shadowed transitive dependencies. Neither IDE supports shadowing
natively so they have to reference the shadowed projects. We fix this by
detecting `shadow` dependencies when in "Intellij mode" or "Eclipse mode"
and adding `runtime` dependencies to the same target. This convinces
IntelliJ and Eclipse to play nice.
2018-07-24 11:53:04 -04:00
Ioannis Kakavas a2dbd83db1
Allow Integ Tests to run in a FIPS-140 JVM (#31989)
* Complete changes for running IT in a fips JVM

- Mute :x-pack:qa:sql:security:ssl:integTest as it
  cannot run in FIPS 140 JVM until the SQL CLI supports key/cert.
- Set default JVM keystore/truststore password in top level build
  script for all integTest tasks in a FIPS 140 JVM
- Changed top level x-pack build script to use keys and certificates
  for trust/key material when spinning up clusters for IT
2018-07-24 12:48:14 +03:00
Alpar Torok 0a511cc76f
Improve message when JAVA_HOME not set (#32022)
closes #31399
2018-07-21 12:50:17 +00:00
Alpar Torok a9a9598e02
Require Gradle 4.9 as minimum version (#32200)
Do the check in the build plugin as well to be more informative to users
of  build-tools.

We use APIs specific to Gradle 4.9 so earlier will not work.
2018-07-20 05:20:35 +00:00
Mayya Sharipova 4c68dfe001
Handle missing values in painless (#32207)
Throw an exception for doc['field'].value
if this document is missing a value for the field.

After deprecation changes have been backported to 6.x,
make this a default behaviour in 7.0

Closes #29286
2018-07-19 17:41:06 -04:00
Vladimir Dolzhenko 7c0fc209bf
ECS Task IAM profile credentials ignored in repository-s3 plugin (#31864)
ECS Task IAM profile credentials ignored in repository-s3 plugin (#31864)

Closes #26913
2018-07-19 12:54:38 +02:00
Ryan Ernst 185689ad75
Build: Remove pom generation for plugin zip files (#32180)
In 1.x and 2.x, plugins were published to maven and the plugin
installer downloaded them from there. This was later changed to install
from the download service, and in 5.0 plugin zips were no longer
published to maven. However, the build still currently produces an
unused pom file. This is troublesome in the special case when the main
jar of a plugin needs to be published (and thus needs a pom file of
the same name).

closes #31946
2018-07-19 01:24:49 -07:00
Vladimir Dolzhenko 8235b254ab
Add EC2 credential test for repository-s3 (#31918)
Add EC2 credential test for repository-s3

Relates to #26913
2018-07-18 12:18:00 +02:00
Alpar Torok 69cbdc47eb
Fix CP for namingConventions when gradle home has spaces (#31914)
* Fix CP for namingConventions when gradle home has spaces

Closes #31736.

Probably not Windows specific, just not common to have spaces on Linux.
2018-07-18 08:23:59 +00:00
Ryan Ernst 6371d51866
Build: Make additional test deps of check (#32015)
This commit moves additional unit test runners from being dependencies
of the test task to dependencies of check. Without this change,
reproduce lines are incorrect due to the additional test runner not
matching any of the reproduce class/method info.

closes #31964
2018-07-17 13:14:46 -07:00
Nik Everett 1b97652a4c
Build: Move shadow customizations into common code (#32014)
Moves the customizations to the build to produce nice shadow jars and
javadocs into common build code, mostly BuildPlugin with a little into
the root build.gradle file. This means that any project that applies the
shadow plugin will automatically be set up just like the high level rest
client:
* The non-shadow jar will not be built
* The shadow jar will not have a "classifier"
* Tests will run against the shadow jar
* Javadoc will include all of the shadowed classes
* Service files in `META-INF/services` will be merged
2018-07-17 14:20:41 -04:00
Andy Bristol 0f7d2044c3 Re-disable packaging tests on suse boxes
This reverts commit 14d7e2c7b2.
2018-07-17 09:15:11 -07:00
Alpar Torok b43fe560a4
Updates the build to gradle 4.9 (#32087)
There are fixes to the dependency report, most importantly for us,
it still works even if `failOnVersionConflict` would fail the build.
2018-07-17 11:41:31 +00:00
Luca Cavanna 049966a829
Check that client methods match API defined in the REST spec (#31825)
We have been encountering name mismatches between API defined in our
REST spec and method names that have been added to the high-level REST
client. We should check this automatically to prevent furher mismatches,
and correct all the current ones.

This commit adds a test for this and corrects the issues found by it.
2018-07-17 11:26:28 +02:00
Andy Bristol 14d7e2c7b2 Revert "[test] disable packaging tests for suse boxes"
This reverts commit 30d6fd3ca7.
2018-07-16 13:57:17 -07:00
Daniel Mitterdorfer e0cfa1689c
Turn off real-mem breaker in REST tests
With this commit we disable the real-memory circuit breaker in REST
tests as this breaker is based on real memory usage over which we have
no (full) control in tests and the REST client is not yet ready to retry
on circuit breaker exceptions.

This is only meant as a temporary measure to avoid spurious test
failures while we ensure that the REST client can handle those
situations appropriately.

Closes #32050
Relates #31767
Relates #31986 
Relates #32074
2018-07-16 10:44:04 +02:00
Alpar Torok 23198ea551
Convert Version to Java - clusterformation part1 (#32009)
Implement buildSrc Version in java

- This allows to move all  all .java files from .groovy.
- Will prevent eclipse from tangling up in this setup
- make it possible to use Version from Java
2018-07-13 08:35:12 +00:00
Andy Bristol 30d6fd3ca7 [test] disable packaging tests for suse boxes
For #30295
2018-07-12 11:12:37 -07:00
Alpar Torok ba3c0f2478
Move build integration tests out of :buildSrc project (#31961)
This way building buildSrc will not be delayed for all invocations, and
will be less disruptive if integration tests fail
2018-07-12 08:41:07 +00:00
Andy Bristol 006c2c9ab0
[test] port archive distribution packaging tests (#31314)
Recreates the rest of the bats packaging tests for the tar distribution
in the java packaging test project, with support for both tar and zip
packaging, both oss and default flavors, and on Linux and Windows. Most
tests are followed fairly closely, some have either been dropped if
unnecessary or folded into others if convenient.
2018-07-10 08:21:20 -07:00
Mayya Sharipova 5481fbc249
Handle missing values in painless (#30975)
* Handle missing values in painless

Throw an exception for `doc['field'].value`
if this document is missing a value for the `field`.

For 7.0:
This is the default behaviour from 7.0

For 6.x:
To enable this behavior from 6.x, a user can set a jvm.option:
 `-Des.script.exception_for_missing_value=true` on a node.
If a user does not enable this behavior, a deprecation warning is logged on start up.

Closes #29286
2018-07-09 11:59:49 -04:00
Alpar Torok cf2295b408
Add JDK11 support and enable in CI (#31644)
* Upgrade bouncycastle

Required to fix
`bcprov-jdk15on-1.55.jar; invalid manifest format `
on jdk 11

* Downgrade bouncycastle to avoid invalid manifest

* Add checksum for new jars

* Update tika permissions for jdk 11

* Mute test failing on jdk 11

* Add JDK11 to CI

* Thread#stop(Throwable) was removed

http://mail.openjdk.java.net/pipermail/core-libs-dev/2018-June/053536.html

* Disable failing tests #31456

* Temprorarily disable doc tests

To see if there are other failures on JDK11

* Only blacklist specific doc tests

* Disable only failing tests in ingest attachment plugin

* Mute failing HDFS tests #31498

* Mute failing lang-painless tests #31500

* Fix backwards compatability builds

Fix JAVA version to 10 for ES 6.3

* Add 6.x to bwx -> java10

* Prefix out and err from buildBwcVersion for readability

```
> Task :distribution:bwc:next-bugfix-snapshot:buildBwcVersion
  [bwc] :buildSrc:compileJava
  [bwc] WARNING: An illegal reflective access operation has occurred
  [bwc] WARNING: Illegal reflective access by org.codehaus.groovy.reflection.CachedClass (file:/home/alpar/.gradle/wrapper/dists/gradle-4.5-all/cg9lyzfg3iwv6fa00os9gcgj4/gradle-4.5/lib/groovy-all-2.4.12.jar) to method java.lang.Object.finalize()
  [bwc] WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.reflection.CachedClass
  [bwc] WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
  [bwc] WARNING: All illegal access operations will be denied in a future release
  [bwc] :buildSrc:compileGroovy
  [bwc] :buildSrc:writeVersionProperties
  [bwc] :buildSrc:processResources
  [bwc] :buildSrc:classes
  [bwc] :buildSrc:jar

```

* Also set RUNTIME_JAVA_HOME for bwcBuild

So that we can make sure it's not too new for the build to understand.

* Align bouncycastle dependency

* fix painles array tets

closes #31500

* Update jar checksums

* Keep 8/10 runtime/compile untill consensus builds on 11

* Only skip failing tests if running on Java 11

* Failures are dependent of compile java version not runtime

* Condition doc test exceptions on compiler java version as well

* Disable hdfs tests based on runtime java

* Set runtime java to minimum supported for bwc

* PR review

* Add comment with ticket for forbidden apis
2018-07-05 03:24:01 +00:00
Nik Everett 144735e62a Build test: Thread linger
Add a thread linger filter to the build test so we don't spuriously fail
waiting on the "connection worker" thread.
2018-06-29 15:19:07 -04:00
Alpar Torok 200e1f45f2
Fix gradle4.8 deprecation warnings (#31654)
* remove explicit wrapper task

It's created by Gradle and triggers a deprecation warning
Simplify configuration

* Upgrade shadow plugin to get rid of Gradle deprecation

* Move compile configuration to base plugin

Solves Gradle deprecation warning from earlier Gradle versions

* Enable stable publishing in the Gradle build

* Replace usage of deprecated property

* bump Gradle version in build compare
2018-06-29 19:17:19 +00:00
Alpar Torok f702086df1 Build: Fix naming conventions task (#31681)
Move min compiler and runtime version to files so we can fix the source
and target comparability of the build script to those

Closes #31665
2018-06-29 14:41:02 -04:00
Zachary Tong 1af31f441a [TEST] Mute failing NamingConventionsTaskIT tests
Note: no awaitsFix available, so I fell back to JUnit @Ignore

Tracking issue: #31665
2018-06-28 16:22:35 +00:00
Alpar Torok 304316ac3b
Print output when the name checker IT fails (#31660) 2018-06-28 17:58:24 +03:00
Alpar Torok 0afec8f31c
Remove deprecation warnings to prepare for Gradle 5 (sourceSets.main.output.classesDirs) (#30389)
* Remove deprecation warnings to prepare for Gradle 5

Gradle replaced `project.sourceSets.main.output.classesDir` of type
`File` with `project.sourceSets.main.output.classesDirs` of type
`FileCollection`
(see [SourceSetOutput](https://github.com/gradle/gradle/blob/master/subprojects/plugins/src/main/java/org/gradle/api/tasks/SourceSetOutput.java))
Build output is now stored on a per language folder.

There are a few places where we use that, here's these and how it's
fixed:

- Randomized Test execution
    - look in all test folders ( pass the multi dir configuration to the
    ant runner )
    - DRY the task configuration by introducing `basedOn` for
      `RandomizedTestingTask` DSL
- Extend the naming convention test to support passing in multiple
  directories
- Fix the standalon test plugin, the dires were not passed trough,
  checked with a debuger and the statement had no affect due to a
  missing `=`.

Closes #30354

* Only check Java tests, PR feedback

- Name checker was ran for Groovy tests that don't adhere to the same
  convections causing the check to fail
- implement PR feedback

* Replace `add` with `addAll`

This worked because the list is passed to `project.files` that does the
right thing.

* Revert "Only check Java tests, PR feedback"

This reverts commit 9bd9389875d8b88aadb50df57a45cd0d2b073241.

* Remove `basedOn` helper

* Bring some changes back

Previus revert accidentally reverted too much

* Fix negation

* add back public

* revert name check changes

* Revert "revert name check changes"

This reverts commit a2800c0b363168339ea65e2a79ec8256e5883e6d.

* Pass all dirs to name check

Only run on Java for build-tools, this is safe because it's a self test.
It needs more work before we could pass in the Groovy classes as well as
these inherit from `GroovyTestCase`

* remove self tests from name check

The self complicates the task setup and disable real checks on
build-tools.
With this change there are no more self tests, and the build-tools tests
adhere to the conventions.
The self test will be replaced by gradle test kit, thus the addition of
the Gradle plugin builder plugin.

* First test to run a Gradle build

* Add tests that replace the name check self test

* Clean up integ test base class

* Always run tests

* Align with test naming conventions

* Make integ. test case inherit from unit test case

The check requires this

* Remove `import static org.junit.Assert.*`
2018-06-28 15:14:34 +03:00
Alpar Torok 8557bbab28
Upgrade gradle wrapper to 4.8 (#31525)
* Move to Gradle 4.8 RC1

* Use latest version of plugin

The current does not work with Gradle 4.8 RC1

* Switch to Gradle GA

* Add and configure build compare plugin

* add work-around for https://github.com/gradle/gradle/issues/5692

* work around https://github.com/gradle/gradle/issues/5696

* Make use of Gradle build compare with reference project

* Make the manifest more compare friendly

* Clear the manifest in compare friendly mode

* Remove animalsniffer from buildscript classpath

* Fix javadoc errors

* Fix doc issues

* reference Gradle issues in comments

* Conditionally configure build compare

* Fix some more doclint issues

* fix typo in build script

* Add sanity check to make sure the test task was replaced

Relates to #31324. It seems like Gradle has an inconsistent behavior and
the taks is not always replaced.

* Include number of non conforming tasks in the exception.

* No longer replace test task, create implicit instead

Closes #31324. The issue has full context in comments.

With this change the `test` task becomes nothing more than an alias for `utest`.
Some of the stand alone tests that had a `test` task now have `integTest`, and a
few of them that used to have `integTest` to run multiple tests now only
have `check`.
This will also help separarate unit/micro tests from integration tests.

* Revert "No longer replace test task, create implicit instead"

This reverts commit f1ebaf7d93e4a0a19e751109bf620477dc35023c.

* Fix replacement of the test task

Based on information from gradle/gradle#5730 replace the task taking
into account the task providres.
Closes #31324.

* Only apply build comapare plugin if needed

* Make sure test runs before integTest

* Fix doclint aftter merge

* PR review comments

* Switch to Gradle 4.8.1 and remove workaround

* PR review comments

* Consolidate task ordering
2018-06-28 08:13:21 +03:00
Nik Everett b8907b04b1
Docs: Skip xpack snippet tests if no xpack (#31619)
Skips tests the require xpack if we run the doc build without xpack. So
this should work:
```
./gradlew -p docs check -Dtests.distribution=oss-zip
```

This is implemented by detecting parts of the doc that look like:
```
[testenv="basic"]
```

Relates to #30665
2018-06-27 15:41:44 -04:00
Alpar Torok db90905405 reduce log level at gradle configuration time 2018-06-26 18:16:24 +03:00
Alpar Torok 08b8d11e30
Add support for switching distribution for all integration tests (#30874)
* remove left-over comment

* make sure of the property for plugins

* skip installing modules if these exist in the distribution

* Log the distrbution being ran

* Don't allow running with integ-tests-zip passed externally

* top level x-pack/qa can't run with oss distro

* Add support for matching objects in lists

Makes it possible to have a key that points to a list and assert that a
certain object is present in the list. All keys have to be present and
values have to match. The objects in the source list may have additional
fields.

example:
```
  match:  { 'nodes.$master.plugins': { name: ingest-attachment }  }
```

* Update plugin and module tests to work with other distributions

Some of the tests expected that the integration tests will always be ran
with  the `integ-test-zip` distribution so that there will be no other
plugins loaded.

With this change, we check for the presence of the plugin without
assuming exclusivity.

* Allow modules to run on other distros as well

To match the behavior of tets.distributions

* Add and use a new `contains` assertion

Replaces the  previus changes that caused `match` to do a partial match.

* Implement PR review comments
2018-06-26 06:49:03 -07:00
Adrien Grand f023e95ae0
Upgrade to Lucene 7.4.0. (#31529)
This moves Elasticsearch from a recent 7.4.0 snapshot to the GA release.
2018-06-22 16:17:17 +02:00
David Kyle 6ebe6e32cf
[DOCS] Omit shard failures assertion for incompatible responses (#31430)
Filter out the assertion for _cat and _xpack/ml/datafeed APIs
2018-06-20 16:57:51 +01:00
Nik Everett 73549281e8
Docs: Use the default distribution to test docs (#31251)
This switches the docs tests from the `oss-zip` distribution to the
`zip` distribution so they have xpack installed and configured with the
default basic license. The goal is to be able to merge the
`x-pack/docs` directory into the `docs` directory, marking the x-pack
docs with some kind of marker. This is the first step in that process.

This also enables `-Dtests.distribution` support for the `docs`
directory so you can run the tests against the `oss-zip` distribution
with something like
```
./gradlew -p docs check -Dtests.distribution=oss-zip
```

We can set up Jenkins to run both.

Relates to #30665
2018-06-18 12:06:42 -04:00
Nhat Nguyen 8453ca638d
Upgrade to Lucene-7.4.0-snapshot-518d303506 (#31360) 2018-06-15 10:58:21 -04:00
Tanguy Leroux 4d7447cb5e
Reenable Checkstyle's unused import rule (#31270) 2018-06-14 09:52:46 +02:00
Tanguy Leroux 8b4d80ad09
Fix AntFixture waiting condition (#31272)
The AntFixture waiting condition is evaluated to false 
but it should be true.
2018-06-13 12:40:22 +02:00
Tanguy Leroux 1f6e874002
Update checkstyle to 8.10.1 (#31269) 2018-06-13 09:22:17 +02:00
Christoph Büscher a178290b60
Remove some line length supressions (#31209) 2018-06-12 10:03:51 +02:00
Nhat Nguyen dda56fc0fc
Move ESIndexLevelReplicationTestCase to test framework (#31243)
Other components might benefit from the testing infra provided by
ESIndexLevelReplicationTestCase. This commit moves it to the test
framework.
2018-06-11 12:47:38 -04:00
Jason Tedor 1082ea57c8
Add recognition of MPL 2.0 (#31226)
This commit adds the ability for the dependencies info check to
recognize version 2.0 of the Mozilla Public License.
2018-06-09 09:46:57 -04:00
Jason Tedor 65c107b47d
Fix unknown licenses (#31223)
The goal of this commit is to address unknown licenses when producing
the dependencies info report. We have two different checks that we run
on licenses. The first check is whether or not we have stashed a copy of
the license text for a dependency in the repository. The second is to
map every dependency to a license type (e.g., BSD 3-clause). The problem
here is that the way we were handling licenses in the second check
differs from how we handle licenses in the first check. The first check
works by finding a license file with the name of the artifact followed
by the text -LICENSE.txt. Yet in some cases we allow mapping an artifact
name to another name used to check for the license (e.g., we map
lucene-.* to lucene, and opensaml-.* to shibboleth. The second check
understood the first way of looking for a license file but not the
second way. So in this commit we teach the second check about the
mappings from artifact names to license names. We do this by copying the
configuration from the dependencyLicenses task to the dependenciesInfo
task and then reusing the code from the first check in the second
check. There were some other challenges here though. For example,
dependenciesInfo was checking too many dependencies. For now, we should
only be checking direct dependencies and leaving transitive dependencies
from another org.elasticsearch artifact to that artifact (we want to do
this differently in a follow-up). We also want to disable
dependenciesInfo for projects that we do not publish, users only care
about licenses they might be exposed to if they use our assembled
products. With all of the changes in this commit we have eliminated all
unknown licenses. A follow-up will enforce that when we add a new
dependency it does not get mapped to unknown, these will be forbidden in
the future. Therefore, with this change and earlier changes are left
having no unknown licenses and two custom licenses; custom here means it
does not map to an SPDX license type. Those two licenses are xz and
ldapsdk. A future change will not allow additional custom licenses
unless they are explicitly whitelisted. This ensures that if a new
dependency is added it is mapped to an SPDX license or mapped to custom
because it does not have an SPDX license.
2018-06-09 07:28:41 -04:00
Jason Tedor 3db1fe7afe
Remove version from license file name for GCS SDK (#31221)
Most of our license file names strip the version off the artifact name
when deducing the license filename. However, the version on the GCS SDK
(google-api-services-storage) does not match the usual format and
instead starts with a vee. This means that the license filename for this
license ended up carrying the version and we should not do that. This
commit adjusts the regex the deduces the license filename to account for
this case, and adjusts the google-api-services-storage license files
accordingly.
2018-06-08 21:19:16 -04:00
Jason Tedor e467e67fd4
Enhance license detection for various licenses (#31198)
This commit enhances the license detection that we have for various
licenses. Here we improve the detection for all licenses (especially the
Apache 2.0 License), the BSD 2-clause license, the MIT (with
attribution) license, and we add detection for the BSD 3-clause
license. One way that we achieved this improvement is by changing how
the license files are read so that rather than reading them as a
multi-line string which ended up represented as "[line1, line2, line3,
...]" internally, we read the full bytes of the license text and replace
all whitespace with a single space so the license text is now loaded as
"line1 line2 line3". For the MIT license we add the actual license text
and remove the "MIT" string as not all copies of the license clearly
indicate that the text is the MIT license. We take a similar strategy
for the BSD-2 and BSD-3 clause licenses. With this change, we reduce the
number of "custom" licenses in the codebase from 31 to 2. The two
remaining appear to be truly custom licenses, not carrying licenses
identifiable by SPDX. A follow-up will address "unknown" licenses.
2018-06-08 08:55:10 -04:00
Nik Everett dfcc939ef8 QA: Better seed nodes for rolling restart
Use all running nodes as unicast seeds in the rolling restart tests to
avoid a race between pinging and the tests. Without this if the tests
are too fast then when a new node comes up and pings its single
configured seed node that node *might* not have a ping from the other
running node.
2018-06-07 13:30:37 -04:00
Tanguy Leroux 9531b7bbcb
Add BlobContainer.writeBlobAtomic() (#30902)
This commit adds a new writeBlobAtomic() method to the BlobContainer
interface that can be implemented by repository implementations which
support atomic writes operations.

When the BlobContainer implementation does not provide a specific 
implementation of writeBlobAtomic(), then the writeBlob() method is used.

Related to #30680
2018-06-05 13:00:43 +02:00
Nhat Nguyen abe61159a8
Upgrade to Lucene-7.4.0-snapshot-0a7c3f462f (#31073)
This snapshot includes:
- LUCENE-8341: Record soft deletes in SegmentCommitInfo which will resolve #30851
- LUCENE-8335: Enforce soft-deletes field up-front
2018-06-04 14:18:46 -04:00
Nhat Nguyen 363f1e84ca
Upgrade to Lucene-7.4-snapshot-1cbadda4d3 (#30928)
This snapshot includes LUCENE-8328 which is needed to stabilize CCR builds.
2018-05-29 12:29:52 -04:00
Jason Tedor 0698dd017c
Enabling testing against an external cluster (#30885)
Today when executing REST tests we take full responsibility for cluster
configuration. Yet, there are use cases for brining your own cluster to
the REST tests. This commit is a small first step towards that effort by
skipping creating the cluster if the tests.rest.cluster and test.cluster
system properties are set. In this case, the user takes full
responsibility for configuring the cluster as expected by the REST
tests. This step is by no means meant to be perfect or complete, only a
baby step.
2018-05-26 08:04:53 -04:00
Ryan Ernst 2c7559c575
Packaging: Ensure upgrade_is_oss flag file is always deleted (#30732)
This commit ensures the delete of the upgrade_is_oss indicator for
the packaging tests is always deleted before each run. It works by
moving the check on version which skips the task into the doFirst block,
replacing the onlyIf.

closes #30682
2018-05-23 23:20:13 -07:00
Andy Bristol a1b538122c
[test] java tests for archive packaging (#30734)
Ports the first couple tests for archive distributions from the old bats
project to the new java project that includes windows platforms,
consolidating them into one test method that tests that the
distributions can be extracted and their contents verified. Includes the
zip distributions which were not tested in the bats project.
2018-05-23 10:37:57 -07:00
Nhat Nguyen 1918a30237
Upgrade to Lucene-7.4.0-snapshot-cc2ee23050 (#30778)
The new snapshot includes LUCENE-8324 which fixes missing checkpoint
after a fully deletes segment is dropped on flush. This snapshot should
resolves failed tests in the CorruptedFileIT suite.

Closes #30741
Closes #30577
2018-05-22 13:11:48 -04:00
Alpar Torok 8b7ff69479
Remove fedora 26, add 28 (#30683)
* Remove fedora 26, add 28

Closes #30579

* Update testing docs with new fedora 28 image
2018-05-22 07:22:00 +03:00
Alpar Torok 638383dc44
disable annotation processor for docs (#30610)
* disable annotation processor for docs

Could not find evidence that the log4j annotation processor is used.
The compiler flag enables the Gradle 5.0 behavior

Closes #30476

* Disable annotation processors for all tests

* remove redundant `-proc:none` already handled by required plugins

* Revert unintentional changes
2018-05-21 09:42:58 +03:00
Nhat Nguyen 67d8fc222d
Upgrade to Lucene-7.4.0-snapshot-59f2b7aec2 (#30726)
This snapshot resolves issues related to ShrinkIndexIT.
2018-05-18 18:21:39 -04:00
Ryan Ernst b3f3a4312b
Plugins: Remove meta plugins (#30670)
Meta plugins existed only for a short time, in order to enable breaking
up x-pack into multiple plugins. However, now that x-pack is no longer
installed as a plugin, the need for them has disappeared. This commit
removes the meta plugins infrastructure.
2018-05-18 10:56:08 -07:00
Andy Bristol 890afadb37
[test] packaging: add windows boxes (#30402)
Adds windows server 2012r2 and 2016 vagrant boxes to packaging tests.
They can only be used if IDs for their images are specified, which are
passed to gradle and then to vagrant via env variables. Adds options
to the project property `vagrant.boxes` to choose between linux and
windows boxes.

Bats tests are run only on linux boxes, and portable packaging tests run
on all boxes. Platform tests are only run on linux boxes since they are
not being maintained.

For #26741
2018-05-16 16:42:08 -07:00
Jason Tedor 4a4e3d70d5
Default to one shard (#30539)
This commit changes the default out-of-the-box configuration for the
number of shards from five to one. We think this will help address a
common problem of oversharding. For users with time-based indices that
need a different default, this can be managed with index templates. For
users with non-time-based indices that find they need to re-shard with
the split API in place they no longer need to resort only to
reindexing.

Since this has the impact of changing the default number of shards used
in REST tests, we want to ensure that we still have coverage for issues
that could arise from multiple shards. As such, we randomize (rarely)
the default number of shards in REST tests to two. This is managed via a
global index template. However, some tests check the templates that are
in the cluster state during the test. Since this template is randomly
there, we need a way for tests to skip adding the template used to set
the number of shards to two. For this we add the default_shards feature
skip. To avoid having to write our docs in a complicated way because
sometimes they might be behind one shard, and sometimes they might be
behind two shards we apply the default_shards feature skip to all docs
tests. That is, these tests will always run with the default number of
shards (one).
2018-05-14 12:22:35 -04:00
Alpar Torok 9a5555963b
Add missing dependencies on testClasses (#30527) 2018-05-14 16:06:56 +03:00
Ryan Ernst be8c094e8c
Build: Add mavenPlugin cluster configuration method (#30541)
This commit adds the ability to specify a plugin from maven for a
test cluster to use. Currently, only local projects may be used as
plugins, except when testing bwc, where the coordinates of the project
are used. However, that assumes all projects always keep the same
coordinates, or are even still plugins, which is no longer the case for
x-pack. The full cluster and rolling restart tests are changed to use
this new method when pulling x-pack versions before 6.3.0.
2018-05-11 21:58:18 -07:00
Nhat Nguyen 519768b5d3
Upgrade to Lucene-7.4-snapshot-6705632810 (#30519)
This snapshot is to include LUCENE-8298 which allows DocValues updates
to reset a value. This is needed for the Lucene rollback work.
2018-05-10 12:31:45 -04:00
Nik Everett f9dc86836d
Docs: Test examples that recreate lang analyzers (#29535)
We have a pile of documentation describing how to rebuild the built in
language analyzers and, previously, our documentation testing framework
made sure that the examples successfully built *an* analyzer but they
didn't assert that the analyzer built by the documentation matches the
built in anlayzer. Unsuprisingly, some of the examples aren't quite
right.

This adds a mechanism that tests that the analyzers built by the docs.
The mechanism is fairly simple and brutal but it seems to be working:
build a hundred random unicode sequences and send them through the
`_analyze` API with the rebuilt analyzer and then again through the
built in analyzer. Then make sure both APIs return the same results.
Each of these calls to `_anlayze` takes about 20ms on my laptop which
seems fine.
2018-05-09 09:23:10 -04:00
Jason Tedor 92a0b0a00c
Stop forking groovyc (#30471)
This is a follow-up to our previous change to only fork javac if needed
to respect the Java compiler home (via JAVA_HOME). This commit makes the
same change for groovyc: we only fork groovyc if the JDK for Gradle is
not the JDK specified for the compiler (via JAVA_HOME).
2018-05-09 06:17:43 -04:00
Jason Tedor 3a912dfab2
Stop forking javac (#30462)
We started forking javac to avoid GC overhead when running builds. Yet,
we do not seem to have this problem anymore and not forking leads to a
substantial speed improvement. This commit stops forking javac.
2018-05-08 16:13:02 -04:00
Nik Everett 5b8bdcd111
Build: Switch to building javadoc with html5 (#30440)
We accidentally switched back to html4 in #30279 when we removed the
gradle hack that we were using to convert the projects one by one.
2018-05-08 07:38:28 -04:00
Jim Ferenczi dbd857341f
Upgrade to 7.4.0-snapshot-1ed95c097b (#30357)
Upgrade to lucene-7.4.0-snapshot-1ed95c097b

This version contains:
* An Analyzer for Korean
* An IntervalQuery and IntervalsSource that retrieve minimum intervals of positional queries.
* A new API to retrieve matches (offsets and positions) of a query for a single document.
* Support for soft deletes in the index writer.
* A fixed shingle filter that handles index time synonyms.
* Support for emoji sequence in ICUTokenizer (with an upgrade to icu 61.1)
2018-05-04 11:44:22 +02:00
Colin Goodheart-Smithe 09b918545d
Adds Eclipse config for xpack licence headers (#30299)
Uses a filter on the copy task for the eclipse settings files to
replace the token @@LICENSE_HEADER_TEXT@@ with the correct licence
header from the new buildSrc/src/main/resources/license-headers
directory
2018-05-03 08:48:49 +01:00
Ryan Ernst 916bf9d26d
Convert server javadoc to html5 (#30279)
This commit converts the remaining javadocs in :server using html4 to html5.
This was mostly converting `tt` to `{@code}`.
2018-05-02 08:08:54 -07:00
Nik Everett 5e9e6fed90
HTML5ify Javadoc for xpack core (#30277)
xpack core contains a fork of `Cron` from quartz who's javadoc has a
`<table>` with non-html5 compatible stuff. This html5ifies the table and
switches the `:x-pack:plugin:core` project to building javadoc with
HTML5.
2018-05-01 08:32:58 -04:00
Andy Bristol 65e5868a55
[test] add java packaging test project (#30161)
[test] add java packaging test project

Adds a project for building and running packaging tests written in java
for portability. The vagrant tasks use jars on the packagingTest
configuration, which are built in the same project. No tests are added
yet.

Corresponding changes are not made to :x-pack:qa:vagrant because the
java packaging tests will all be consolidated into one project.

For #26741
2018-04-30 16:35:26 -07:00
Nik Everett 50945051b6
HTML5ify Javadoc for core and test framework (#30234)
`javadoc` will switch from detaulting to html4 to html5 in "a future
release". We should get ahead of it so we're not surprised. Also, HTML5
is the future! Er, the present. Anyway, this follows up from #30220 to
make the Javadoc for two of the four remaining projects HTML5
compatible.
2018-04-30 09:39:50 -04:00
Nik Everett 9c8e015552
Build: Mostly silence warning about html4 javadoc (#30220)
This *mostly* silences `javadoc`'s warning about defaulting to
generating html4 files by enabling generating html5 file for the
projects for which that works. It didn't work in a half dozen projects,
about half of which I've fixed in this PR, entirely by replacing
`<tt>thing</tt>` with `{@code thing}`.

There are a few remaining projects that contain javadoc with invalid
html5. I'll fix those projects in a followup.
2018-04-28 09:50:54 -04:00
Andy Bristol 67c0cf1dbf
[test] include oss tar in packaging tests (#30155)
Add the oss tar distribution to the packaging test plugin. Test the oss
tar distribution in the core packaging tests, and the non-oss tar
distribution in the x-pack packaging tests.
2018-04-26 06:58:41 -07:00
Nhat Nguyen 16490d7dfa
TEST: Update settings should go through cluster state (#29682)
Today we update index settings directly via IndexService instead of the
cluster state in IndexServiceTests. However, those changes will be lost
if there is a cluster state update. In general, we should update index
settings via client and limit the direct usage in only special tests.

This commit replaces direct usages by the updateSettings api of client.

Closes #24491
2018-04-26 09:28:14 -04:00
Jason Tedor b952a9954f
Set JAVA_HOME before forking setup commands (#29647)
Today when forking setup commands we do not set JAVA_HOME. This means
that we might not use a version of Java compatible with the version of
Java the command is expecting to run on (for example, 5.6 nodes would
expect JDK 8, and this is true even for their setup commands). This
commit sets JAVA_HOME when configuring setup command tasks.
2018-04-25 22:43:00 -04:00
Ryan Ernst 0d8aa7527e Reorganize license files
This commit moves the apache and elastic license files into a new
root level `licenses` directory and rewrites the top level LICENSE.txt
to clarify the repository has a mix of apache and elastic licensed code.
2018-04-20 15:33:59 -07:00
Jason Tedor 4fe1d64070 Add POM and license generation for X-Pack artifacts
With the switch to X-Pack as a module, we lost production of POMs for
the JARs that we publish, and did not have a license/notice file in the
zip archives nor the exploded module. This commit ensures that we
generate these POMs, and license/notice files.
2018-04-20 15:33:58 -07:00
Ryan Ernst fab5e21e7d Build: Split distributions into oss and default
This commit makes x-pack a module and adds it to the default
distrubtion. It also creates distributions for zip, tar, deb and rpm
which contain only oss code.
2018-04-20 15:33:57 -07:00
Ryan Ernst 4f282e9e32
Build: Move java home checks to pre-execution phase (#29548)
This commit moves the checks on JAVAX_HOME (where X is the java version
number) existing to the end of gradle's configuration phase, and based
on whether the tasks needing the java home are configured to execute.

relates #29519
2018-04-19 09:51:52 -07:00
olcbean b3e3b80f1b REST high-level client: add support for Indices Update Settings API [take 2] (#29327)
Relates to #27205
2018-04-16 21:39:11 +02:00
Jason Tedor 9125684d86 Minor cleanup in NodeInfo.groovy
This commit is a minor cleanup of a code block in NodeInfo.groovy. We
remove an unused variable, make the formatting of the code consistent,
and cast a property that is typed as an Object to a String to avoid an
annoying IDE warning.
2018-04-14 20:48:48 -04:00
Jason Tedor b883e1217f
Lazy configure build tasks that require older JDKs (#29519)
Some build tasks require older JDKs. For example, the BWC build tasks
for older versions of Elasticsearch require older JDKs. It is onerous to
require these be configured when merely compiling Elasticsearch, the
requirement that they be strictly set to appropriate values should only
be enforced if these tasks are going to be executed. To address this, we
lazy configure these tasks.
2018-04-14 15:44:43 -04:00
Jason Tedor 85ac541ab3
Make NodeInfo#nodeVersion strongly-typed as Version (#29515)
Today we have a nodeVersion property on the NodeInfo class that we use
to carry around information about a standalone node that we will start
during tests. This property is a String which we usually end up parsing
to a Version anyway to do various checks on it. This can end up
happening a lot during configuration so it would be more efficient and
safer to have this already be strongly-typed as a Version and parsed
from a String only once for each instance of NodeInfo. Therefore, this
commit makes NodeInfo#nodeVersion strongly-typed as a Version.
2018-04-13 16:57:59 -04:00
Jason Tedor 82a753dcc7
Enable license header exclusions (#29379)
There are some scenarios where the license on a source file is one that
is compatible with our projects yet we do not want to add the license to
the list of approved license headers (to keep the number of files with
that compatible license contained). This commit adds the ability to
exclude a file from the license check.
2018-04-13 15:59:33 -04:00
Jason Tedor 27fafa24f5
Use proper Java version for BWC builds (#29493)
Today we have JAVA_HOME for the compiler Java home and RUNTIME_JAVA_HOME
for the test Java home. However, when we compile BWC nodes and run them,
neither of these Java homes might be the version that was suitable for
that BWC node (e.g., 5.6 requires JDK 8 to compile and to run). This
commit adds support for the environment variables JAVA\d+_HOME and uses
the appropriate Java home based on the version of the node being
started. We even do this for reindex-from-old which requires JDK 7 for
these very old nodes. Note that these environment variables are not
required if not running BWC tests, and they are strictly required if
running BWC tests.
2018-04-13 12:41:18 -04:00
Jay Modi e0ec8571ea
Build: introduce keystoreFile for cluster config (#29491)
This commit introduces built in support for adding files to the
keystore when configuring the integration test cluster for a project.

In order to use this support, simply add `keystoreFile` followed by the
secure setting name and the path to the source file inside the
integTestCluster closure for a project. The built in support will
handle the creation of the keystore and the addition of the file to the
keystore.
2018-04-12 07:28:34 -06:00
Christoph Büscher e2dbc77cd0 Remove redundant checkstyle LineLength suppressions 2018-04-06 19:49:33 +02:00
David Roberts d649449d37
[TEST] Put options that enable assertions earlier on command line (#29395)
This change moves the -ea and -esa options that enable assertions for
test nodes before the cluster-specific JVM arguments on the Java command
line.  This opens up the possibility for the cluster-specific JVM
arguments to disable assertions for one particular package or class,
which can be useful in BWC testing where incorrect assertions cannot be
removed from released versions of the product.
2018-04-06 11:34:08 +01:00
Alan Woodward dccd43af47
Upgrade to lucene 7.3.0 (#29387) 2018-04-05 10:34:44 +01:00
Jason Tedor 5cdd831a31
Remove silent batch mode from install plugin (#29359)
Today we have a silent batch mode in the install plugin command when
standard input is closed or there is no tty. It appears that
historically this was useful when running tests where we want to accept
plugin permissions without having to acknowledge them. Now that we have
an explicit batch mode flag, this use-case is removed. The motivation
for removing this now is that there is another place where silent batch
mode arises and that is when a user attempts to install a plugin inside
a Docker container without keeping standard input open and attaching a
tty. In this case, the install plugin command will treat the situation
as a silent batch mode and therefore the user will never have the chance
to acknowledge the additional permissions required by a plugin. This
commit removes this silent batch mode in favor of using the --batch flag
when running tests and requiring the user to take explicit action to
acknowledge the additional permissions (either by leaving standard input
open and attaching a tty, or by passing the --batch flags themselves).

Note that with this change the user will now see a null pointer
exception when they try to install a plugin in a Docker container
without keeping standard input open and attaching a tty. This will be
addressed in an immediate follow-up, but because the implications of
that change are larger, they should be handled separately from this one.
2018-04-03 21:23:01 -04:00
Uwe Schindler 7c6d5cbf1f Build: Fix Java9 MR build (#29312)
Correctly setup classpath/dependencies and fix checkstyle task that was partly broken because delayed setup of Java9 sourcesets. This also cleans packaging of META-INF. It also prepares forbiddenapis 2.6 upgrade

relates #29292
2018-04-03 10:22:12 -07:00
Adrien Grand 3bdfc8f3fb
Upgrade to lucene-7.3.0-snapshot-98a6b3d. (#29298)
Most notable changes include:
 - this release doesn't have the 7.2.1 version constant so I had to create one
 - spatial4j and jts were upgraded
2018-04-03 09:27:14 +02:00
Lee Hinman 6b2167f462
Begin moving XContent to a separate lib/artifact (#29300)
* Begin moving XContent to a separate lib/artifact

This commit moves a large portion of the XContent code from the `server` project
to the `libs/xcontent` project. For the pieces that have been moved, some
helpers have been duplicated to allow them to be decoupled from ES helper
classes. In addition, `Booleans` and `CheckedFunction` have been moved to the
`elasticsearch-core`  project.

This decoupling is a move so that we can eventually make things like the
high-level REST client not rely on the entire ES jar, only the parts it needs.

There are some pieces that are still not decoupled, in particular some of the
XContent tests still remain in the server project, this is because they test a
large portion of the pluggable xcontent pieces through
`XContentElasticsearchException`. They may be decoupled in future work.
Additionally, there may be more piecese that we want to move to the xcontent lib
in the future that are not part of this PR, this is a starting point.

Relates to #28504
2018-04-02 15:58:31 -06:00
Mayya Sharipova e70cd35bda
Revert "REST high-level client: add support for Indices Update Settings API (#28892)" (#29323)
This reverts commit b67b5b1bbd.
2018-03-30 16:26:46 -07:00