Commit Graph

481 Commits

Author SHA1 Message Date
Ryan Ernst 0e8958d1e8 Fix jarhell marker to be written on info logging, and document how the
marker file works.
2015-11-19 10:21:25 -08:00
Ryan Ernst fce0052fd9 Build: Add jar hell check before tests run
Because jar hell checks run during static initialization of tests, a
failure will result in all tests failing. However, only the first test
within each jvm shows the jarhell failure, and later tests get a class
not found exception for the class that failed to load when static init
failed.

This change adds a task to run as part of precommit, which checks the
test runtime classpath for jarhell.

closes #14721
2015-11-19 02:07:15 -08:00
Ryan Ernst d44dbd4757 Build: Fix integTest output if the elasticsearch script fails
If there is a failure in the elasticsearch start script, we currently
completely lose the failure. This is due to how spawning works with ant.
This change avoids the issue by introducing an intermediate script,
built dynamically before running ant exec, which runs elasticsearch and
redirects the output to a log file. This essentially causes us to run
elasticsearch in the foreground and capture the output, but at the same
time keep a running script which ant can pump streams from (which will
always be empty).
2015-11-18 23:30:33 -08:00
Ryan Ernst c273327246 Build: Improve test output on errors and when debugging
There were a number of subtle issues with the existing logging that
wraps events from Junit4 and ant. This change:
* Tweaks at what level certain events are logged
* Fixes -Dtests.output=always to force everything to be logged
* Makes -Dtests.class imply -Dtests.output=always
* Captures ant logging from junit4, so that direct jvm output will be
  logged on failure when not using gradle info logging
2015-11-18 16:36:07 -08:00
Nik Everett 8ffb63a177 Merge pull request #14718 from nik9000/gradle_deb
Get packaging working in gradle
2015-11-18 14:17:11 -05:00
Nik Everett 9b0a47d8e3 Fix rpm and deb distributions
and test them with vagrant
2015-11-18 14:16:42 -05:00
Nik Everett 19919a53a5 [build] Log launch command before starting ES 2015-11-18 12:25:27 -05:00
Nik Everett 1f67ef2526 [build] Log more information when es doesn't start 2015-11-18 10:58:52 -05:00
Nik Everett 22873dc952 [build] Output a message when can't find log file
When the build tries to start an elasticsearch instance but the start fails
if it fails to find the log file then we log a line about how we can't find
the file.
2015-11-18 10:43:01 -05:00
Ryan Ernst 9f59ba5abf Tweak debug startup message for run command, per PR comment 2015-11-17 17:18:44 -08:00
Ryan Ernst 0e8d605e2d Build: Add --debug-jvm option to run and integTest tasks
Sometimes when running elasticsearch, it is useful to attach a remote
debugger. This change adds a --debug-jvm option (the same name gradle
uses for its tests debug option), which adds java agent config for a
remote debugger. The configuration is set to hava java suspend until the
remove debugger is attached.

closes #14772
2015-11-17 16:29:01 -08:00
Nik Everett e0646eeb89 Merge pull request #14754 from nik9000/build
Make Build work without git
2015-11-17 13:30:23 -05:00
Nik Everett 0119caa4a6 Make Build work without git
If you build elasticsearch without a git repository it was creating a null
shortHash which was causing Elasticsearch not to be able to form transport
connections.

Closes #14748
2015-11-17 13:25:09 -05:00
Nik Everett 74aa18088c [build] Update progress logger after each test
This makes the rest tests **tons** more responsive.

Also stop test progress output from jumping by using formating. The progess
now looks like:
Suites [004/549], Tests [0019|0|0], in 1.58s J2 completed UpdateNumberOfReplicasTests

The changes included are:
1. The suites, total tests, and JVM id are now padded based on their maximum
size. The maximum number of tests is just a guess because that data isn't
easily available when the suite starts. JVM id rarely matters because only
the most crazy individuals use more than 10 JVMs.

2. The suite information is reordered. Now its runtime, jvm id, suite name,
and, optionally, method name. This reordering is useful because the thing
that varies in length, the suite and method name, are on the right hand
side. This means that nothing jumps around during the test run.
2015-11-17 13:16:17 -05:00
Ryan Ernst e837140385 Merge pull request #14791 from rjernst/run_away
Build: Make run command work within plugins
2015-11-17 09:09:26 -08:00
Ryan Ernst 50b48e7d8c Build: Make run command work within plugins
We recently got a run command with gradle, but it is sometimes useful to
run ES with a specific plugin. This is a start, by making each esplugin
have a run command which installs the plugin and runs elasticsearch in
the foreground.
2015-11-16 17:29:01 -08:00
Ryan Ernst 176f9e9d89 Build: Use a marker file to indicate forbidden patterns has run
This makes forbidden patterns a little smarter, so it does not need to
run on every build. It works because the marker file timestamp will be
compared against the source files (which are the inputs to forbidden
patterns).

closes #14788
2015-11-16 16:48:56 -08:00
Michael McCandless 01f12fc8d5 Merge pull request #14784 from mikemccand/upgrade_lucene_1714615
Upgrade Lucene to 5.4.0-snapshot-1714615
2015-11-16 16:07:46 -05:00
Michael McCandless a0bf253d16 upgrade lucene 5.4 snapshot 2015-11-16 14:38:05 -05:00
Ryan Ernst d872e2c527 Build: Use JDK at JAVA_HOME for compiling/testing, and improve build info output
We currently enforce JAVA_HOME is set, but use gradle's java version to
compile, and JAVA_HOME version to test. This change makes the compile
and test both use JAVA_HOME, and clarifies which java version is used by
gradle and that used for the compile/test.
2015-11-16 09:15:31 -08:00
Ryan Ernst 759fc94947 Build: Override Build-Date in jar manifest to be iso 8601 2015-11-14 22:38:10 -08:00
Ryan Ernst 5eb5480ea6 Improve wording of transitive deps hack explanation 2015-11-13 15:05:40 -08:00
Ryan Ernst 24d306c2f4 Add explanation of the configuration trick used to find transitive deps
for exclusion
2015-11-13 14:59:45 -08:00
Ryan Ernst be147719ff Remove artifacts left from testing 2015-11-13 14:20:14 -08:00
Ryan Ernst 1d38f0ac8e Remove forgotten println 2015-11-13 14:18:14 -08:00
Ryan Ernst 837c593ec2 Build: Effectively remove transitive deps from generated maven poms
With gradle, deploying to maven means first generating poms. These are
filled in based on dependencies of the project. Recently, we started
disallowing transitive dependencies. However, this configuration does
not translate to maven poms because maven has no concept of excluding
all transitive dependencies.

This change adds exclusions for each of the transitive deps of each
dependency being added to the maven pom. It does so by creating dummy
configurations for each direct dependency (which does not have
transitive deps excluded), so that we can iterate the transitive deps
when building the pom.

Note, this should be simpler (just modifying maven's pom model), but
gradle tries to hide that from their api, causing us to need to
manipulate the xml directly.
https://discuss.gradle.org/t/modifying-maven-pom-generation-to-add-excludes/12744
2015-11-13 14:10:17 -08:00
Ryan Ernst 9373382155 Build: Use ant exec for starting elasticsearch in integ tests
Currently elasticsearch in integ tests is started using an ant task on
windows, or gradle exec on everything else. However, gradle exec has
some flaws, one being Ctrl-C does not run finalizedBy tasks, which means
interrupting integ tests will leak a jvm. This change makes all systems
use ant exec. One caveat is, if there is any output by the jvm, we lose
it in ant bit heaven. But this is no different than what we had with
gradle. In the future, we should look at using a separate thread to
pump streams from the elasticsearch process.

closes #14701
2015-11-12 22:43:38 -08:00
Robert Muir 776bb288b5 fix gradle check under jigsaw
closes #14726

Squashed commit of the following:

commit 5b591e98570e3fa481b2816a44063b98bff36ddf
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Nov 13 00:54:08 2015 -0500

    add assumption for self-signing in PluginManagerTests

commit ed11e5371b6f71591dc41c6f60d033502cfcf029
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Nov 13 00:20:59 2015 -0500

    show error output from integ test startup

commit d8b187a10e95d89a0e775333dcbe1aaa903fb376
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Nov 12 22:14:11 2015 -0500

    fix gradle check under jigsaw
2015-11-13 00:57:12 -05:00
Ryan Ernst f327beac49 Build: Enforce JAVA_HOME is set
If we use JAVA_HOME consistently for tests, we can run tests with a
different version of java than gradle runs with. For example, this
enables running tests with jigsaw, but building with java 8. The only
caveat is intellij does not set JAVA_HOME. This change enforces
JAVA_HOME is set, but ignores for intellij.
2015-11-12 10:22:56 -08:00
Ryan Ernst 505a4d9e09 Build: Simplify where min java version is specified
This moves the min java version used by elasticsearch to one place, a
constant in BuildPlugin. For me on java 9, this fixed my jar to have the
correct target/source versions.

closes #14702
2015-11-11 23:09:57 -08:00
Ryan Ernst 83f3a981b7 Merge pull request #14668 from rjernst/transitive_goes_poof
Remove transitive dependencies
2015-11-10 19:33:47 -08:00
Jason Tedor bfe27407ae More explicit handling of jps executable on Windows 2015-11-10 18:03:16 -05:00
Ryan Ernst 4b5f87cb7d Build: Remove transitive dependencies
Transitive dependencies can be confusing and hard to deal with when
conflicts arise between them. This change removes transitive
dependencies from elasticsearch, and forces any dependency conflicts to
be resolved manually, instead of automatically by gradle.

closes #14627
2015-11-10 15:01:41 -08:00
Jason Tedor ebd8bbcffe Get correct jps path on Windows
This commit fixes an issue with the jps executable sometimes being
available under jps.exe but not jps.
2015-11-10 14:56:45 -05:00
Ryan Ernst d64d1c99d4 Build: Upgrade minimum gradle version to 2.8
In gradle 2.7 (or groovy 2.3.10, not sure which), there appears to be a
bug on linux where using a fully qualified class (without an import
statement) does not work. This change forces gradle 2.8 or above. It
also moves the logic around a little for the version check so the build
info is printed before checks against that info.
2015-11-10 11:17:30 -08:00
Ryan Ernst 4f5de0f453 Add back eclipse configuration to buildSrc 2015-11-10 07:30:11 -08:00
Ryan Ernst 7a6155e12f Build: Consolidate dependencies specified in multiple places
Some dependencies must be specified in a couple places in the build.
e.g. randomized runner is specified both in buildSrc (for the gradle
wrapper plugin), as well as in the test-framework.

This change creates buildSrc/versions.properties which acts similar to
the set of shared version properties we used to have in the maven parent
pom.
2015-11-09 23:59:56 -08:00
Ryan Ernst fc7bd7bfba Build: Fix automatic installation of plugin for integ tests
The esplugin gradle plugin automatically adds the pluging being built to
the integTest cluster. However, there were two issues with this. First
was a bug in the name, which should have been the configured
esplugin.name instead of the project name. Second, the files
configuration was overcomplicated (trying to use the groovy spreader
operator after delaying calls to singleFile). Instead, we can just pass
the file collections (which will just be a single file at execution
time).
2015-11-09 17:27:34 -08:00
Ryan Ernst 7034feeceb Merge pull request #14451 from andrejserafim/run
Add gradle run command to replace run.bat and run.sh
2015-11-09 11:15:53 -08:00
Jason Tedor 5c3a0d63ea Get jps path relative to the JVM running Gradle
This commit addresses an issue in getting a path to the jps bin. The
solution is to get the path to the JDK relative to the JVM running
Gradle.

Closes #14614
2015-11-09 13:24:59 -05:00
Jason Tedor 013347be9b Revert "Merge pull request #14623 from nik9000/no_home_home"
This reverts commit cf8ffd32ac, reversing
changes made to 40650ad2d5.
2015-11-09 10:37:16 -05:00
javanna 10ddd691a3 Internal: move to lucene BoostQuery
Latest version of lucene deprecated Query#setBoost and Query#getBoost which made queries effectively immutable. Those methods need to be replaced with `BoostQuery` that wraps any query that needs boosting.

This commit replaces usages of setBoost with BoostQuery and adds it to forbidden-apis for prod code.

Usages of `getBoost` are only partially removed, as some will have to stay for backwards compatibility.

Closes #14264
2015-11-09 15:01:35 +01:00
Nik Everett a7b78bf67a [build] Use always set java.home
`System.getenv('JAVA_HOME')` relies on JAVA_HOME being setup by the user.
`System.getProperty('java.home')` is set by Java all the time.

Closes #14614
2015-11-09 08:12:08 -05:00
andrejserafim 2bd353d72d replacing run.bat and run.sh with gradle run
run.sh and run.bat were calling out to the old maven build system.
This is no longer in place, so we've created new gradle tasks to
start an elasticsearch node from the current codebase.

fixed #14423
2015-11-08 17:07:19 +00:00
Ryan Ernst 8d78f970b0 Build: Enforce name is set for all plugins
The plugin name currently defaults to the gradle project name. But the
gradle project name for standalone repo (like an external plugin would
be) defaults to the directory name of the repo. This is trappy, since it
depends on how the repo was checked out.

This change enforces the plugin name is always set.

closes #14603
2015-11-07 16:57:11 -08:00
Ryan Ernst d0f5950c80 Merge pull request #14598 from rjernst/integ_test_parity
Improve integ test to match ant behavior
2015-11-07 13:09:04 -08:00
Ryan Ernst 22bcf197b7 Add some comments to particularly nasty groovy code 2015-11-07 13:06:14 -08:00
Robert Muir 866a514785 add groovy build code (buildSrc/) in 'gradle eclipse'
This makes it a groovy project that works in eclipse.
You will have to install a plugin for groovy language support
(I used a snapshot build from https://github.com/groovy/groovy-eclipse/wiki)
2015-11-07 15:14:00 -05:00
Ryan Ernst 9ee315a9c8 Build: Improve integ test to match ant behavior
Many other improvements:
* Use spaces in ES path
* Use space in path for plugin file installation
* Use a different cwd than ES home
* Use jps to ensure process being stopped is actually elasticsearch
* Stop ES if pid file already exists
* Delete pid file when successfully killed

Also, refactored the cluster formation code to be a little more organized.

closes #14464
2015-11-07 00:45:19 -08:00
Jason Tedor 97644e3046 Merge pull request #14588 from jasontedor/enforce-java-8
Enforce Java 8 in Gradle builds
2015-11-06 12:33:28 -05:00
Jason Tedor b0c7065000 Cleaner version check 2015-11-06 12:28:31 -05:00
Jason Tedor faf5f55f41 Enforce Java 8 in Gradle builds
This commit forces builds with Gradle to require at least Java 8.

Closes #14499
2015-11-06 11:39:14 -05:00
Ryan Ernst c71cc0f21d Revert "Build: Fail integ test if elasticsearch fails to start on windows"
This reverts commit f974d5e470.

Apprently spawn does not support failonerror in ant.
2015-11-06 08:22:17 -08:00
Nik Everett 047e3b9b69 Merge pull request #14531 from nik9000/gradle_vagrant
Gradle-ify vagrant tests
2015-11-05 14:29:06 -05:00
Nik Everett 1dd00dddd5 [test] Gradle-ify vagrant tests
This gets the tar and tar_plugins tests working in gradle. It does so by
adding a subproject, qa/vagrant, which adds the following tasks:

Verification
------------
checkPackages - Check the packages against a representative sample of the
                linux distributions we have in our Vagrantfile
checkPackagesAllDistros - Check the packages against all the linux
                          distributions we have in our Vagrantfile

Package Verification
--------------------
checkCentos6 - Run packaging tests against centos-6
checkCentos7 - Run packaging tests against centos-7
checkDebian8 - Run packaging tests against debian-8
checkFedora22 - Run packaging tests against fedora-22
checkOel7 - Run packaging tests against oel-7
checkOpensuse13 - Run packaging tests against opensuse-13
checkSles12 - Run packaging tests against sles-12
checkUbuntu1204 - Run packaging tests against ubuntu-1204
checkUbuntu1404 - Run packaging tests against ubuntu-1404
checkUbuntu1504 - Run packaging tests against ubuntu-1504

Vagrant
-------
smokeTestCentos6 - Smoke test the centos-6 VM
smokeTestCentos7 - Smoke test the centos-7 VM
smokeTestDebian8 - Smoke test the debian-8 VM
smokeTestFedora22 - Smoke test the fedora-22 VM
smokeTestOel7 - Smoke test the oel-7 VM
smokeTestOpensuse13 - Smoke test the opensuse-13 VM
smokeTestSles12 - Smoke test the sles-12 VM
smokeTestUbuntu1204 - Smoke test the ubuntu-1204 VM
smokeTestUbuntu1404 - Smoke test the ubuntu-1404 VM
smokeTestUbuntu1504 - Smoke test the ubuntu-1504 VM
vagrantHaltCentos6 - Shutdown the vagrant VM running centos-6
vagrantHaltCentos7 - Shutdown the vagrant VM running centos-7
vagrantHaltDebian8 - Shutdown the vagrant VM running debian-8
vagrantHaltFedora22 - Shutdown the vagrant VM running fedora-22
vagrantHaltOel7 - Shutdown the vagrant VM running oel-7
vagrantHaltOpensuse13 - Shutdown the vagrant VM running opensuse-13
vagrantHaltSles12 - Shutdown the vagrant VM running sles-12
vagrantHaltUbuntu1204 - Shutdown the vagrant VM running ubuntu-1204
vagrantHaltUbuntu1404 - Shutdown the vagrant VM running ubuntu-1404
vagrantHaltUbuntu1504 - Shutdown the vagrant VM running ubuntu-1504
vagrantSmokeTest - Smoke test some representative distros from the Vagrantfile
vagrantSmokeTestAllDistros - Smoke test all distros from the Vagrantfile
vagrantUpCentos6 - Startup a vagrant VM running centos-6
vagrantUpCentos7 - Startup a vagrant VM running centos-7
vagrantUpDebian8 - Startup a vagrant VM running debian-8
vagrantUpFedora22 - Startup a vagrant VM running fedora-22
vagrantUpOel7 - Startup a vagrant VM running oel-7
vagrantUpOpensuse13 - Startup a vagrant VM running opensuse-13
vagrantUpSles12 - Startup a vagrant VM running sles-12
vagrantUpUbuntu1204 - Startup a vagrant VM running ubuntu-1204
vagrantUpUbuntu1404 - Startup a vagrant VM running ubuntu-1404
vagrantUpUbuntu1504 - Startup a vagrant VM running ubuntu-1504

It does not make the "check" task depend on "checkPackages" so running the
vagrant tests is still optional. They are slow and depend on vagrant and
virtualbox.

The Package Verification tasks are useful for testing individual distros.

The Vagrant tasks are listed in `gradle tasks` primarily for discoverability.
2015-11-05 14:28:30 -05:00
Ryan Ernst f974d5e470 Build: Fail integ test if elasticsearch fails to start on windows 2015-11-05 00:07:59 -08:00
Ryan Ernst 2723bac296 Build: Add back support for tests.jvm.argline
closes #14541
2015-11-04 23:59:01 -08:00
Robert Muir 1a0a910d1e Merge pull request #14526 from rjernst/standalone_test
Add qa/evil-tests
2015-11-04 13:49:55 -05:00
Ryan Ernst a534ace896 Fix standalone test to work with check, and comment out mavenLocal 2015-11-04 10:28:31 -08:00
Daniel Mitterdorfer ae9fcd4f2b Merge remote-tracking branch 'danielmitterdorfer/feature/12596' 2015-11-04 09:31:12 +01:00
Daniel Mitterdorfer fea88bbc02 Add System#exit(), Runtime#exit() and Runtime#halt() to forbidden APIs
Closes #12596
2015-11-04 09:29:55 +01:00
Ryan Ernst fc1c66083b Fix integ test exec on windows 2015-11-03 23:02:29 -08:00
Ryan Ernst 95f885e11f Build: Change tar extension back to tar.gz instead of tgz
Gradle defaults to tgz extension when tar is compressed. This changes
the tar distribution back to tar.gz. Note that this also means the maven
packaging type is now tar.gz.
2015-11-03 11:24:27 -08:00
Ryan Ernst bd158e5e88 Add environment vars for JAVA_HOME and ES_GC_OPTS back to integ tests 2015-11-03 10:13:02 -08:00
Ryan Ernst 9246ca98fe rest test refactor 2015-11-03 10:12:16 -08:00
Ryan Ernst 3e4f87d4d6 Fix tar issues and build distro dependency for integ tests 2015-11-02 22:19:29 -08:00
Ryan Ernst f5556224f5 Build: Add back integ tests to distributions
rpm and deb are still skipped, but this configures rest tests to run for
zip and tgz distributions

closes #14361
2015-11-02 17:51:08 -08:00
Ryan Ernst 5bfecd4f76 Build: Simplify plugin properties generation
The gradle task to generate plugin properties files now is a simple copy
with expansion (ie maven filtering). It also no longer depends on
compiling.

closes #14450
2015-11-02 12:43:05 -08:00
Ryan Ernst 0f429f32de Merge pull request #14447 from rjernst/ide_plugin_props
Make idea/eclipse project generation build generated resources for plugins
2015-11-02 11:38:10 -08:00
Ryan Ernst b7f8e5c1df Build: Make idea/eclipse project generation build generated resources for plugins
This adds a generated-resources dir that the plugin properties are
generated into. This must be outside of the build dir, since intellij
has build as "excluded".

closes #14392
2015-11-02 11:35:02 -08:00
Robert Muir 73a136e9fc Use ${exception_var} for the case of nested generated try/catch 2015-11-02 11:58:34 -05:00
Robert Muir 1e00b85411 Fix auto-generated eclipse try/catch to be less trappy
When generating a try-catch block, the eclipse default is something like this:

```
try {
  something();
} catch (Exception e) {
  // TODO: auto-generated stub
  e.printStackTrace();
}

which is terrible, so the ES eclipse changes this to rethrow a RuntimeException instead.

```
try {
  something();
} catch (Exception e) {
  throw new RuntimeException();
}
```

Unfortunately, this loses the original exception entirely, instead it should be:

```
try {
  something();
} catch (Exception e) {
  throw new RuntimeException(e);
}
```
2015-11-02 11:45:55 -05:00
Robert Muir 951bc5a4a7 disable null pointer analysis so the IDE is not crashing. 2015-11-02 10:55:31 -05:00
Robert Muir 4587a94fcf generate complete eclipse configuration from 'gradle eclipse' 2015-11-02 10:44:51 -05:00
Ryan Ernst 0dca49fae4 fix eclipse config for qa projects 2015-10-30 21:32:16 -07:00
Ryan Ernst 197ed57ea4 Build: Move RR gradle plugin files to match external repo paths
The RR gradle plugin is at
https://github.com/randomizedtesting/gradle-randomized-testing-plugin.
However, we currently have a copy of this, since the plugin is still in
heavy development. This change moves the files around so they can be
copied directly from the elasticsearch fork to that repo, for ease of
syncing.
2015-10-30 13:22:59 -07:00
Ryan Ernst c1e3c4a78e Build: Add back -Dtests.output support
This change fixes the test logging for randomized testing to support
tests.output like maven/ant do.
2015-10-30 13:05:46 -07:00
Ryan Ernst 0a593109d3 Build: Add gradle version check and some build info that is always output 2015-10-29 17:12:45 -07:00
Ryan Ernst 9146b59fc1 Build: Move the real forbidden api signature files to their new location
within buildSrc

The gradle branch used copies of the forbidden api signature files. This
moves the files to their correct location.

closes #14363
2015-10-29 15:40:47 -07:00
Ryan Ernst 561436eba3 Build: Add back manifest info to jars
closes #14365
2015-10-29 13:59:52 -07:00
Ryan Ernst c86100f636 Switch build system to Gradle
See #13930
2015-10-29 11:40:19 -07:00