Commit Graph

690 Commits

Author SHA1 Message Date
Ali Beyad a7f6488216 Merge remote-tracking branch 'upstream/master' into
feature-suggest-refactoring
2016-02-26 17:21:12 -05:00
Nik Everett ba5be0332d Remove optional logger wrappers
Removes all our logger wrappers except the wrapper for log4j1.2. If you
depend on Elasticsearch's jar in your application you'll need to declare
log4j 1.2 and/or some bridge to your favorite logger.

We did this to simplify our builds and code. No more commons-logging like
log implementation sniffing. No more optional dependency hacks in gradle.

We might one day want to use j.u.l instead of log4j. If we do want that
we can recover its wrapper by studying this commit. We didn't go directly
to j.u.l in this commit because that is a bigger change. Our logging
configuration is based on log4j1.2 and people are used to it. So it'd
be a much more fraught breaking change to do that conversion.
2016-02-26 16:41:07 -05:00
Lee Hinman 5d359f737f Merge branch 'pr/16801' 2016-02-26 13:22:14 -07:00
James Bertouch a249225993 Update to Jackson 2.7.1
https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.7.1
2016-02-25 16:15:03 -05:00
Ryan Ernst 80fc0496bb Merge pull request #16782 from rjernst/fix/extra_config_files
Build: Fix extra config files to work with more than one file
2016-02-25 11:28:39 -08:00
Nik Everett 7c4420bf3d Merge pull request #16807 from nik9000/not_mvn
Remove mention of mvn from java source files
2016-02-25 11:05:46 -05:00
Nik Everett 284f41f7f5 Rename tests.maven to tests.gradle 2016-02-25 10:26:12 -05:00
jaymode c9c4cac0ac Testing: workaround HttpClient issue with IPv6 hostname verification
This commit works around an issue with hostname verification in HttpClient when using IPv6
addresses in URLs. When an IPv6 address is used in a URL it is typically wrapped with square
brackets. The hostname verifier for HttpClient does not recognize these as valid IPv6 addresses
and instead treats them as a DNS name. We wrap the strict hostname verifier for this version
of HttpClient and strip brackets if we need to.

The corresponding issue in HttpClient is https://issues.apache.org/jira/browse/HTTPCLIENT-1698
but the fix has not been released yet in a stable version.
2016-02-25 08:04:41 -05:00
James Bertouch cb35455745 Upgrade to Jackson 2.7.0
Closes #16294
2016-02-24 18:06:38 -05:00
Ryan Ernst 3772350031 Build: Fix extra config files to work with more than one file
The extra config copy task for integ tests was previously overwriting
the source and destination for each file to be copied, leaving only the
last. This fixes it to create sub copy specs (which is what was
originally intended) for each file.
2016-02-23 07:54:40 -08:00
Christoph Büscher d888b5c267 Merge branch 'master' into feature-suggest-refactoring 2016-02-22 11:22:28 -08:00
Jack Conradson 7986770e5f Moved Painless from a plugin to a module.
Closes #16755
2016-02-21 16:50:54 -08:00
Mike McCandless 2e3160824c Upgrade to Lucene 5.5.0 official release
Closes #16742
2016-02-21 05:26:22 -05:00
Simon Willnauer c8e462e5dc Apply feedback from @rjernst 2016-02-20 17:12:48 -08:00
Simon Willnauer 4f7dd1284f Remove SNAPSHOT notion from code
Today we have the notion of a snapshot inside Version.java which makes
releasing complicated since to do a release Version.java must be changed.
This commit removes all notions of snapshot from the code and allows to
switch between snapshot and release build by specifying a system property on
the build. For instance running:

```
gradle run -Dbuild.snapshot=false
```

will build and package a release build while the default always
builds snapshots. Calls to the main rest action will still get the snapshot
information rendered out with the response.
2016-02-20 16:06:26 -08:00
Mike McCandless 5fffede2b0 Upgrade to Lucene 5.5.0 official release 2016-02-20 17:34:16 -05:00
Ali Beyad beea413820 Merge remote-tracking branch 'upstream/master' into feature-suggest-refactoring 2016-02-12 11:54:44 -05:00
Simon Willnauer b5aee2075f Merge pull request #16610 from s1monw/test_indices_request_cache
Refactor IndicesRequestCache to make it testable.
2016-02-12 09:27:20 +01:00
Boaz Leskes 8bc2332d9a Decouple recovery source/target logic and transport piping
The current logic for doing recovery from a source to a target shourd is tightly coupled with the underlying network pipes. This changes decouple the two, making it easier to add unit tests for shard recovery that doesn't involve the node and network environment.

On top that, RecoveryTarget is renamed to RecoveryTargetService leaving space to renaming RecoveryStatus to RecoveryTarget (and thus avoid the confusion we have today with RecoveryState).

Correspondingly RecoverySource is renamed to RecoverySourceService.

Closes #16605
2016-02-11 21:36:48 +01:00
Nicholas Knize 52ee4c7027 upgrade to lucene 5.5.0-snapshot-850c6c2 2016-02-11 14:28:50 -06:00
Simon Willnauer d538dd64c2 Refactor IndicesRequestCache to make it testable.
This commit moves IndicesRequestCache into o.e.indics and makes all API in this
class package private. All references to SearchReqeust, SearchContext etc. have been factored
out and relevant glue code has been added to IndicesService. The IndicesRequestCache is not a
simple class without any hard dependencies on ThreadPool nor SearchService or IndexShard. This now
allows to add unittests.
This commit also removes two settings `indices.requests.cache.clean_interval` and `indices.fielddata.cache.clean_interval`
in favor of `indices.cache.clean_interval` which cleans both caches.
2016-02-11 16:51:44 +01:00
Christoph Büscher 421ed1228b Merge branch 'master' into feature-suggest-refactoring
Conflicts:
	core/src/main/java/org/elasticsearch/search/suggest/phrase/PhraseSuggestionBuilder.java
2016-02-10 11:37:51 +01:00
Jim Ferenczi b146f3ecb3 Pack all the plugin files into a single folder named elasticsearch at the root of the plugin zip. 2016-02-10 10:13:05 +01:00
Nik Everett f76366909a Fail build if we define equals but not hashCode
That is like some kind of cardinal sin or something, right?

We had two violations though they weren't super likely to be keys in a hashmap
any time soon.
2016-02-09 21:29:08 -05:00
Jason Tedor 75da71ff0b Forbid use of java.security.MessageDigest#clone()
This commit forbids the usage of java.security.MessageDigest#clone() in
favor of getting a thread local instance using
org.elasticsearch.common.hash.MessageDigests. This is to prevent running
into java.lang.CloneNotSupportedExceptions for message digest providers
that do not support clone.

Closes #16543
2016-02-09 11:35:47 -05:00
Alexander Reelsen 0d4711c2fc Ingest: Add attachment processor
This is a simple port of the mapper attachment plugin to the ingest
functionality, no new features. The only option is to limit
the number of chars to prevent indexing of huge documents.

Fields can be selected in the processor as well.

Close #16303
2016-02-09 17:03:30 +01:00
Christoph Büscher 2ae6420fd6 Suggest: Add parsing from xContent to PhraseSuggestionBuilder
For the ongoing search refactoring (#10217) the PhraseSuggestionBuilder
gets a way of parsing from xContent that will eventually replace the
current SuggestParseElement. This PR adds the fromXContent method
to the PhraseSuggestionBuilder and also adds parsing code for the
common suggestion parameters to SuggestionBuilder.

Also adding links from the Suggester implementations registeres in the
Suggesters registry to the corresponding prototype that is going to
be used for parsing once the refactoring is done and we switch from
parsing on shard to parsing on coordinating node.
2016-02-08 12:28:12 +01:00
Christoph Büscher 97172aa2fd Merge pull request #16465 from cbuescher/issue-13884-test
Tests: Add test for minimum_should_match for SimpleQueryStringQuery
2016-02-05 13:12:53 +01:00
Christoph Büscher d885fe62ea Add test for minimum_should_match, one term and multiple fields
This adds a test case similar to the issue in #13884 which was
fixed in #16155.
2016-02-05 12:33:13 +01:00
Jim Ferenczi 5d23d9f034 Merge pull request #16454 from jimferenczi/rename_plugin_script
Rename bin/plugin in bin/elasticsearch-plugin
2016-02-05 11:25:00 +01:00
Yannick Welsch 89a5eadfea Change path separator for Checkstyle suppressions to be Windows compatible 2016-02-05 10:18:18 +01:00
Jim Ferenczi 7d0181b5d4 Rename bin/plugin in bin/elasticsearch-plugin 2016-02-05 10:09:14 +01:00
Ryan Ernst b9342b305b Also use separte buildSrc build dir for intellij 2016-02-04 15:58:05 -08:00
Ryan Ernst 2fef5dee96 Build: use separate build directories for eclipse and intellij
This fixes the problem where doing a gradle clean from the command line
completely breaks the IDE.
2016-02-04 15:44:48 -08:00
Nik Everett 7a2947a4ee Suppress LineLength in more files
These are files that grew long lines between when the last line length
suppression patch was proposed about a day ago.
2016-02-04 15:39:25 -05:00
Nik Everett 63aedb8720 Fail build for lines longer than 140 characters
140 characters is our official line length from CONTRIBUTING.md. It is a
little longer than fits well in github but you can use a userstyle to fix
that. It is perfect for Eclipse and unified diffs but too wide for side by
side diffs. Regardless, its the official limit we've had for years. We just
haven't enforced it automatically and we haven't enforced it using github
because line limits are hard to notice there.

This only hits about 2/3 of the java files - those that didn't have failures
already. If they did have a failure it suppresses them. We should pick files
off that list as time goes on.

For posterity I generated the suppressions by running checkstyle with
ignoreFailures = true, piping the output to a file, and then running it
through this:
perl -ne 'print if s{.*\[ant:checkstyle\] /.+/elasticsearch/}{  <suppress files="} && s{\.java.+}{\.java" checks="LineLength" />}' | uniq
2016-02-04 15:02:39 -05:00
Jason Tedor 6707a89c3b Fail early on JDK with compiler bug
Early versions of JDK 8 have a compiler bug that prevents assignment to
a definitely unassigned final variable inside the body of a lambda. This
commit adds an early-out to the build process which also gives a useful
error message.

Closes #16418
2016-02-03 13:25:39 -05:00
Robert Muir 20b584c7e1 Merge pull request #16400 from rmuir/lucene_4de5f1d
Upgrade to lucene 5.5.0-snapshot-4de5f1d
2016-02-03 06:45:50 -05:00
Robert Muir d5dc05f69e Upgrade to lucene 5.5.0-snapshot-1725675 2016-02-02 22:53:39 -05:00
Nik Everett 900133f1b3 Suppress checkstyle on generated files in painless
Closes #16387
2016-02-02 18:50:39 -05:00
Jason Tedor 105411060c Uppercase ells ('L') in long literals
This commit removes and forbids the use of lowercase ells ('l') in long
literals because they are often hard to distinguish from the digit
representing one ('1').

Closes #16329
2016-01-30 22:16:02 -05:00
Nik Everett e6c022b520 Mimimal checkstyle configuration
This adds a small pile of checkstyle checks that all pass without any changes.
It moves some checks from ForbiddenPatterns that were java only into
checkstyle.

Don't duplicate forbiddenPatterns in checkstyle
2016-01-30 14:59:50 -05:00
Simon Willnauer 9b3559f473 Remove `test.cluster.node.seed` and special case `tests.portsfile`
* `test.cluster.node.seed` was only used in one place where it wasn't adding value and was now replaced with a constant
 * `tests.portsfile` is now an official setting and has been renamed to `node.portsfile`

this commit also convert `search.default_keep_alive` and `search.keep_alive_interval` to the new settings infrastrucutre
2016-01-22 15:47:32 +01:00
Ryan Ernst df24019261 Merge pull request #16038 from rjernst/remove_site_plugin
Plugins: Remove site plugins
2016-01-21 12:32:22 -08:00
Robert Muir 6e7e3a2274 Update lucene to r1725675
Adds DFI (divergence from independence) provider.
Fixes test bugs passing invalid values for BM25 parameters.
2016-01-20 03:32:51 -05:00
Robert Muir 79cc3c2dd0 set -Djna.nosys when running tests.
This is consistent with what happens in elasticsearch.sh/.bat, and it means
tests will work even if there is a crazy "system" JNA installed on the machine.
2016-01-19 18:49:47 -05:00
Nik Everett 2cb7e8ce76 Use our standard xlint with standalone-test
We were not changing the xlint settings there at all. Also cleans up some
generic array warnings that this found by switching them to an ArrayList.
2016-01-19 17:54:47 -05:00
Ryan Ernst 8bf8ca6222 Fix rest tests to use ip and port again instead of localhost and port. 2016-01-18 19:28:41 -08:00
Ryan Ernst d7537075d6 Add TODO for splitting rest tests from client tests. 2016-01-18 16:59:42 -08:00
Ryan Ernst ef4f0a8699 Test: Make rest test framework accept http directly for the test cluster
The rest test framework, because it used to be tightly integrated with
ESIntegTestCase, currently expects the addresses for the test cluster to
be passed using the transport protocol port. However, it only uses this
to then find the http address.

This change makes ESRestTestCase extend from ESTestCase instead of
ESIntegTestCase, and changes the sysprop used to tests.rest.cluster,
which now takes the http address.

closes #15459
2016-01-18 16:44:14 -08:00
Ryan Ernst 3b78267c71 Plugins: Remove site plugins
Site plugins used to be used for things like kibana and marvel, but
there is no longer a need since kibana (and marvel as a kibana plugin)
uses node.js. This change removes site plugins, as well as the flag for
jvm plugins. Now all plugins are jvm plugins.
2016-01-16 22:45:37 -08:00
Daniel Mitterdorfer 871b38afcb Check cluster health in integration test wait condition
With this commit we do not check only if an endpoint is up but
we also check that the cluster status is green. Previously,
builds sporadically failed to pass this condition.
2016-01-15 09:14:32 +01:00
Daniel Mitterdorfer dc51dd0056 Add more logging to wait condition for elasticsearch nodes in tests 2016-01-14 09:48:27 +01:00
Ryan Ernst cde5bb88b1 Add printing timestamp to wait condition for elasticsearch nodes 2016-01-12 11:15:50 -08:00
Nik Everett 01ce49e94e Ban Serializable
1. Uses forbidden patterns to prevent things from referencing
java.io.Serializable or from mentioning serialVersionUID.
2. Uses -Xlint:-serial so we don't have to hear from javac that we aren't
declaring serialVersionUID on any classes that we make that happen to extend
Serializable.
3. Remove Serializable and serialVersionUID declarations.

I didn't use forbidden apis because it doesn't look like it has a way to ban
explicitly implementing Serializable. If you try to ban Serializable with
forbidden apis you end up banning all Exceptions and all Strings.

Closes #15847
2016-01-11 16:57:31 -05:00
Jason Tedor abaf816d00 Remove and forbid use of IndexWriter#isLocked
This commit removes and now forbids use of
org.apache.lucene.index.IndexWriter#isLocked as this method was
deprecated in LUCENE-6508. The deprecation is due to the fact that
checking if a lock is held before acquiring that lock is subject to a
time-of-check-to-time-of-use race condition. There were three uses of
IndexWriter#isLocked in the code base:
 - a logging statement in o.e.i.e.InternalEngine where we are already in
   an exceptional condition that the lock was held; in this case,
   logging whether or not the directory is locked is superfluous
 - in o.e.c.l.u.VersionsTests where we were verifying that a write lock
   is released upon closing an IndexWriter; in this case, the check is
   not needed as successfully closing an IndexWriter releases its
   write lock
 - in o.e.t.s.MockFSDirectoryService where we were verifying that a
   directory is not write-locked before (implicitly) trying to obtain
   such a write lock in org.apache.lucene.index.CheckIndex#<init> (this
   is the exact type of a situation that is subject to a race
   condition); in this case we can proceed by just (implicitly) trying
   to obtain the write lock and failing if we encounter a
   LockObtainFailedException
2016-01-10 08:28:44 -05:00
Jason Tedor 871d1b4885 Remove and forbid use of j.u.c.ThreadLocalRandom
This commit removes and now forbids all uses of
java.util.concurrent.ThreadLocalRandom across the codebase. The
underlying issue with ThreadLocalRandom is that it can not be
seeded. This means that if ThreadLocalRandom is used in production code,
then tests that cover any code path containing ThreadLocalRandom will be
prevented from being reproducible by use of ThreadLocalRandom. Instead,
using org.elasticsearch.common.random.Randomness#get will give
reproducible sources of random when running under tests and otherwise
still give an instance of ThreadLocalRandom when running as production
code.
2016-01-08 12:23:48 -05:00
Yannick Welsch d5191518ba Fix thirdPartyAudit check on Windows in case of jar hell with JDK 2015-12-31 12:10:48 +01:00
Robert Muir 180ab2493e Improve thirdPartyAudit check, round 3 2015-12-28 22:38:55 -05:00
Robert Muir 010d1a89c5 Merge branch 'master' into hdfs2-only 2015-12-22 00:40:54 -05:00
Adrien Grand cf52e96c42 Upgrade to lucene-5.5.0-snapshot-1721183.
Some files that implement or use the Scorer API had to be changed because of
https://issues.apache.org/jira/browse/LUCENE-6919.
2015-12-21 17:02:08 +01:00
Robert Muir f4f8b6e3fe Merge branch 'master' of github.com:elastic/elasticsearch into hdfs2-only 2015-12-20 21:59:02 -05:00
Ryan Ernst a518599e27 Allow plugins to upgrade slf4j-api to a compile dep
This was originally intended to be general purpose in #15555, but
that still had problems. Instead, this change fixes the issue explicitly
for slf4j-api, since that is the problematic dep that is not actually
included in the distributions.
2015-12-20 17:59:00 -08:00
Robert Muir 12a8428dfb Add MiniHDFS test fixture, started before integTest and shut down after.
Currently uses a hardcoded port (9999), need to apply MavenFilteringHack after it starts.
2015-12-20 16:00:37 -05:00
Ryan Ernst 9cb4c82c58 Build: Add fixture capabilities to integ tests
This change adds a Fixture class for use by gradle. A Fixture is an
external process that integration tests will use. It can be added as a
dependsOn for integTest, and will automatically be shutdown upon success
or failure, as well as relevant information dumped on failure. There is
also an example fixture in this change.
2015-12-19 15:46:21 -08:00
Ryan Ernst ad46a09cf5 Revert "Build: Allow plugins to "upgrade" provided deps"
This reverts commit 583ba76d0b.
2015-12-19 01:22:12 -08:00
Ryan Ernst 583ba76d0b Build: Allow plugins to "upgrade" provided deps
This only really applies to slf4j, since the others are in the
distribution, but it is necessary if a plugin wants to depend on slf4j.
2015-12-19 00:06:29 -08:00
Ryan Ernst af11707da0 Remove unused method impl in AntTask and make abstract 2015-12-18 14:37:15 -08:00
Ryan Ernst beec7ca9db Merge branch 'master' into wildcard_imports 2015-12-18 13:18:08 -08:00
Ryan Ernst d0a10b337e Fix silly typo in variable name... 2015-12-18 13:17:05 -08:00
Ryan Ernst a639b3d3e4 Remove wildcard imports for groovy too 2015-12-18 12:47:35 -08:00
Ryan Ernst 4ea19995cf Remove wildcard imports 2015-12-18 12:43:47 -08:00
Ryan Ernst 853e9c0fd1 Merge branch 'master' into wildcard_imports 2015-12-18 12:24:30 -08:00
Ryan Ernst 44edac0081 use better variable name for build listener, and change access back to
private for internal method of thirdPartyAudit
2015-12-18 12:07:30 -08:00
Ryan Ernst 5b9bf8e738 Make a new ant builder per AntTask invocation 2015-12-18 12:01:54 -08:00
Ryan Ernst 9f1dfdbaea Build: Add AntTask to simplify controlling logging when running ant from gradle
This new task allows setting code, similar to a doLast or doFirst,
except it is specifically geared at running ant (and thus called doAnt).
It adjusts the ant logging while running the ant so that the log
level/behavior can be tweaked, and automatically buffers based on gradle
logging level, and dumps the ant output upon failure.
2015-12-18 11:45:52 -08:00
Robert Muir 2ce54640f5 Remove unnecessary license categories/matchers 2015-12-18 13:46:59 -05:00
Robert Muir c4f823903a fix indent 2015-12-18 12:00:52 -05:00
Robert Muir e82808917c don't fail on missing source dirs 2015-12-18 11:58:18 -05:00
Robert Muir 94d6b22137 add gradle licenseHeaders to precommit
This is a port of the logic from apache lucene that uses Rat
2015-12-18 11:43:58 -05:00
Ryan Ernst fbff877ec3 Fix thirdpartyaudit to ignore all elasticsearch packages 2015-12-17 22:31:40 -08:00
Ryan Ernst 6a99796b02 Build: Move test framework under a "test" top level dir
This allows adding more test projects, eg integ test fixtures that will
be coming soon.
2015-12-17 21:25:06 -08:00
Robert Muir 6692e42d9a thirdPartyAudit round 2
This fixes the `lenient` parameter to be `missingClasses`. I will remove this boolean and we can handle them via the normal whitelist.
It also adds a check for sheisty classes (jar hell with the jdk).
This is inspired by the lucene "sheisty" classes check, but it has false positives. This check is more evil, it validates every class file against the extension classloader as a resource, to see if it exists there. If so: jar hell.

This jar hell is a problem for several reasons:

1. causes insanely-hard-to-debug problems (like bugs in forbidden-apis)
2. hides problems (like internal api access)
3. the code you think is executing, is not really executing
4. security permissions are not what you think they are
5. brings in unnecessary dependencies
6. its jar hell

The more difficult problems are stuff like jython, where these classes are simply 'uberjared' directly in, so you cant just fix them by removing a bogus dependency. And there is a legit reason for them to do that, they want to support java 1.4.
2015-12-17 02:35:00 -05:00
Robert Muir 42138007db add some more comments about internal api usage 2015-12-16 18:56:02 -05:00
Robert Muir 49f37a526c fix bad indent 2015-12-16 16:50:52 -05:00
Robert Muir ee79d46583 Add gradle thirdPartyAudit to precommit tasks 2015-12-16 16:38:16 -05:00
Ryan Ernst 5c8a0da1fd Build: Change rest integ tests to not have hardcoded ports
This change removes hardcoded ports from cluster formation. It passes
port 0 for http and transport, and then uses a special property to have
the node log the ports used for http and transport (just for tests).
This does not yet work for multi node tests. This brings us one step
closer to working with --parallel.
2015-12-11 17:36:08 -08:00
Ryan Ernst 47e922ba85 Add excludes and patterns as explicit inputs to forbidden patterns task 2015-12-11 08:26:38 -08:00
Jason Tedor 3383c24be0 Remove and forbid use of Collections#shuffle(List) and Random#<init>()
This commit removes and now forbids all uses of
Collections#shuffle(List) and Random#<init>() across the codebase. The
rationale for removing and forbidding these methods is to increase test
reproducibility. As these methods use non-reproducible seeds, production
code and tests that rely on these methods contribute to
non-reproducbility of tests.

Instead of Collections#shuffle(List) the method
Collections#shuffle(List, Random) can be used. All that is required then
is a reproducible source of randomness. Consequently, the utility class
Randomness has been added to assist in creating reproducible sources of
randomness.

Instead of Random#<init>(), Random#<init>(long) with a reproducible seed
or the aforementioned Randomess class can be used.

Closes #15287
2015-12-11 11:16:38 -05:00
Robert Muir bc140659a0 fail build on wildcard imports
Wildcard imports are terrible, they cause ambiguity in the code,
make it not compile with the future versions of java in many cases.

We should simply fail the build on this, it is messiness, caused by
messy Intellij configuration
2015-12-11 11:03:44 -05:00
Robert Muir 2741888498 Remove RuntimePermission("accessDeclaredMembers")
Upgrades lucene to 5.5.0-1719088, randomizedtesting to 2.3.2, and securemock to 1.2
2015-12-10 14:26:55 -05:00
Robert Muir e454fadc22 Merge branch 'master' into shave_mustache 2015-12-10 07:58:24 -05:00
Robert Muir b6fbd18e09 Merge pull request #15322 from rmuir/awesome
make slow tests more obvious
2015-12-09 15:05:53 -05:00
Ryan Ernst 46d73a13ff Merge branch 'master' into shave_mustache 2015-12-08 21:11:48 -08:00
Ryan Ernst 18698b9308 Build: Add build plugin for "messy" tests
We have some tests which have crazy dependencies, like on other plugins.
This change adds a "messy-test" gradle plugin which can be used for qa
projects that these types of tests can run in. What this adds over
regular standalone tests is the plugin properties and metadata on the
classpath, so that the plugins are properly initialized.
2015-12-08 19:48:39 -08:00
Robert Muir 2e299860e5 refactor mustache to lang-mustache plugin. No rest tests yet. 2015-12-08 22:45:21 -05:00
Robert Muir b0febc1b9f make slow tests more obvious 2015-12-08 13:05:15 -05:00
Ryan Ernst fa1c708ccd Merge branch 'master' into standalone_tests_intellij 2015-12-07 16:01:32 -08:00
Ryan Ernst 2dbd93d3b0 Tweaked module error message against non integ-test-zip and made checks
for limiting configuration to certain distributions prettier
2015-12-04 11:40:08 -08:00
Ryan Ernst 01d48e2062 Merge branch 'master' into jigsaw 2015-12-04 11:29:49 -08:00
Ryan Ernst a6752c4da9 Merge pull request #15210 from rjernst/buildsrc_test_cleanup
Cleanup integ test classes in buildSrc to be less groovyish
2015-12-04 11:24:21 -08:00
Ryan Ernst 4b1fc8ff22 Merge pull request #15226 from rjernst/fix_update_shas
Fix updateShas to not barf on disabled license checks and even compile correctly
2015-12-03 23:34:33 -08:00
Ryan Ernst f5b6b40a92 Fix distros to put modules dir int he right place and qa rest tests to
use the real zip
2015-12-03 23:03:14 -08:00
Ryan Ernst a8e9403204 added gradle checks for modules configuration, and ability to add
modules to integ test cluster
2015-12-03 20:53:06 -08:00
Ryan Ernst f2cebd9b3b Merge branch 'master' into jigsaw 2015-12-03 20:02:44 -08:00
Ryan Ernst 0be141e021 Build: Add .settings for buildSrc on gradle eclipse
We have eclipse settings added to all projects when running gradle
eclipse, but buildSrc is its own special project that is not
encapsulated by allprojects blocks. This adds eclipse settings to
buildSrc.
2015-12-03 19:56:11 -08:00
Ryan Ernst 17e6195fd6 Build: Fix updateShas to not barf on disabled license checks and even compile correctly
These were just results of not testing properly after refactoring.

closes #15224
2015-12-03 15:21:14 -08:00
Ryan Ernst caf77f7eea Add integ-test-zip distribution 2015-12-03 14:52:51 -08:00
Ryan Ernst b00c59398b Build: Cleanup test classes in buildSrc to be less groovyish 2015-12-03 01:49:54 -08:00
Robert Muir 1f626d3458 forbidden third-party-signatures -> core-signatures
This is a relic from shading where it was trickier to implement.
Third party signatures are already in e.g. the test list, there
is no reason to separate them out.

Instead, we could have a third party signatures that does
something different... like keep tabs on third party libraries.
2015-12-03 01:45:06 -05:00
Ryan Ernst f8026ed8fd Build: Fix dependency licenses check to correctly skip projects without dependencies 2015-12-02 11:03:56 -08:00
Jason Tedor 05430a788a Remove and forbid use of the type-unsafe empty Collections fields
This commit removes and now forbids all uses of the type-unsafe empty
Collections fields Collections#EMPTY_LIST, Collections#EMPTY_MAP, and
Collections#EMPTY_SET. The type-safe methods Collections#emptyList,
Collections#emptyMap, and Collections#emptySet should be used instead.
2015-12-02 10:41:59 -05:00
Robert Muir b7f497627f Merge pull request #15172 from rmuir/nuke_unused_deps
remove unused core dependencies
2015-12-02 07:20:11 -05:00
David Pilato 9c77cdc201 Fix build failure caused by Licenses Check in test-framework module
Typical failure:

```
:test-framework:dependencyLicenses (Thread[main,5,main]) started.
:test-framework:dependencyLicenses
Executing task ':test-framework:dependencyLicenses' (up-to-date check took 0.0 secs) due to:
  Task has not declared any outputs.
:test-framework:dependencyLicenses FAILED
:test-framework:dependencyLicenses (Thread[main,5,main]) completed. Took 0.023 secs.

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':test-framework:dependencyLicenses'.
> Licences dir /mnt/jenkins/workspace/es_core_master_strong/test-framework/licenses does not exist, but there are dependencies
```

Related to #15168
2015-12-02 10:08:35 +01:00
Ryan Ernst d68c6673a2 Build: Cleanup precommit task gradle code
This change attempts to simplify the gradle tasks for precommit. One
major part of that is using a "less groovy style", as well as being more
consistent about how tasks are created and where they are configured. It
also allows the things creating the tasks to set up inter task
dependencies, instead of assuming them (ie decoupling from tasks
eleswhere in the build).
2015-12-01 22:36:54 -08:00
Robert Muir 6e5c301739 remove unused dependencies 2015-12-01 22:22:11 -05:00
Ryan Ernst fcb6f44acf Build: Add ability to specify the compact profile for javac 2015-12-01 11:04:46 -08:00
Robert Muir f7e7a6bfad compile against compact3 profile 2015-12-01 09:40:54 -05:00
Ryan Ernst 4bb1eed766 Build: Increase the number of failed tests shown in test summary
We had increased this in maven, but it was lost in the transition to
gradle. This change adds it as a configurable setting the the logger for
randomized testing and bumps it to 25.
2015-11-30 13:40:39 -08:00
Ryan Ernst 0fa05edf03 Build: Improve output when integ test fails
This outputs a lot more info when integ tests fail to start, as well as
(should) fix windows (at least in my VM run) integ tests.
2015-11-29 19:44:43 -08:00
Robert Muir 93758874fb Merge pull request #15093 from rmuir/gradle_windoze_integ_tests
Fix integration tests and 'gradle run' to work on windoze
2015-11-29 12:15:30 -05:00
Robert Muir a01f86aaac Fix integration tests and gradle run to work on windoze 2015-11-29 07:39:18 -08:00
Ryan Ernst 8e25dc4f80 Merge pull request #15089 from rjernst/extra_config_file_delay
Fix delayed extra config file checks to be right before copy
2015-11-28 19:44:16 -08:00
Ryan Ernst 018c766bf7 Build: Fix delayed extra config file checks to be right before copy
The current delay waits until later after normal configuration, but
still just after resolution (ie when paths would be known for
dependencies but not actual execution). This delays the checks further
to be done right before we actually execute the copy task.

closes #15068
2015-11-28 18:06:36 -08:00
Ryan Ernst 1ee59e4977 Merge pull request #15088 from rjernst/integ_test_command_line
Move debug options setting to before command line is logged
2015-11-28 17:19:39 -08:00
Ryan Ernst f15ee9fa54 Build: Move debug options setting to before command line is logged
Currently if running with --info, the command line for ES, along with
env vars, are logged before they may be ammended to add debug options.
This moves the adding JAVA_OPTS to before we print the command.
2015-11-28 17:05:11 -08:00
Ryan Ernst 30fe74b5df Build: Setup standalone tests to compile in intellij
This adds the standalone tests so they will compile (and thus can be
modified with import completion) within IntelliJ. It also explicitly
sets up buildSrc as a module.

Note that this does *not* mean eg evil-tests can be run from intellij.
These are special tests that require special settings (eg disabling
security manager). They need to be run from the command line.

closes #15075
2015-11-28 16:57:09 -08:00
Nik Everett eba594fc08 Remove strange checks and clean dirs 2015-11-28 19:44:53 -05:00
Nik Everett 45e71329ee [build] Fix deb and rpm tests
Gradle is causing so much trouble here! Its too cute for its own good.
2015-11-28 19:07:06 -05:00
Robert Muir 66ae46ef8b Give enough ram so javac won't OOM
Today its based on the automatic -Xmx, but this
will fail in some environments (e.g. vagrant).
2015-11-26 02:16:46 -07:00
Ryan Ernst 139d688e24 Merge pull request #14996 from rjernst/logged_exec
Add wrapper around gradle Exec task for easier logging
2015-11-25 11:26:30 -08:00
Ryan Ernst bcb4be322f Build: Add wrapper around gradle Exec task for easier logging
The Exec task outputs stdout/stderr to the standard streams by default.
However, to keep output short, we currently capture this, and only
output if the task failed. This change makes a small wrapper around Exec
to facilitate this behavior anywhere we use Exec.
2015-11-25 11:26:02 -08:00
Ryan Ernst 52f31ee14a Build: Allow extra config for integ test to be anything project.file() accepts
This change delays the lookup for whatever is passed to extra config as
the source file to happen at execution time. This allows using eg a task
which generates a file, but maintains the checks that the file is not a
dir and that it exists at runtime.
2015-11-25 00:34:56 -08:00
Ryan Ernst d6969fcf3a Build: Split extra config files for integ test into separate task
This doesn't work when there are no extra config files, since gradle
sees the copy task and says "there is nothing to do".
2015-11-24 16:37:32 -08:00
Ryan Ernst 84d6cbd32a Build: Fix extra config to create a file, not a directory for the destination file
Also added some checks to catch misconfiguration (dir or non existing file).
2015-11-24 16:09:47 -08:00
Ryan Ernst 7b0a0ce0d9 Build: Add ability to specify extra configuration files for integ test
This change allows copy extra files into the integ test cluster before
it runs. However, it explicitly forbids overwriting elasticsearch.yml,
since that is generated.
2015-11-24 15:19:51 -08:00
Ryan Ernst 3a77d993fd Build: Copy rest specs for plugins that are added to integ test cluster
This change allows having rest tests using the api spec of plugins that
the rest test is testing against.
2015-11-24 14:34:21 -08:00
Ryan Ernst 1e0f929281 Build: Simplify adding plugins that are another project in the build
The current mechanism for adding plugins to the integTest cluster is to
have a FileCollection. This works well for the integTests for a single
plugin, which automatically adds itself to be installed. However, for qa
tests where many plugins may be installed, and from other projects, it
is cumbersome to add configurations, dependencies and dependsOn
statements over and over. This simplifies installing a plugin from
another project by moving this common setup into the cluster
configuration code.
2015-11-24 12:53:11 -08:00
Ryan Ernst f0e780bf86 Merge pull request #14959 from rjernst/cluster_waiting
Allow customizing wait condition and cluster settings
2015-11-24 08:24:00 -08:00
Ryan Ernst 647f324c46 Change waitfor max time back to 30 seconds, in case jenkins is slow 2015-11-24 08:23:16 -08:00
Ryan Ernst af496f5e7f Merge pull request #14925 from rjernst/maven_test_parity
Add back leaveTemporary and ifNoTests
2015-11-24 00:53:25 -08:00
Ryan Ernst 31b7e91e5a Build: Allow customizing wait condition and cluster settings
The current wait condition for an integ test cluster being up is a
simple http get on the root path for elasticsearch. However, it is
useful to allow having arbitrary wait conditions. This change reworks
the wait task to first check that each node process started successfully
and has a socket up, followed by an arbitrary wait condition which
defaults to the current http get.

Also, cluster settings are allowed to be added, and overriden. Finally,
custom setup commands are made relative to the elasticsearch home dir
for each node.
2015-11-24 00:48:58 -08:00
Ryan Ernst 529e442e2a Merge branch 'master' into attach_without_subdir 2015-11-23 15:01:55 -08:00
Ryan Ernst 5a62282150 Build: Rework extra plugins support to be through sibling directories
This change removes the subdirectory support for extra-plugins, and
replaces it with an iteration of sibling directories with the prefix
"extra-plugin-".
2015-11-23 14:15:45 -08:00
Michael McCandless e13b0d4bde upgrade lucene to 5.4.0-snapshot-1715952 2015-11-23 17:13:49 -05:00
Ryan Ernst 503d836ee4 Build: Add back leaveTemporary and ifNoTests
This change adds back the last of the missing test options to the juni4
wrapper. leaveTemporary is important in that setting it to true (which
is how we had it set in maven) removes the warnings we currently get
about a leftover file that cannot be deleted (from jna).
2015-11-22 22:18:42 -08:00
Ryan Ernst 7d3da7e6ee Build: Get multi node smoke tests working
This change adds back the multi node smoke test, as well as making the
cluster formation for any test allow multiple nodes. The main changes in
cluster formation are abstracting out the node specific configuration to
a helper struct, as well as making a single wait task that waits for all
nodes after their start tasks have run. The output on failure was also
improved to log which node's info is being printed.
2015-11-22 14:49:05 -08:00
Robert Muir 30529c008d Ban write access to system properties
* Forbid System.setProperties & co in forbidden APIs.
* Ban property write access at runtime with security manager.

Plugins that need to modify system properties will need to request permission in their plugin-security.policy
2015-11-21 22:33:06 -05:00
Ryan Ernst a2ef228d12 Build: Add precommit checks to standalone tests 2015-11-21 11:50:43 -08:00
Nik Everett 9b6e59bd57 [build] Survive missing directories 2015-11-20 17:38:50 -05:00
Ryan Ernst 61d62078f5 Merge pull request #14779 from rjernst/fork_compile
Use JDK at JAVA_HOME for compiling/testing, and improve build info output
2015-11-20 12:12:49 -08:00
Ryan Ernst df9cd96e9f Merge pull request #14858 from rjernst/check_jarhell
Add jar hell check before tests run
2015-11-20 11:35:49 -08:00
Ryan Ernst 6b734dc009 Remove debugging output from running jjs scripts 2015-11-20 11:31:46 -08:00
Ryan Ernst 63a7b155d8 Merge branch 'master' into fork_compile 2015-11-20 11:30:36 -08:00
Ryan Ernst c650d31ef1 Build: Improve integ test startup behavior
As part of the refactoring to allow --debug-jvm with gradle run, the way
java options are passed for integ tests was changed. However, we need to
make sure the jvm argline passed goes to ES_GC_OPTS because this
allows overriding things like which garbage collector we run, which we
do for testing from jenkins. This change adds back ES_GC_OPTS.
2015-11-19 13:06:47 -08:00
Ryan Ernst 1952b06cfe Merge pull request #14854 from rjernst/trick_ant
Fix integTest output if the elasticsearch script fails
2015-11-19 12:03:03 -08:00
Ryan Ernst e832fc6000 Make tmp file handling for jjs scripts better 2015-11-19 11:24:31 -08:00
Ryan Ernst d1a27aed16 Make the tmp file created for jjs scripts be deleted when gradle exits 2015-11-19 10:33:31 -08:00
Ryan Ernst c93f074183 Merge branch 'master' into fork_compile 2015-11-19 10:26:06 -08:00
Ryan Ernst 0e8958d1e8 Fix jarhell marker to be written on info logging, and document how the
marker file works.
2015-11-19 10:21:25 -08:00
Ryan Ernst fce0052fd9 Build: Add jar hell check before tests run
Because jar hell checks run during static initialization of tests, a
failure will result in all tests failing. However, only the first test
within each jvm shows the jarhell failure, and later tests get a class
not found exception for the class that failed to load when static init
failed.

This change adds a task to run as part of precommit, which checks the
test runtime classpath for jarhell.

closes #14721
2015-11-19 02:07:15 -08:00
Ryan Ernst d44dbd4757 Build: Fix integTest output if the elasticsearch script fails
If there is a failure in the elasticsearch start script, we currently
completely lose the failure. This is due to how spawning works with ant.
This change avoids the issue by introducing an intermediate script,
built dynamically before running ant exec, which runs elasticsearch and
redirects the output to a log file. This essentially causes us to run
elasticsearch in the foreground and capture the output, but at the same
time keep a running script which ant can pump streams from (which will
always be empty).
2015-11-18 23:30:33 -08:00
Ryan Ernst c273327246 Build: Improve test output on errors and when debugging
There were a number of subtle issues with the existing logging that
wraps events from Junit4 and ant. This change:
* Tweaks at what level certain events are logged
* Fixes -Dtests.output=always to force everything to be logged
* Makes -Dtests.class imply -Dtests.output=always
* Captures ant logging from junit4, so that direct jvm output will be
  logged on failure when not using gradle info logging
2015-11-18 16:36:07 -08:00
Nik Everett 8ffb63a177 Merge pull request #14718 from nik9000/gradle_deb
Get packaging working in gradle
2015-11-18 14:17:11 -05:00
Nik Everett 9b0a47d8e3 Fix rpm and deb distributions
and test them with vagrant
2015-11-18 14:16:42 -05:00
Nik Everett 19919a53a5 [build] Log launch command before starting ES 2015-11-18 12:25:27 -05:00
Nik Everett 1f67ef2526 [build] Log more information when es doesn't start 2015-11-18 10:58:52 -05:00
Nik Everett 22873dc952 [build] Output a message when can't find log file
When the build tries to start an elasticsearch instance but the start fails
if it fails to find the log file then we log a line about how we can't find
the file.
2015-11-18 10:43:01 -05:00
Ryan Ernst 9f59ba5abf Tweak debug startup message for run command, per PR comment 2015-11-17 17:18:44 -08:00
Ryan Ernst 0e8d605e2d Build: Add --debug-jvm option to run and integTest tasks
Sometimes when running elasticsearch, it is useful to attach a remote
debugger. This change adds a --debug-jvm option (the same name gradle
uses for its tests debug option), which adds java agent config for a
remote debugger. The configuration is set to hava java suspend until the
remove debugger is attached.

closes #14772
2015-11-17 16:29:01 -08:00
Nik Everett e0646eeb89 Merge pull request #14754 from nik9000/build
Make Build work without git
2015-11-17 13:30:23 -05:00
Nik Everett 0119caa4a6 Make Build work without git
If you build elasticsearch without a git repository it was creating a null
shortHash which was causing Elasticsearch not to be able to form transport
connections.

Closes #14748
2015-11-17 13:25:09 -05:00
Nik Everett 74aa18088c [build] Update progress logger after each test
This makes the rest tests **tons** more responsive.

Also stop test progress output from jumping by using formating. The progess
now looks like:
Suites [004/549], Tests [0019|0|0], in 1.58s J2 completed UpdateNumberOfReplicasTests

The changes included are:
1. The suites, total tests, and JVM id are now padded based on their maximum
size. The maximum number of tests is just a guess because that data isn't
easily available when the suite starts. JVM id rarely matters because only
the most crazy individuals use more than 10 JVMs.

2. The suite information is reordered. Now its runtime, jvm id, suite name,
and, optionally, method name. This reordering is useful because the thing
that varies in length, the suite and method name, are on the right hand
side. This means that nothing jumps around during the test run.
2015-11-17 13:16:17 -05:00
Ryan Ernst e837140385 Merge pull request #14791 from rjernst/run_away
Build: Make run command work within plugins
2015-11-17 09:09:26 -08:00
Ryan Ernst 50b48e7d8c Build: Make run command work within plugins
We recently got a run command with gradle, but it is sometimes useful to
run ES with a specific plugin. This is a start, by making each esplugin
have a run command which installs the plugin and runs elasticsearch in
the foreground.
2015-11-16 17:29:01 -08:00
Ryan Ernst 176f9e9d89 Build: Use a marker file to indicate forbidden patterns has run
This makes forbidden patterns a little smarter, so it does not need to
run on every build. It works because the marker file timestamp will be
compared against the source files (which are the inputs to forbidden
patterns).

closes #14788
2015-11-16 16:48:56 -08:00
Michael McCandless 01f12fc8d5 Merge pull request #14784 from mikemccand/upgrade_lucene_1714615
Upgrade Lucene to 5.4.0-snapshot-1714615
2015-11-16 16:07:46 -05:00
Michael McCandless a0bf253d16 upgrade lucene 5.4 snapshot 2015-11-16 14:38:05 -05:00
Ryan Ernst d872e2c527 Build: Use JDK at JAVA_HOME for compiling/testing, and improve build info output
We currently enforce JAVA_HOME is set, but use gradle's java version to
compile, and JAVA_HOME version to test. This change makes the compile
and test both use JAVA_HOME, and clarifies which java version is used by
gradle and that used for the compile/test.
2015-11-16 09:15:31 -08:00
Ryan Ernst 759fc94947 Build: Override Build-Date in jar manifest to be iso 8601 2015-11-14 22:38:10 -08:00
Ryan Ernst 5eb5480ea6 Improve wording of transitive deps hack explanation 2015-11-13 15:05:40 -08:00
Ryan Ernst 24d306c2f4 Add explanation of the configuration trick used to find transitive deps
for exclusion
2015-11-13 14:59:45 -08:00
Ryan Ernst be147719ff Remove artifacts left from testing 2015-11-13 14:20:14 -08:00
Ryan Ernst 1d38f0ac8e Remove forgotten println 2015-11-13 14:18:14 -08:00
Ryan Ernst 837c593ec2 Build: Effectively remove transitive deps from generated maven poms
With gradle, deploying to maven means first generating poms. These are
filled in based on dependencies of the project. Recently, we started
disallowing transitive dependencies. However, this configuration does
not translate to maven poms because maven has no concept of excluding
all transitive dependencies.

This change adds exclusions for each of the transitive deps of each
dependency being added to the maven pom. It does so by creating dummy
configurations for each direct dependency (which does not have
transitive deps excluded), so that we can iterate the transitive deps
when building the pom.

Note, this should be simpler (just modifying maven's pom model), but
gradle tries to hide that from their api, causing us to need to
manipulate the xml directly.
https://discuss.gradle.org/t/modifying-maven-pom-generation-to-add-excludes/12744
2015-11-13 14:10:17 -08:00
Ryan Ernst 9373382155 Build: Use ant exec for starting elasticsearch in integ tests
Currently elasticsearch in integ tests is started using an ant task on
windows, or gradle exec on everything else. However, gradle exec has
some flaws, one being Ctrl-C does not run finalizedBy tasks, which means
interrupting integ tests will leak a jvm. This change makes all systems
use ant exec. One caveat is, if there is any output by the jvm, we lose
it in ant bit heaven. But this is no different than what we had with
gradle. In the future, we should look at using a separate thread to
pump streams from the elasticsearch process.

closes #14701
2015-11-12 22:43:38 -08:00
Robert Muir 776bb288b5 fix gradle check under jigsaw
closes #14726

Squashed commit of the following:

commit 5b591e98570e3fa481b2816a44063b98bff36ddf
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Nov 13 00:54:08 2015 -0500

    add assumption for self-signing in PluginManagerTests

commit ed11e5371b6f71591dc41c6f60d033502cfcf029
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Nov 13 00:20:59 2015 -0500

    show error output from integ test startup

commit d8b187a10e95d89a0e775333dcbe1aaa903fb376
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Nov 12 22:14:11 2015 -0500

    fix gradle check under jigsaw
2015-11-13 00:57:12 -05:00
Ryan Ernst f327beac49 Build: Enforce JAVA_HOME is set
If we use JAVA_HOME consistently for tests, we can run tests with a
different version of java than gradle runs with. For example, this
enables running tests with jigsaw, but building with java 8. The only
caveat is intellij does not set JAVA_HOME. This change enforces
JAVA_HOME is set, but ignores for intellij.
2015-11-12 10:22:56 -08:00
Ryan Ernst 505a4d9e09 Build: Simplify where min java version is specified
This moves the min java version used by elasticsearch to one place, a
constant in BuildPlugin. For me on java 9, this fixed my jar to have the
correct target/source versions.

closes #14702
2015-11-11 23:09:57 -08:00
Ryan Ernst 83f3a981b7 Merge pull request #14668 from rjernst/transitive_goes_poof
Remove transitive dependencies
2015-11-10 19:33:47 -08:00
Jason Tedor bfe27407ae More explicit handling of jps executable on Windows 2015-11-10 18:03:16 -05:00
Ryan Ernst 4b5f87cb7d Build: Remove transitive dependencies
Transitive dependencies can be confusing and hard to deal with when
conflicts arise between them. This change removes transitive
dependencies from elasticsearch, and forces any dependency conflicts to
be resolved manually, instead of automatically by gradle.

closes #14627
2015-11-10 15:01:41 -08:00
Jason Tedor ebd8bbcffe Get correct jps path on Windows
This commit fixes an issue with the jps executable sometimes being
available under jps.exe but not jps.
2015-11-10 14:56:45 -05:00
Ryan Ernst d64d1c99d4 Build: Upgrade minimum gradle version to 2.8
In gradle 2.7 (or groovy 2.3.10, not sure which), there appears to be a
bug on linux where using a fully qualified class (without an import
statement) does not work. This change forces gradle 2.8 or above. It
also moves the logic around a little for the version check so the build
info is printed before checks against that info.
2015-11-10 11:17:30 -08:00
Ryan Ernst 4f5de0f453 Add back eclipse configuration to buildSrc 2015-11-10 07:30:11 -08:00
Ryan Ernst 7a6155e12f Build: Consolidate dependencies specified in multiple places
Some dependencies must be specified in a couple places in the build.
e.g. randomized runner is specified both in buildSrc (for the gradle
wrapper plugin), as well as in the test-framework.

This change creates buildSrc/versions.properties which acts similar to
the set of shared version properties we used to have in the maven parent
pom.
2015-11-09 23:59:56 -08:00
Ryan Ernst fc7bd7bfba Build: Fix automatic installation of plugin for integ tests
The esplugin gradle plugin automatically adds the pluging being built to
the integTest cluster. However, there were two issues with this. First
was a bug in the name, which should have been the configured
esplugin.name instead of the project name. Second, the files
configuration was overcomplicated (trying to use the groovy spreader
operator after delaying calls to singleFile). Instead, we can just pass
the file collections (which will just be a single file at execution
time).
2015-11-09 17:27:34 -08:00
Ryan Ernst 7034feeceb Merge pull request #14451 from andrejserafim/run
Add gradle run command to replace run.bat and run.sh
2015-11-09 11:15:53 -08:00
Jason Tedor 5c3a0d63ea Get jps path relative to the JVM running Gradle
This commit addresses an issue in getting a path to the jps bin. The
solution is to get the path to the JDK relative to the JVM running
Gradle.

Closes #14614
2015-11-09 13:24:59 -05:00
Jason Tedor 013347be9b Revert "Merge pull request #14623 from nik9000/no_home_home"
This reverts commit cf8ffd32ac, reversing
changes made to 40650ad2d5.
2015-11-09 10:37:16 -05:00
javanna 10ddd691a3 Internal: move to lucene BoostQuery
Latest version of lucene deprecated Query#setBoost and Query#getBoost which made queries effectively immutable. Those methods need to be replaced with `BoostQuery` that wraps any query that needs boosting.

This commit replaces usages of setBoost with BoostQuery and adds it to forbidden-apis for prod code.

Usages of `getBoost` are only partially removed, as some will have to stay for backwards compatibility.

Closes #14264
2015-11-09 15:01:35 +01:00
Nik Everett a7b78bf67a [build] Use always set java.home
`System.getenv('JAVA_HOME')` relies on JAVA_HOME being setup by the user.
`System.getProperty('java.home')` is set by Java all the time.

Closes #14614
2015-11-09 08:12:08 -05:00
andrejserafim 2bd353d72d replacing run.bat and run.sh with gradle run
run.sh and run.bat were calling out to the old maven build system.
This is no longer in place, so we've created new gradle tasks to
start an elasticsearch node from the current codebase.

fixed #14423
2015-11-08 17:07:19 +00:00
Ryan Ernst 8d78f970b0 Build: Enforce name is set for all plugins
The plugin name currently defaults to the gradle project name. But the
gradle project name for standalone repo (like an external plugin would
be) defaults to the directory name of the repo. This is trappy, since it
depends on how the repo was checked out.

This change enforces the plugin name is always set.

closes #14603
2015-11-07 16:57:11 -08:00
Ryan Ernst d0f5950c80 Merge pull request #14598 from rjernst/integ_test_parity
Improve integ test to match ant behavior
2015-11-07 13:09:04 -08:00
Ryan Ernst 22bcf197b7 Add some comments to particularly nasty groovy code 2015-11-07 13:06:14 -08:00
Robert Muir 866a514785 add groovy build code (buildSrc/) in 'gradle eclipse'
This makes it a groovy project that works in eclipse.
You will have to install a plugin for groovy language support
(I used a snapshot build from https://github.com/groovy/groovy-eclipse/wiki)
2015-11-07 15:14:00 -05:00
Ryan Ernst 9ee315a9c8 Build: Improve integ test to match ant behavior
Many other improvements:
* Use spaces in ES path
* Use space in path for plugin file installation
* Use a different cwd than ES home
* Use jps to ensure process being stopped is actually elasticsearch
* Stop ES if pid file already exists
* Delete pid file when successfully killed

Also, refactored the cluster formation code to be a little more organized.

closes #14464
2015-11-07 00:45:19 -08:00
Jason Tedor 97644e3046 Merge pull request #14588 from jasontedor/enforce-java-8
Enforce Java 8 in Gradle builds
2015-11-06 12:33:28 -05:00
Jason Tedor b0c7065000 Cleaner version check 2015-11-06 12:28:31 -05:00
Jason Tedor faf5f55f41 Enforce Java 8 in Gradle builds
This commit forces builds with Gradle to require at least Java 8.

Closes #14499
2015-11-06 11:39:14 -05:00
Ryan Ernst c71cc0f21d Revert "Build: Fail integ test if elasticsearch fails to start on windows"
This reverts commit f974d5e470.

Apprently spawn does not support failonerror in ant.
2015-11-06 08:22:17 -08:00
Nik Everett 047e3b9b69 Merge pull request #14531 from nik9000/gradle_vagrant
Gradle-ify vagrant tests
2015-11-05 14:29:06 -05:00
Nik Everett 1dd00dddd5 [test] Gradle-ify vagrant tests
This gets the tar and tar_plugins tests working in gradle. It does so by
adding a subproject, qa/vagrant, which adds the following tasks:

Verification
------------
checkPackages - Check the packages against a representative sample of the
                linux distributions we have in our Vagrantfile
checkPackagesAllDistros - Check the packages against all the linux
                          distributions we have in our Vagrantfile

Package Verification
--------------------
checkCentos6 - Run packaging tests against centos-6
checkCentos7 - Run packaging tests against centos-7
checkDebian8 - Run packaging tests against debian-8
checkFedora22 - Run packaging tests against fedora-22
checkOel7 - Run packaging tests against oel-7
checkOpensuse13 - Run packaging tests against opensuse-13
checkSles12 - Run packaging tests against sles-12
checkUbuntu1204 - Run packaging tests against ubuntu-1204
checkUbuntu1404 - Run packaging tests against ubuntu-1404
checkUbuntu1504 - Run packaging tests against ubuntu-1504

Vagrant
-------
smokeTestCentos6 - Smoke test the centos-6 VM
smokeTestCentos7 - Smoke test the centos-7 VM
smokeTestDebian8 - Smoke test the debian-8 VM
smokeTestFedora22 - Smoke test the fedora-22 VM
smokeTestOel7 - Smoke test the oel-7 VM
smokeTestOpensuse13 - Smoke test the opensuse-13 VM
smokeTestSles12 - Smoke test the sles-12 VM
smokeTestUbuntu1204 - Smoke test the ubuntu-1204 VM
smokeTestUbuntu1404 - Smoke test the ubuntu-1404 VM
smokeTestUbuntu1504 - Smoke test the ubuntu-1504 VM
vagrantHaltCentos6 - Shutdown the vagrant VM running centos-6
vagrantHaltCentos7 - Shutdown the vagrant VM running centos-7
vagrantHaltDebian8 - Shutdown the vagrant VM running debian-8
vagrantHaltFedora22 - Shutdown the vagrant VM running fedora-22
vagrantHaltOel7 - Shutdown the vagrant VM running oel-7
vagrantHaltOpensuse13 - Shutdown the vagrant VM running opensuse-13
vagrantHaltSles12 - Shutdown the vagrant VM running sles-12
vagrantHaltUbuntu1204 - Shutdown the vagrant VM running ubuntu-1204
vagrantHaltUbuntu1404 - Shutdown the vagrant VM running ubuntu-1404
vagrantHaltUbuntu1504 - Shutdown the vagrant VM running ubuntu-1504
vagrantSmokeTest - Smoke test some representative distros from the Vagrantfile
vagrantSmokeTestAllDistros - Smoke test all distros from the Vagrantfile
vagrantUpCentos6 - Startup a vagrant VM running centos-6
vagrantUpCentos7 - Startup a vagrant VM running centos-7
vagrantUpDebian8 - Startup a vagrant VM running debian-8
vagrantUpFedora22 - Startup a vagrant VM running fedora-22
vagrantUpOel7 - Startup a vagrant VM running oel-7
vagrantUpOpensuse13 - Startup a vagrant VM running opensuse-13
vagrantUpSles12 - Startup a vagrant VM running sles-12
vagrantUpUbuntu1204 - Startup a vagrant VM running ubuntu-1204
vagrantUpUbuntu1404 - Startup a vagrant VM running ubuntu-1404
vagrantUpUbuntu1504 - Startup a vagrant VM running ubuntu-1504

It does not make the "check" task depend on "checkPackages" so running the
vagrant tests is still optional. They are slow and depend on vagrant and
virtualbox.

The Package Verification tasks are useful for testing individual distros.

The Vagrant tasks are listed in `gradle tasks` primarily for discoverability.
2015-11-05 14:28:30 -05:00
Ryan Ernst f974d5e470 Build: Fail integ test if elasticsearch fails to start on windows 2015-11-05 00:07:59 -08:00
Ryan Ernst 2723bac296 Build: Add back support for tests.jvm.argline
closes #14541
2015-11-04 23:59:01 -08:00
Robert Muir 1a0a910d1e Merge pull request #14526 from rjernst/standalone_test
Add qa/evil-tests
2015-11-04 13:49:55 -05:00
Ryan Ernst a534ace896 Fix standalone test to work with check, and comment out mavenLocal 2015-11-04 10:28:31 -08:00
Daniel Mitterdorfer ae9fcd4f2b Merge remote-tracking branch 'danielmitterdorfer/feature/12596' 2015-11-04 09:31:12 +01:00
Daniel Mitterdorfer fea88bbc02 Add System#exit(), Runtime#exit() and Runtime#halt() to forbidden APIs
Closes #12596
2015-11-04 09:29:55 +01:00
Ryan Ernst fc1c66083b Fix integ test exec on windows 2015-11-03 23:02:29 -08:00
Ryan Ernst 95f885e11f Build: Change tar extension back to tar.gz instead of tgz
Gradle defaults to tgz extension when tar is compressed. This changes
the tar distribution back to tar.gz. Note that this also means the maven
packaging type is now tar.gz.
2015-11-03 11:24:27 -08:00
Ryan Ernst bd158e5e88 Add environment vars for JAVA_HOME and ES_GC_OPTS back to integ tests 2015-11-03 10:13:02 -08:00
Ryan Ernst 9246ca98fe rest test refactor 2015-11-03 10:12:16 -08:00
Ryan Ernst 3e4f87d4d6 Fix tar issues and build distro dependency for integ tests 2015-11-02 22:19:29 -08:00
Ryan Ernst f5556224f5 Build: Add back integ tests to distributions
rpm and deb are still skipped, but this configures rest tests to run for
zip and tgz distributions

closes #14361
2015-11-02 17:51:08 -08:00
Ryan Ernst 5bfecd4f76 Build: Simplify plugin properties generation
The gradle task to generate plugin properties files now is a simple copy
with expansion (ie maven filtering). It also no longer depends on
compiling.

closes #14450
2015-11-02 12:43:05 -08:00
Ryan Ernst 0f429f32de Merge pull request #14447 from rjernst/ide_plugin_props
Make idea/eclipse project generation build generated resources for plugins
2015-11-02 11:38:10 -08:00
Ryan Ernst b7f8e5c1df Build: Make idea/eclipse project generation build generated resources for plugins
This adds a generated-resources dir that the plugin properties are
generated into. This must be outside of the build dir, since intellij
has build as "excluded".

closes #14392
2015-11-02 11:35:02 -08:00
Robert Muir 73a136e9fc Use ${exception_var} for the case of nested generated try/catch 2015-11-02 11:58:34 -05:00
Robert Muir 1e00b85411 Fix auto-generated eclipse try/catch to be less trappy
When generating a try-catch block, the eclipse default is something like this:

```
try {
  something();
} catch (Exception e) {
  // TODO: auto-generated stub
  e.printStackTrace();
}

which is terrible, so the ES eclipse changes this to rethrow a RuntimeException instead.

```
try {
  something();
} catch (Exception e) {
  throw new RuntimeException();
}
```

Unfortunately, this loses the original exception entirely, instead it should be:

```
try {
  something();
} catch (Exception e) {
  throw new RuntimeException(e);
}
```
2015-11-02 11:45:55 -05:00
Robert Muir 951bc5a4a7 disable null pointer analysis so the IDE is not crashing. 2015-11-02 10:55:31 -05:00
Robert Muir 4587a94fcf generate complete eclipse configuration from 'gradle eclipse' 2015-11-02 10:44:51 -05:00
Ryan Ernst 0dca49fae4 fix eclipse config for qa projects 2015-10-30 21:32:16 -07:00
Ryan Ernst 197ed57ea4 Build: Move RR gradle plugin files to match external repo paths
The RR gradle plugin is at
https://github.com/randomizedtesting/gradle-randomized-testing-plugin.
However, we currently have a copy of this, since the plugin is still in
heavy development. This change moves the files around so they can be
copied directly from the elasticsearch fork to that repo, for ease of
syncing.
2015-10-30 13:22:59 -07:00
Ryan Ernst c1e3c4a78e Build: Add back -Dtests.output support
This change fixes the test logging for randomized testing to support
tests.output like maven/ant do.
2015-10-30 13:05:46 -07:00
Ryan Ernst 0a593109d3 Build: Add gradle version check and some build info that is always output 2015-10-29 17:12:45 -07:00
Ryan Ernst 9146b59fc1 Build: Move the real forbidden api signature files to their new location
within buildSrc

The gradle branch used copies of the forbidden api signature files. This
moves the files to their correct location.

closes #14363
2015-10-29 15:40:47 -07:00
Ryan Ernst 561436eba3 Build: Add back manifest info to jars
closes #14365
2015-10-29 13:59:52 -07:00
Ryan Ernst c86100f636 Switch build system to Gradle
See #13930
2015-10-29 11:40:19 -07:00