Commit Graph

356 Commits

Author SHA1 Message Date
Jason Tedor 8a05c2a2be Bootstrap does not set system properties
Today, certain bootstrap properties are set and read via system
properties. This action-at-distance way of managing these properties is
rather confusing, and completely unnecessary. But another problem exists
with setting these as system properties. Namely, these system properties
are interpreted as Elasticsearch settings, not all of which are
registered. This leads to Elasticsearch failing to startup if any of
these special properties are set. Instead, these properties should be
kept as local as possible, and passed around as method parameters where
needed. This eliminates the action-at-distance way of handling these
properties, and eliminates the need to register these non-setting
properties. This commit does exactly that.

Additionally, today we use the "-D" command line flag to set the
properties, but this is confusing because "-D" is a special flag to the
JVM for setting system properties. This creates confusion because some
"-D" properties should be passed via arguments to the JVM (so via
ES_JAVA_OPTS), and some should be passed as arguments to
Elasticsearch. This commit changes the "-D" flag for Elasticsearch
settings to "-E".
2016-03-13 20:09:15 -04:00
Simon Willnauer 121e7c8ca4 Add infrastructure to run REST tests on a multi-version cluster
This change adds the infrastructure to run the rest tests on a multi-node
cluster that users 2 different minor versions of elasticsearch. It doesn't implement
any dedicated BWC tests but rather leverages the existing REST tests.

Since we don't have a real version to test against, the tests uses the current version
until the first minor / RC is released to ensure the infrastructure works.

Relates to #14406
Closes #17072
2016-03-13 10:52:39 +01:00
Simon Willnauer e91245e25f Use a seed node to form multi-node cluster in integ tests
Today we use hardcoded ports to form a cluster in the mulit-node case.
The hardcoded URIs are passed to the unicast host list which is error prone and
might cause problems if those ports are exhausted etc. This commit moves to a
less error prone way of forming the cluster where all nodes are started with port `0`
and all but the first node wait for the first node to write it's ports file to form a
cluster. This seed node is enough to form a cluster.
2016-03-11 19:20:37 +01:00
Yannick Welsch b17f4b40ba Check that number of placeholders in log message matches number of parameters 2016-03-11 10:30:59 +01:00
Ali Beyad 06487b0ac5 Change internal representation of suggesters to instances of
SuggestBuilder instead of raw bytes.
2016-03-10 00:12:39 -05:00
Christoph Büscher 1264f37a1b Merge branch 'master' into feature-suggest-refactoring 2016-03-08 11:10:08 +01:00
Simon Willnauer 9cbc602487 Remove SNAPSHOT from versions in plugin descriptors
We removed leniencey from version parsing which caught problems with
-SNAPSHOT suffixes on plugin properies. This commit removes the -SNAPSHOT
from both es and the extension version and adds tests to ensure we can
parse older versions that allowed -SNAPSHOT in BWC way.
2016-03-07 17:41:38 +01:00
Christoph Büscher 5ff413074a Adding tests for `time_zone` parameter for date range aggregation 2016-03-07 15:38:24 +01:00
Robert Muir 54018a5d37 upgrade to lucene 6.0.0-snapshot-bea235f
Closes #16964

Squashed commit of the following:

commit a23f9d2d29220991aa498214530753d7a5a148c6
Merge: eec9c4e 0b0a251
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Mar 7 04:12:02 2016 -0500

    Merge branch 'master' into lucene6

commit eec9c4e5cd11e9c3e0b426f04894bb2a6dae4f21
Merge: bc67205 675d940
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 13:45:00 2016 -0500

    Merge branch 'master' into lucene6

commit bc67205bdfe1526eae277ab7856fc050ecbdb7b2
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 09:56:31 2016 -0500

    fix test bug

commit a60723b007ff12d97b1810cef473bd7b553a0327
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Mar 4 15:35:35 2016 +0100

    Fix SimpleValidateQueryIT to put braces around boosted terms

commit ae3a49d7ba7ced448d2a5262e5d8ec98671a9090
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Mar 4 15:27:25 2016 +0100

    fix multimatchquery

commit ae23fdb88a8f6d3fb7ba60fd1aaf3fd72d899aa5
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Mar 4 15:20:49 2016 +0100

    Rewrite DecayFunctionScoreIT to be independent of the similarity used

    This test relied a lot on the term scoring and compared scores
    that are dependent on the similarity. This commit changes the base query
    to be a predictable constant score query.

commit 366c2d518c35d31251033f1b6f6a93f6e2ae327d
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Mar 4 14:06:14 2016 +0100

    Fix scoring in tests due to changes to idf calculation.

    Lucene 6 uses a different default similarity as well as a different
    way to calculate IDF. In contrast to older version lucene 6 uses docCount per field
    to calculate the IDF not the # of docs in the index to overcome the sparse field
    cases.

commit dac99fd64ac2fa71b8d8d106fe68825e574c49f8
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 08:21:57 2016 -0500

    don't hardcoded expected termquery score

commit 6e9f340ba49ab10eed512df86d52a121aa775b0f
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 08:04:45 2016 -0500

    suppress deprecation warning until migrated to points

commit 3ac8908424b3fdad44a90a4f7bdb3eff7efd077d
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 07:21:43 2016 -0500

    Remove invalid test: all commits have IDs, and its illegal to do this.

commit c12976288124ad1a26467e7e848fb810548e7eab
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 07:06:14 2016 -0500

    don't test with unsupported back compat

commit 18bbfe76128570bc70883bf91ff4c44c82d27817
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 07:02:18 2016 -0500

    remove now invalid lucene 4 backcompat test

commit 7e730e572886f0ef2d3faba712e4256216ff01ec
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 06:58:52 2016 -0500

    remove now invalid lucene 4 backwards test

commit 244d2ab6868ba5ac9e0bcde3c2833743751a25ec
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 06:47:23 2016 -0500

    use 6.0 codec

commit 5f64d4a431a6fdaa1234adca23f154c2a1de8284
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 06:43:08 2016 -0500

    compile, javadocs, forbidden-apis, etc

commit 1f273cd62a7fe9ca8f8944acbbfc5cbdd3d81ccb
Merge: cd33921 29e3443
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Mar 4 10:45:29 2016 +0100

    Merge branch 'master' into lucene6

commit cd33921ac742ef9fb351012eff35f3c7dbda7264
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:58:37 2016 -0500

    fix hunspell dictionary loading

commit c7fdbd837b01f7defe9cb1c24e2ec65604b0dc96
Merge: 4d4190f d8948ba
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:41:53 2016 -0500

    Merge branch 'master' into lucene6

commit 4d4190fd82601aaafac6b8254ccb3edf218faa34
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:39:14 2016 -0500

    remove nocommit

commit 77ca69e288b1a41aa9595c921ed166c272a00ea8
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:38:24 2016 -0500

    clean up numericutils vs legacynumericutils

commit a466d696fbaad04b647ffbc0857a9439b583d0bf
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:32:43 2016 -0500

    upgrade spatial4j

commit 5412c747a8cfe638bacedbc8233163cb75cc3dc5
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:19:28 2016 -0500

    move to 6.0.0-snapshot-8eada27

commit b32bfe924626b87e540692375ece09e7c2edb189
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 11:30:09 2016 +0100

    Fix some test compile errors.

commit 6ccde35e9840b03c68d1a2cd47c7923a06edf64a
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 11:25:51 2016 +0100

    Current Lucene version is 6.0.0.

commit f62e1015d931b4cc04c778298a8fa1ba65e97ad9
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 11:20:48 2016 +0100

    Fix compile errors in NGramTokenFilterFactory.

commit 6837c6eabf96075f743649da9b9b52dd39611c58
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 10:50:59 2016 +0100

    Fix the edge ngram tokenizer/filter.

commit ccd7f070de5efcdfbeb34b9555c65c4990bf1ba6
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 10:42:44 2016 +0100

    The missing value is now accessible through a getter.

commit bd3b77f9b28e5b05daa3d49683a9922a6baf2963
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 10:41:51 2016 +0100

    Remove IndexCacheableQuery.

commit 05f3091c347aeae80eeb16349ac51d2b53cf86f7
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 10:39:43 2016 +0100

    Fix compilation of function_score queries.

commit 81cda79a2431ac78f56b0cc5a5765387f662d801
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 10:35:02 2016 +0100

    Fix compile errors in BlendedTermQuery.

commit 70994ce8dd1eca0b995870974a38e20f26f96a7b
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 23:33:03 2016 -0500

    add bug ID

commit 29d4f1a71f36f646b5a6060bed3db019564a279d
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 21:02:32 2016 -0500

    easy .store changes

commit 5e1a1e6fd665fa455e88d3a8987362fad5f44bb1
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 20:47:24 2016 -0500

    cleanups mostly around boosting

commit 333a669ec6c305ada5645d13ed1da0e19ec1d053
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 20:27:56 2016 -0500

    more simple fixes

commit bd5cd98a1e089c866b6b4a5e159400b110140ce6
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 19:49:38 2016 -0500

    more easy fixes and removal of ancient cruft

commit a68f419ee47da5f9c9ce5b372f01d707e902474c
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 19:35:02 2016 -0500

    cutover numerics

commit 4ca5dc1fa47dd5892db00899032133318fff3116
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 18:34:18 2016 -0500

    fix some constants

commit 88710a17817086e477c6c021ec346d0534b7fb88
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 18:14:25 2016 -0500

    Add spatial-extras jar as a core dependency

commit c8cd6726583e5ce3f546ed355d4eca037164a30d
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 18:03:33 2016 -0500

    update to lucene 6 jars
2016-03-07 04:12:23 -05:00
Christoph Büscher c0e5e40e02 Merge branch 'master' into feature-suggest-refactoring 2016-03-02 14:22:03 +01:00
Jason Tedor aa8ee74c6c Bump Elasticsearch version to 5.0.0-SNAPSHOT
This commit bumps the Elasticsearch version to 5.0.0-SNAPSHOT in line
with the alignment of versions across the stack.

Closes #16862
2016-03-01 17:03:47 -05:00
Christoph Büscher aecf51cb42 Merge branch 'master' into feature-suggest-refactoring
Conflicts:
	core/src/main/java/org/elasticsearch/rest/action/search/RestSearchAction.java
2016-03-01 17:20:15 +01:00
Christoph Büscher a7053afdb9 Merge branch 'master' into feature-suggest-refactoring
Conflicts:
	docs/reference/migration/migrate_5_0.asciidoc
2016-03-01 12:37:05 +01:00
javanna b7597d7aea Rename SearchServiceTransportAction to SearchTransportService
The suffix TransportAction is misleading as it may make think that it extends TransportAction, but it does not. This class makes accessible the different search operations exposed by SearchService through the transport layer. Also resolved few compiler warnings in the class itself.
2016-03-01 11:35:04 +01:00
Nik Everett 95cc3e38fc Check test naming conventions on all modules
The big win here is catching tests that are incorrectly named and will
be skipped by gradle, providing a false sense of security.

The whole thing takes about 10 seconds on my Macbook Air, not counting
compiling the test classes, which seems worth it. Because this runs as
a gradle task with propery UP-TO-DATE handling it can be skipped if the
tests haven't been changed which should save some time.

I chose to keep this in test:framework rather than a new subproject of
buildSrc because ESIntegTestCase and doesn't inroduce any additional
dependencies.
2016-02-29 16:31:49 -05:00
javanna 9738ef3a90 Cleanup search sub transport actions and collapse o.e.action.search.type package into o.e.action.search
TransportSearchTypeAction and subclasses are not actually transport actions, but just support classes useful for their inner async actions that can easily be extracted out so that we get rid of one too many level of abstraction.

Same pattern can be applied to TransportSearchScrollQueryAndFetchAction & TransportSearchScrollQueryThenFetchAction which we could remove in favour of keeping only their inner classes named SearchScrollQueryAndFetchAsyncAction and SearchScrollQueryThenFetchAsyncAction.

Remove org.elasticsearch.action.search.type package, collapsed remaining classes into existing org.elasticsearch.action.search package

Make also ParsedScrollId ScrollIdForNode and TransportSearchHelper classes and their methods package private.

Closes #11710
2016-02-29 14:11:18 +01:00
Ali Beyad a7f6488216 Merge remote-tracking branch 'upstream/master' into
feature-suggest-refactoring
2016-02-26 17:21:12 -05:00
Nik Everett ba5be0332d Remove optional logger wrappers
Removes all our logger wrappers except the wrapper for log4j1.2. If you
depend on Elasticsearch's jar in your application you'll need to declare
log4j 1.2 and/or some bridge to your favorite logger.

We did this to simplify our builds and code. No more commons-logging like
log implementation sniffing. No more optional dependency hacks in gradle.

We might one day want to use j.u.l instead of log4j. If we do want that
we can recover its wrapper by studying this commit. We didn't go directly
to j.u.l in this commit because that is a bigger change. Our logging
configuration is based on log4j1.2 and people are used to it. So it'd
be a much more fraught breaking change to do that conversion.
2016-02-26 16:41:07 -05:00
Lee Hinman 5d359f737f Merge branch 'pr/16801' 2016-02-26 13:22:14 -07:00
James Bertouch a249225993 Update to Jackson 2.7.1
https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.7.1
2016-02-25 16:15:03 -05:00
Ryan Ernst 80fc0496bb Merge pull request #16782 from rjernst/fix/extra_config_files
Build: Fix extra config files to work with more than one file
2016-02-25 11:28:39 -08:00
Nik Everett 7c4420bf3d Merge pull request #16807 from nik9000/not_mvn
Remove mention of mvn from java source files
2016-02-25 11:05:46 -05:00
Nik Everett 284f41f7f5 Rename tests.maven to tests.gradle 2016-02-25 10:26:12 -05:00
jaymode c9c4cac0ac Testing: workaround HttpClient issue with IPv6 hostname verification
This commit works around an issue with hostname verification in HttpClient when using IPv6
addresses in URLs. When an IPv6 address is used in a URL it is typically wrapped with square
brackets. The hostname verifier for HttpClient does not recognize these as valid IPv6 addresses
and instead treats them as a DNS name. We wrap the strict hostname verifier for this version
of HttpClient and strip brackets if we need to.

The corresponding issue in HttpClient is https://issues.apache.org/jira/browse/HTTPCLIENT-1698
but the fix has not been released yet in a stable version.
2016-02-25 08:04:41 -05:00
James Bertouch cb35455745 Upgrade to Jackson 2.7.0
Closes #16294
2016-02-24 18:06:38 -05:00
Ryan Ernst 3772350031 Build: Fix extra config files to work with more than one file
The extra config copy task for integ tests was previously overwriting
the source and destination for each file to be copied, leaving only the
last. This fixes it to create sub copy specs (which is what was
originally intended) for each file.
2016-02-23 07:54:40 -08:00
Christoph Büscher d888b5c267 Merge branch 'master' into feature-suggest-refactoring 2016-02-22 11:22:28 -08:00
Jack Conradson 7986770e5f Moved Painless from a plugin to a module.
Closes #16755
2016-02-21 16:50:54 -08:00
Mike McCandless 2e3160824c Upgrade to Lucene 5.5.0 official release
Closes #16742
2016-02-21 05:26:22 -05:00
Simon Willnauer c8e462e5dc Apply feedback from @rjernst 2016-02-20 17:12:48 -08:00
Simon Willnauer 4f7dd1284f Remove SNAPSHOT notion from code
Today we have the notion of a snapshot inside Version.java which makes
releasing complicated since to do a release Version.java must be changed.
This commit removes all notions of snapshot from the code and allows to
switch between snapshot and release build by specifying a system property on
the build. For instance running:

```
gradle run -Dbuild.snapshot=false
```

will build and package a release build while the default always
builds snapshots. Calls to the main rest action will still get the snapshot
information rendered out with the response.
2016-02-20 16:06:26 -08:00
Mike McCandless 5fffede2b0 Upgrade to Lucene 5.5.0 official release 2016-02-20 17:34:16 -05:00
Ali Beyad beea413820 Merge remote-tracking branch 'upstream/master' into feature-suggest-refactoring 2016-02-12 11:54:44 -05:00
Simon Willnauer b5aee2075f Merge pull request #16610 from s1monw/test_indices_request_cache
Refactor IndicesRequestCache to make it testable.
2016-02-12 09:27:20 +01:00
Boaz Leskes 8bc2332d9a Decouple recovery source/target logic and transport piping
The current logic for doing recovery from a source to a target shourd is tightly coupled with the underlying network pipes. This changes decouple the two, making it easier to add unit tests for shard recovery that doesn't involve the node and network environment.

On top that, RecoveryTarget is renamed to RecoveryTargetService leaving space to renaming RecoveryStatus to RecoveryTarget (and thus avoid the confusion we have today with RecoveryState).

Correspondingly RecoverySource is renamed to RecoverySourceService.

Closes #16605
2016-02-11 21:36:48 +01:00
Nicholas Knize 52ee4c7027 upgrade to lucene 5.5.0-snapshot-850c6c2 2016-02-11 14:28:50 -06:00
Simon Willnauer d538dd64c2 Refactor IndicesRequestCache to make it testable.
This commit moves IndicesRequestCache into o.e.indics and makes all API in this
class package private. All references to SearchReqeust, SearchContext etc. have been factored
out and relevant glue code has been added to IndicesService. The IndicesRequestCache is not a
simple class without any hard dependencies on ThreadPool nor SearchService or IndexShard. This now
allows to add unittests.
This commit also removes two settings `indices.requests.cache.clean_interval` and `indices.fielddata.cache.clean_interval`
in favor of `indices.cache.clean_interval` which cleans both caches.
2016-02-11 16:51:44 +01:00
Christoph Büscher 421ed1228b Merge branch 'master' into feature-suggest-refactoring
Conflicts:
	core/src/main/java/org/elasticsearch/search/suggest/phrase/PhraseSuggestionBuilder.java
2016-02-10 11:37:51 +01:00
Jim Ferenczi b146f3ecb3 Pack all the plugin files into a single folder named elasticsearch at the root of the plugin zip. 2016-02-10 10:13:05 +01:00
Nik Everett f76366909a Fail build if we define equals but not hashCode
That is like some kind of cardinal sin or something, right?

We had two violations though they weren't super likely to be keys in a hashmap
any time soon.
2016-02-09 21:29:08 -05:00
Jason Tedor 75da71ff0b Forbid use of java.security.MessageDigest#clone()
This commit forbids the usage of java.security.MessageDigest#clone() in
favor of getting a thread local instance using
org.elasticsearch.common.hash.MessageDigests. This is to prevent running
into java.lang.CloneNotSupportedExceptions for message digest providers
that do not support clone.

Closes #16543
2016-02-09 11:35:47 -05:00
Alexander Reelsen 0d4711c2fc Ingest: Add attachment processor
This is a simple port of the mapper attachment plugin to the ingest
functionality, no new features. The only option is to limit
the number of chars to prevent indexing of huge documents.

Fields can be selected in the processor as well.

Close #16303
2016-02-09 17:03:30 +01:00
Christoph Büscher 2ae6420fd6 Suggest: Add parsing from xContent to PhraseSuggestionBuilder
For the ongoing search refactoring (#10217) the PhraseSuggestionBuilder
gets a way of parsing from xContent that will eventually replace the
current SuggestParseElement. This PR adds the fromXContent method
to the PhraseSuggestionBuilder and also adds parsing code for the
common suggestion parameters to SuggestionBuilder.

Also adding links from the Suggester implementations registeres in the
Suggesters registry to the corresponding prototype that is going to
be used for parsing once the refactoring is done and we switch from
parsing on shard to parsing on coordinating node.
2016-02-08 12:28:12 +01:00
Christoph Büscher 97172aa2fd Merge pull request #16465 from cbuescher/issue-13884-test
Tests: Add test for minimum_should_match for SimpleQueryStringQuery
2016-02-05 13:12:53 +01:00
Christoph Büscher d885fe62ea Add test for minimum_should_match, one term and multiple fields
This adds a test case similar to the issue in #13884 which was
fixed in #16155.
2016-02-05 12:33:13 +01:00
Jim Ferenczi 5d23d9f034 Merge pull request #16454 from jimferenczi/rename_plugin_script
Rename bin/plugin in bin/elasticsearch-plugin
2016-02-05 11:25:00 +01:00
Yannick Welsch 89a5eadfea Change path separator for Checkstyle suppressions to be Windows compatible 2016-02-05 10:18:18 +01:00
Jim Ferenczi 7d0181b5d4 Rename bin/plugin in bin/elasticsearch-plugin 2016-02-05 10:09:14 +01:00
Ryan Ernst b9342b305b Also use separte buildSrc build dir for intellij 2016-02-04 15:58:05 -08:00
Ryan Ernst 2fef5dee96 Build: use separate build directories for eclipse and intellij
This fixes the problem where doing a gradle clean from the command line
completely breaks the IDE.
2016-02-04 15:44:48 -08:00
Nik Everett 7a2947a4ee Suppress LineLength in more files
These are files that grew long lines between when the last line length
suppression patch was proposed about a day ago.
2016-02-04 15:39:25 -05:00
Nik Everett 63aedb8720 Fail build for lines longer than 140 characters
140 characters is our official line length from CONTRIBUTING.md. It is a
little longer than fits well in github but you can use a userstyle to fix
that. It is perfect for Eclipse and unified diffs but too wide for side by
side diffs. Regardless, its the official limit we've had for years. We just
haven't enforced it automatically and we haven't enforced it using github
because line limits are hard to notice there.

This only hits about 2/3 of the java files - those that didn't have failures
already. If they did have a failure it suppresses them. We should pick files
off that list as time goes on.

For posterity I generated the suppressions by running checkstyle with
ignoreFailures = true, piping the output to a file, and then running it
through this:
perl -ne 'print if s{.*\[ant:checkstyle\] /.+/elasticsearch/}{  <suppress files="} && s{\.java.+}{\.java" checks="LineLength" />}' | uniq
2016-02-04 15:02:39 -05:00
Jason Tedor 6707a89c3b Fail early on JDK with compiler bug
Early versions of JDK 8 have a compiler bug that prevents assignment to
a definitely unassigned final variable inside the body of a lambda. This
commit adds an early-out to the build process which also gives a useful
error message.

Closes #16418
2016-02-03 13:25:39 -05:00
Robert Muir 20b584c7e1 Merge pull request #16400 from rmuir/lucene_4de5f1d
Upgrade to lucene 5.5.0-snapshot-4de5f1d
2016-02-03 06:45:50 -05:00
Robert Muir d5dc05f69e Upgrade to lucene 5.5.0-snapshot-1725675 2016-02-02 22:53:39 -05:00
Nik Everett 900133f1b3 Suppress checkstyle on generated files in painless
Closes #16387
2016-02-02 18:50:39 -05:00
Jason Tedor 105411060c Uppercase ells ('L') in long literals
This commit removes and forbids the use of lowercase ells ('l') in long
literals because they are often hard to distinguish from the digit
representing one ('1').

Closes #16329
2016-01-30 22:16:02 -05:00
Nik Everett e6c022b520 Mimimal checkstyle configuration
This adds a small pile of checkstyle checks that all pass without any changes.
It moves some checks from ForbiddenPatterns that were java only into
checkstyle.

Don't duplicate forbiddenPatterns in checkstyle
2016-01-30 14:59:50 -05:00
Simon Willnauer 9b3559f473 Remove `test.cluster.node.seed` and special case `tests.portsfile`
* `test.cluster.node.seed` was only used in one place where it wasn't adding value and was now replaced with a constant
 * `tests.portsfile` is now an official setting and has been renamed to `node.portsfile`

this commit also convert `search.default_keep_alive` and `search.keep_alive_interval` to the new settings infrastrucutre
2016-01-22 15:47:32 +01:00
Ryan Ernst df24019261 Merge pull request #16038 from rjernst/remove_site_plugin
Plugins: Remove site plugins
2016-01-21 12:32:22 -08:00
Robert Muir 6e7e3a2274 Update lucene to r1725675
Adds DFI (divergence from independence) provider.
Fixes test bugs passing invalid values for BM25 parameters.
2016-01-20 03:32:51 -05:00
Robert Muir 79cc3c2dd0 set -Djna.nosys when running tests.
This is consistent with what happens in elasticsearch.sh/.bat, and it means
tests will work even if there is a crazy "system" JNA installed on the machine.
2016-01-19 18:49:47 -05:00
Nik Everett 2cb7e8ce76 Use our standard xlint with standalone-test
We were not changing the xlint settings there at all. Also cleans up some
generic array warnings that this found by switching them to an ArrayList.
2016-01-19 17:54:47 -05:00
Ryan Ernst 8bf8ca6222 Fix rest tests to use ip and port again instead of localhost and port. 2016-01-18 19:28:41 -08:00
Ryan Ernst d7537075d6 Add TODO for splitting rest tests from client tests. 2016-01-18 16:59:42 -08:00
Ryan Ernst ef4f0a8699 Test: Make rest test framework accept http directly for the test cluster
The rest test framework, because it used to be tightly integrated with
ESIntegTestCase, currently expects the addresses for the test cluster to
be passed using the transport protocol port. However, it only uses this
to then find the http address.

This change makes ESRestTestCase extend from ESTestCase instead of
ESIntegTestCase, and changes the sysprop used to tests.rest.cluster,
which now takes the http address.

closes #15459
2016-01-18 16:44:14 -08:00
Ryan Ernst 3b78267c71 Plugins: Remove site plugins
Site plugins used to be used for things like kibana and marvel, but
there is no longer a need since kibana (and marvel as a kibana plugin)
uses node.js. This change removes site plugins, as well as the flag for
jvm plugins. Now all plugins are jvm plugins.
2016-01-16 22:45:37 -08:00
Daniel Mitterdorfer 871b38afcb Check cluster health in integration test wait condition
With this commit we do not check only if an endpoint is up but
we also check that the cluster status is green. Previously,
builds sporadically failed to pass this condition.
2016-01-15 09:14:32 +01:00
Daniel Mitterdorfer dc51dd0056 Add more logging to wait condition for elasticsearch nodes in tests 2016-01-14 09:48:27 +01:00
Ryan Ernst cde5bb88b1 Add printing timestamp to wait condition for elasticsearch nodes 2016-01-12 11:15:50 -08:00
Nik Everett 01ce49e94e Ban Serializable
1. Uses forbidden patterns to prevent things from referencing
java.io.Serializable or from mentioning serialVersionUID.
2. Uses -Xlint:-serial so we don't have to hear from javac that we aren't
declaring serialVersionUID on any classes that we make that happen to extend
Serializable.
3. Remove Serializable and serialVersionUID declarations.

I didn't use forbidden apis because it doesn't look like it has a way to ban
explicitly implementing Serializable. If you try to ban Serializable with
forbidden apis you end up banning all Exceptions and all Strings.

Closes #15847
2016-01-11 16:57:31 -05:00
Jason Tedor abaf816d00 Remove and forbid use of IndexWriter#isLocked
This commit removes and now forbids use of
org.apache.lucene.index.IndexWriter#isLocked as this method was
deprecated in LUCENE-6508. The deprecation is due to the fact that
checking if a lock is held before acquiring that lock is subject to a
time-of-check-to-time-of-use race condition. There were three uses of
IndexWriter#isLocked in the code base:
 - a logging statement in o.e.i.e.InternalEngine where we are already in
   an exceptional condition that the lock was held; in this case,
   logging whether or not the directory is locked is superfluous
 - in o.e.c.l.u.VersionsTests where we were verifying that a write lock
   is released upon closing an IndexWriter; in this case, the check is
   not needed as successfully closing an IndexWriter releases its
   write lock
 - in o.e.t.s.MockFSDirectoryService where we were verifying that a
   directory is not write-locked before (implicitly) trying to obtain
   such a write lock in org.apache.lucene.index.CheckIndex#<init> (this
   is the exact type of a situation that is subject to a race
   condition); in this case we can proceed by just (implicitly) trying
   to obtain the write lock and failing if we encounter a
   LockObtainFailedException
2016-01-10 08:28:44 -05:00
Jason Tedor 871d1b4885 Remove and forbid use of j.u.c.ThreadLocalRandom
This commit removes and now forbids all uses of
java.util.concurrent.ThreadLocalRandom across the codebase. The
underlying issue with ThreadLocalRandom is that it can not be
seeded. This means that if ThreadLocalRandom is used in production code,
then tests that cover any code path containing ThreadLocalRandom will be
prevented from being reproducible by use of ThreadLocalRandom. Instead,
using org.elasticsearch.common.random.Randomness#get will give
reproducible sources of random when running under tests and otherwise
still give an instance of ThreadLocalRandom when running as production
code.
2016-01-08 12:23:48 -05:00
Yannick Welsch d5191518ba Fix thirdPartyAudit check on Windows in case of jar hell with JDK 2015-12-31 12:10:48 +01:00
Robert Muir 180ab2493e Improve thirdPartyAudit check, round 3 2015-12-28 22:38:55 -05:00
Robert Muir 010d1a89c5 Merge branch 'master' into hdfs2-only 2015-12-22 00:40:54 -05:00
Adrien Grand cf52e96c42 Upgrade to lucene-5.5.0-snapshot-1721183.
Some files that implement or use the Scorer API had to be changed because of
https://issues.apache.org/jira/browse/LUCENE-6919.
2015-12-21 17:02:08 +01:00
Robert Muir f4f8b6e3fe Merge branch 'master' of github.com:elastic/elasticsearch into hdfs2-only 2015-12-20 21:59:02 -05:00
Ryan Ernst a518599e27 Allow plugins to upgrade slf4j-api to a compile dep
This was originally intended to be general purpose in #15555, but
that still had problems. Instead, this change fixes the issue explicitly
for slf4j-api, since that is the problematic dep that is not actually
included in the distributions.
2015-12-20 17:59:00 -08:00
Robert Muir 12a8428dfb Add MiniHDFS test fixture, started before integTest and shut down after.
Currently uses a hardcoded port (9999), need to apply MavenFilteringHack after it starts.
2015-12-20 16:00:37 -05:00
Ryan Ernst 9cb4c82c58 Build: Add fixture capabilities to integ tests
This change adds a Fixture class for use by gradle. A Fixture is an
external process that integration tests will use. It can be added as a
dependsOn for integTest, and will automatically be shutdown upon success
or failure, as well as relevant information dumped on failure. There is
also an example fixture in this change.
2015-12-19 15:46:21 -08:00
Ryan Ernst ad46a09cf5 Revert "Build: Allow plugins to "upgrade" provided deps"
This reverts commit 583ba76d0b.
2015-12-19 01:22:12 -08:00
Ryan Ernst 583ba76d0b Build: Allow plugins to "upgrade" provided deps
This only really applies to slf4j, since the others are in the
distribution, but it is necessary if a plugin wants to depend on slf4j.
2015-12-19 00:06:29 -08:00
Ryan Ernst af11707da0 Remove unused method impl in AntTask and make abstract 2015-12-18 14:37:15 -08:00
Ryan Ernst beec7ca9db Merge branch 'master' into wildcard_imports 2015-12-18 13:18:08 -08:00
Ryan Ernst d0a10b337e Fix silly typo in variable name... 2015-12-18 13:17:05 -08:00
Ryan Ernst a639b3d3e4 Remove wildcard imports for groovy too 2015-12-18 12:47:35 -08:00
Ryan Ernst 4ea19995cf Remove wildcard imports 2015-12-18 12:43:47 -08:00
Ryan Ernst 853e9c0fd1 Merge branch 'master' into wildcard_imports 2015-12-18 12:24:30 -08:00
Ryan Ernst 44edac0081 use better variable name for build listener, and change access back to
private for internal method of thirdPartyAudit
2015-12-18 12:07:30 -08:00
Ryan Ernst 5b9bf8e738 Make a new ant builder per AntTask invocation 2015-12-18 12:01:54 -08:00
Ryan Ernst 9f1dfdbaea Build: Add AntTask to simplify controlling logging when running ant from gradle
This new task allows setting code, similar to a doLast or doFirst,
except it is specifically geared at running ant (and thus called doAnt).
It adjusts the ant logging while running the ant so that the log
level/behavior can be tweaked, and automatically buffers based on gradle
logging level, and dumps the ant output upon failure.
2015-12-18 11:45:52 -08:00
Robert Muir 2ce54640f5 Remove unnecessary license categories/matchers 2015-12-18 13:46:59 -05:00
Robert Muir c4f823903a fix indent 2015-12-18 12:00:52 -05:00
Robert Muir e82808917c don't fail on missing source dirs 2015-12-18 11:58:18 -05:00
Robert Muir 94d6b22137 add gradle licenseHeaders to precommit
This is a port of the logic from apache lucene that uses Rat
2015-12-18 11:43:58 -05:00
Ryan Ernst fbff877ec3 Fix thirdpartyaudit to ignore all elasticsearch packages 2015-12-17 22:31:40 -08:00
Ryan Ernst 6a99796b02 Build: Move test framework under a "test" top level dir
This allows adding more test projects, eg integ test fixtures that will
be coming soon.
2015-12-17 21:25:06 -08:00
Robert Muir 6692e42d9a thirdPartyAudit round 2
This fixes the `lenient` parameter to be `missingClasses`. I will remove this boolean and we can handle them via the normal whitelist.
It also adds a check for sheisty classes (jar hell with the jdk).
This is inspired by the lucene "sheisty" classes check, but it has false positives. This check is more evil, it validates every class file against the extension classloader as a resource, to see if it exists there. If so: jar hell.

This jar hell is a problem for several reasons:

1. causes insanely-hard-to-debug problems (like bugs in forbidden-apis)
2. hides problems (like internal api access)
3. the code you think is executing, is not really executing
4. security permissions are not what you think they are
5. brings in unnecessary dependencies
6. its jar hell

The more difficult problems are stuff like jython, where these classes are simply 'uberjared' directly in, so you cant just fix them by removing a bogus dependency. And there is a legit reason for them to do that, they want to support java 1.4.
2015-12-17 02:35:00 -05:00
Robert Muir 42138007db add some more comments about internal api usage 2015-12-16 18:56:02 -05:00
Robert Muir 49f37a526c fix bad indent 2015-12-16 16:50:52 -05:00
Robert Muir ee79d46583 Add gradle thirdPartyAudit to precommit tasks 2015-12-16 16:38:16 -05:00
Ryan Ernst 5c8a0da1fd Build: Change rest integ tests to not have hardcoded ports
This change removes hardcoded ports from cluster formation. It passes
port 0 for http and transport, and then uses a special property to have
the node log the ports used for http and transport (just for tests).
This does not yet work for multi node tests. This brings us one step
closer to working with --parallel.
2015-12-11 17:36:08 -08:00
Ryan Ernst 47e922ba85 Add excludes and patterns as explicit inputs to forbidden patterns task 2015-12-11 08:26:38 -08:00
Jason Tedor 3383c24be0 Remove and forbid use of Collections#shuffle(List) and Random#<init>()
This commit removes and now forbids all uses of
Collections#shuffle(List) and Random#<init>() across the codebase. The
rationale for removing and forbidding these methods is to increase test
reproducibility. As these methods use non-reproducible seeds, production
code and tests that rely on these methods contribute to
non-reproducbility of tests.

Instead of Collections#shuffle(List) the method
Collections#shuffle(List, Random) can be used. All that is required then
is a reproducible source of randomness. Consequently, the utility class
Randomness has been added to assist in creating reproducible sources of
randomness.

Instead of Random#<init>(), Random#<init>(long) with a reproducible seed
or the aforementioned Randomess class can be used.

Closes #15287
2015-12-11 11:16:38 -05:00
Robert Muir bc140659a0 fail build on wildcard imports
Wildcard imports are terrible, they cause ambiguity in the code,
make it not compile with the future versions of java in many cases.

We should simply fail the build on this, it is messiness, caused by
messy Intellij configuration
2015-12-11 11:03:44 -05:00
Robert Muir 2741888498 Remove RuntimePermission("accessDeclaredMembers")
Upgrades lucene to 5.5.0-1719088, randomizedtesting to 2.3.2, and securemock to 1.2
2015-12-10 14:26:55 -05:00
Robert Muir e454fadc22 Merge branch 'master' into shave_mustache 2015-12-10 07:58:24 -05:00
Robert Muir b6fbd18e09 Merge pull request #15322 from rmuir/awesome
make slow tests more obvious
2015-12-09 15:05:53 -05:00
Ryan Ernst 46d73a13ff Merge branch 'master' into shave_mustache 2015-12-08 21:11:48 -08:00
Ryan Ernst 18698b9308 Build: Add build plugin for "messy" tests
We have some tests which have crazy dependencies, like on other plugins.
This change adds a "messy-test" gradle plugin which can be used for qa
projects that these types of tests can run in. What this adds over
regular standalone tests is the plugin properties and metadata on the
classpath, so that the plugins are properly initialized.
2015-12-08 19:48:39 -08:00
Robert Muir 2e299860e5 refactor mustache to lang-mustache plugin. No rest tests yet. 2015-12-08 22:45:21 -05:00
Robert Muir b0febc1b9f make slow tests more obvious 2015-12-08 13:05:15 -05:00
Ryan Ernst fa1c708ccd Merge branch 'master' into standalone_tests_intellij 2015-12-07 16:01:32 -08:00
Ryan Ernst 2dbd93d3b0 Tweaked module error message against non integ-test-zip and made checks
for limiting configuration to certain distributions prettier
2015-12-04 11:40:08 -08:00
Ryan Ernst 01d48e2062 Merge branch 'master' into jigsaw 2015-12-04 11:29:49 -08:00
Ryan Ernst a6752c4da9 Merge pull request #15210 from rjernst/buildsrc_test_cleanup
Cleanup integ test classes in buildSrc to be less groovyish
2015-12-04 11:24:21 -08:00
Ryan Ernst 4b1fc8ff22 Merge pull request #15226 from rjernst/fix_update_shas
Fix updateShas to not barf on disabled license checks and even compile correctly
2015-12-03 23:34:33 -08:00
Ryan Ernst f5b6b40a92 Fix distros to put modules dir int he right place and qa rest tests to
use the real zip
2015-12-03 23:03:14 -08:00
Ryan Ernst a8e9403204 added gradle checks for modules configuration, and ability to add
modules to integ test cluster
2015-12-03 20:53:06 -08:00
Ryan Ernst f2cebd9b3b Merge branch 'master' into jigsaw 2015-12-03 20:02:44 -08:00
Ryan Ernst 0be141e021 Build: Add .settings for buildSrc on gradle eclipse
We have eclipse settings added to all projects when running gradle
eclipse, but buildSrc is its own special project that is not
encapsulated by allprojects blocks. This adds eclipse settings to
buildSrc.
2015-12-03 19:56:11 -08:00
Ryan Ernst 17e6195fd6 Build: Fix updateShas to not barf on disabled license checks and even compile correctly
These were just results of not testing properly after refactoring.

closes #15224
2015-12-03 15:21:14 -08:00
Ryan Ernst caf77f7eea Add integ-test-zip distribution 2015-12-03 14:52:51 -08:00
Ryan Ernst b00c59398b Build: Cleanup test classes in buildSrc to be less groovyish 2015-12-03 01:49:54 -08:00
Robert Muir 1f626d3458 forbidden third-party-signatures -> core-signatures
This is a relic from shading where it was trickier to implement.
Third party signatures are already in e.g. the test list, there
is no reason to separate them out.

Instead, we could have a third party signatures that does
something different... like keep tabs on third party libraries.
2015-12-03 01:45:06 -05:00
Ryan Ernst f8026ed8fd Build: Fix dependency licenses check to correctly skip projects without dependencies 2015-12-02 11:03:56 -08:00
Jason Tedor 05430a788a Remove and forbid use of the type-unsafe empty Collections fields
This commit removes and now forbids all uses of the type-unsafe empty
Collections fields Collections#EMPTY_LIST, Collections#EMPTY_MAP, and
Collections#EMPTY_SET. The type-safe methods Collections#emptyList,
Collections#emptyMap, and Collections#emptySet should be used instead.
2015-12-02 10:41:59 -05:00
Robert Muir b7f497627f Merge pull request #15172 from rmuir/nuke_unused_deps
remove unused core dependencies
2015-12-02 07:20:11 -05:00
David Pilato 9c77cdc201 Fix build failure caused by Licenses Check in test-framework module
Typical failure:

```
:test-framework:dependencyLicenses (Thread[main,5,main]) started.
:test-framework:dependencyLicenses
Executing task ':test-framework:dependencyLicenses' (up-to-date check took 0.0 secs) due to:
  Task has not declared any outputs.
:test-framework:dependencyLicenses FAILED
:test-framework:dependencyLicenses (Thread[main,5,main]) completed. Took 0.023 secs.

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':test-framework:dependencyLicenses'.
> Licences dir /mnt/jenkins/workspace/es_core_master_strong/test-framework/licenses does not exist, but there are dependencies
```

Related to #15168
2015-12-02 10:08:35 +01:00
Ryan Ernst d68c6673a2 Build: Cleanup precommit task gradle code
This change attempts to simplify the gradle tasks for precommit. One
major part of that is using a "less groovy style", as well as being more
consistent about how tasks are created and where they are configured. It
also allows the things creating the tasks to set up inter task
dependencies, instead of assuming them (ie decoupling from tasks
eleswhere in the build).
2015-12-01 22:36:54 -08:00
Robert Muir 6e5c301739 remove unused dependencies 2015-12-01 22:22:11 -05:00
Ryan Ernst fcb6f44acf Build: Add ability to specify the compact profile for javac 2015-12-01 11:04:46 -08:00
Robert Muir f7e7a6bfad compile against compact3 profile 2015-12-01 09:40:54 -05:00
Ryan Ernst 4bb1eed766 Build: Increase the number of failed tests shown in test summary
We had increased this in maven, but it was lost in the transition to
gradle. This change adds it as a configurable setting the the logger for
randomized testing and bumps it to 25.
2015-11-30 13:40:39 -08:00
Ryan Ernst 0fa05edf03 Build: Improve output when integ test fails
This outputs a lot more info when integ tests fail to start, as well as
(should) fix windows (at least in my VM run) integ tests.
2015-11-29 19:44:43 -08:00
Robert Muir 93758874fb Merge pull request #15093 from rmuir/gradle_windoze_integ_tests
Fix integration tests and 'gradle run' to work on windoze
2015-11-29 12:15:30 -05:00
Robert Muir a01f86aaac Fix integration tests and gradle run to work on windoze 2015-11-29 07:39:18 -08:00
Ryan Ernst 8e25dc4f80 Merge pull request #15089 from rjernst/extra_config_file_delay
Fix delayed extra config file checks to be right before copy
2015-11-28 19:44:16 -08:00
Ryan Ernst 018c766bf7 Build: Fix delayed extra config file checks to be right before copy
The current delay waits until later after normal configuration, but
still just after resolution (ie when paths would be known for
dependencies but not actual execution). This delays the checks further
to be done right before we actually execute the copy task.

closes #15068
2015-11-28 18:06:36 -08:00
Ryan Ernst 1ee59e4977 Merge pull request #15088 from rjernst/integ_test_command_line
Move debug options setting to before command line is logged
2015-11-28 17:19:39 -08:00
Ryan Ernst f15ee9fa54 Build: Move debug options setting to before command line is logged
Currently if running with --info, the command line for ES, along with
env vars, are logged before they may be ammended to add debug options.
This moves the adding JAVA_OPTS to before we print the command.
2015-11-28 17:05:11 -08:00
Ryan Ernst 30fe74b5df Build: Setup standalone tests to compile in intellij
This adds the standalone tests so they will compile (and thus can be
modified with import completion) within IntelliJ. It also explicitly
sets up buildSrc as a module.

Note that this does *not* mean eg evil-tests can be run from intellij.
These are special tests that require special settings (eg disabling
security manager). They need to be run from the command line.

closes #15075
2015-11-28 16:57:09 -08:00
Nik Everett eba594fc08 Remove strange checks and clean dirs 2015-11-28 19:44:53 -05:00
Nik Everett 45e71329ee [build] Fix deb and rpm tests
Gradle is causing so much trouble here! Its too cute for its own good.
2015-11-28 19:07:06 -05:00
Robert Muir 66ae46ef8b Give enough ram so javac won't OOM
Today its based on the automatic -Xmx, but this
will fail in some environments (e.g. vagrant).
2015-11-26 02:16:46 -07:00
Ryan Ernst 139d688e24 Merge pull request #14996 from rjernst/logged_exec
Add wrapper around gradle Exec task for easier logging
2015-11-25 11:26:30 -08:00
Ryan Ernst bcb4be322f Build: Add wrapper around gradle Exec task for easier logging
The Exec task outputs stdout/stderr to the standard streams by default.
However, to keep output short, we currently capture this, and only
output if the task failed. This change makes a small wrapper around Exec
to facilitate this behavior anywhere we use Exec.
2015-11-25 11:26:02 -08:00
Ryan Ernst 52f31ee14a Build: Allow extra config for integ test to be anything project.file() accepts
This change delays the lookup for whatever is passed to extra config as
the source file to happen at execution time. This allows using eg a task
which generates a file, but maintains the checks that the file is not a
dir and that it exists at runtime.
2015-11-25 00:34:56 -08:00
Ryan Ernst d6969fcf3a Build: Split extra config files for integ test into separate task
This doesn't work when there are no extra config files, since gradle
sees the copy task and says "there is nothing to do".
2015-11-24 16:37:32 -08:00