Commit Graph

275 Commits

Author SHA1 Message Date
Adrien Grand 5d9fb2e8a6 Upgrade to lucene-5.3.0.
From a user perspective, the main benefit from this upgrade is that the new
Lucene53Codec has disk-based norms. The elasticsearch directory has been fixed
to load these norms through mmap instead of nio.

Other changes include the removal of `max_thread_states`, the fact that
PhraseQuery and BooleanQuery are now immutable, and that deleted docs are now
applied on top of the Scorer API.

This change introduces a couple of `AwaitsFix`s but I don't think it should
hold us from merging.
2015-09-01 11:58:45 +02:00
Jason Tedor e9b6949407 More portable extraction of short hostname
This commit increases the portability of extracting the short hostname
on a Unix-like system.

Closes #13107
2015-08-25 14:29:16 -04:00
David Pilato 8b85a8ff73 Update joda-time to 2.8.2
- DateTimeZone data updated to version 2015f

- Fixed to handle JDK 8u60 [#288, #291]
  Without this fix, formatting a time-zone will print "+00:00" instead of "GMT" for the GMT time-zone

- DateTimeZone data updated to version 2015e
2015-08-25 10:51:24 +02:00
Nik Everett d94630d938 Fix importing into eclipse
Renaming the distribution's artifactIds to "elasticsearch" caused the eclipse
import process to balk. Fix those modules by moving their eclipse package to
"[groupId].[artifactId]".
2015-08-21 13:36:26 -07:00
Ryan Ernst 164efaecbe Networking: Move multicast discovery to a plugin
Multicast has known issues (see #12999 and #12993). This change moves
multicast into a plugin, and deprecates it in the docs.  It also allows
for plugging in multiple zen ping implementations.

closes #13019
2015-08-20 16:43:25 -07:00
Robert Muir e2ab62596f Default to unicast discovery, with default host list of 127.0.0.1, [::1]
Fix unicast discovery to work when a host has multiple addresses.
Ban dangerous methods in java.net with forbidden APIs.
Fix ipv6 bugs and formatting of network addresses everywhere.

Closes #12999
Closes #12993

Squashed commit of the following:

commit 6c1aa001d091c5cf25212a53dc701fb704337f1e
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 14:25:43 2015 -0400

    Fix these to be correct with addresses just in case

commit 648215627e84abf58a71400e7dc9ae775efb71d6
Merge: d00561b 41d8fbe
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 13:23:09 2015 -0400

    Merge branch 'master' into unicast_all_the_way_down

commit d00561b76fd1aa5850699f7901f3dae3d4d402b7
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 16:38:50 2015 +0200

    limit local ports to 5 in UnicastZenPing

commit e2e15c594006746cbe24432694294a71cc99deb8
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 10:32:47 2015 -0400

    fix port limiting

commit 10153cb7adadda81a1f482445e703836b65cf5e2
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 10:18:37 2015 -0400

    don't serialize scopeids: that's broken

commit 2aa63d43db2baec68a2e9bc227cfeb85dfeb4f83
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 16:06:51 2015 +0200

    restore @Network

commit c840f1d1ef438826ae1ecfd5e45942a0e30dc9c0
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 16:02:30 2015 +0200

     Use NetworkAddress.formatAddress where applicable in plugins

commit 374ce878852b35d626b7a29c8c4773545b0e9ddd
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 15:34:06 2015 +0200

    Use NetworkAddress.formatAddress where applicable

commit e7a606d63f1bc43c1b62b6e17adf707c76d43a15
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 10:17:57 2015 +0200

    Add @Multicast annotation to disable multicast tests by default.

    We only run multicast tests now when we explicitly state it. A working
    multicast env is required which is not always the case.

commit 2d7d2d0347179696ab41f71f048b13305014c85b
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 09:51:28 2015 +0200

    Remove extra check for local mode in InternalTestCluster

commit dda59ac39aa136d4687b9274c2692cd77f8b8f66
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 09:37:03 2015 +0200

    Handle node mode across entire test cluster

    We used static methods reading sys properties to define the node mode
    per cluster. this had lots of problems when tests couldn't cope with
    mixed or only local mode. Now we are passing it down to the cluster from the test
    which allows to @SuppressNetworkMode / @SupressLocalMode on the test to force
    consistent node configurations.

commit 058197b7a408318995c88ce7f6762e32348de0de
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 03:19:14 2015 -0400

    really ban InetSocketAddress's trappy method and break build and go to sleep, sorry

commit ac8779185aee1e17e6f5a81766290fdfc9c603ba
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 03:16:52 2015 -0400

    Ban methods that might surprisingly cause DNS lookups

commit e64fe3dff2b11503e5f2831eb9863d64f56c5538
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 02:59:05 2015 -0400

    Add unit test

commit f15434f20fb1a3691b1cc16028597d8fae937e05
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 02:39:02 2015 -0400

    fix ipv6 formatting bugs

commit 05c2c74098052c75fbb79ea1818a295ef2e03e30
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 02:12:05 2015 -0400

    format addresses correctly so I can actually read what comes out of our logs and stats apis

commit 4f9389dcf1e8925f23153c5eb271b4ce2294dbaf
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Aug 19 21:26:52 2015 -0400

    ban dangerous methods in java.net

commit 6aacd4d9925f324903d1d099a6cf5f862aeaf677
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Aug 19 20:59:24 2015 -0400

    ban lenient method

commit f466a842c60163d1f4554bdce8a4163edb534c2c
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 00:29:00 2015 +0200

    fix tests to not mix local transport and zen unicast disco

commit 0de007a33b33fb68cf85cd86db4ca4f8ce10bbc9
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 00:10:07 2015 +0200

    fix tests to not mix local transport and zen unicast disco

commit 539f6ca6e5137e0d496239adc8684688dedcc824
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 00:02:01 2015 +0200

    fix tests to not mix local transport and zen unicast disco

commit 004c2881b25467f332acc8c9f9e92b1f0f9d314e
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Aug 19 17:51:45 2015 -0400

    Fix multinode

commit 54113af325ce31571811c49fdaae89d5687be4ba
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Aug 19 17:36:45 2015 -0400

    fix integration tests

commit 0156a77a56319d6b9737ec6a531992052e50bd59
Author: Simon Willnauer <simonw@apache.org>
Date:   Wed Aug 19 23:32:18 2015 +0200

    enable multicast in MulticastZenPingIT.java

commit 1791caa35da853ce0122485fa3fd4674c671ec6e
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Aug 19 17:23:16 2015 -0400

    Fix constant

commit 22820b53e0b2dc9fd47145c2bc29ce912a8fd484
Author: Simon Willnauer <simonw@apache.org>
Date:   Wed Aug 19 22:59:09 2015 +0200

    give it some extra ids for local transport crazyness

commit b2138fafa94a8a085813fd48356df63e57ade5b3
Author: Simon Willnauer <simonw@apache.org>
Date:   Wed Aug 19 22:51:42 2015 +0200

    pass on local addresses from configured transport rather than hard code IP addresses

commit 1bf5de1f457b081e0ce262b57d2b55d39c434156
Author: Simon Willnauer <simonw@apache.org>
Date:   Wed Aug 19 22:04:31 2015 +0200

    fix PluggableTransportModuleIT.java to use local disco and detach port limit for node local disco

commit b6706eddfa04c43947c16551359ae98a463d34aa
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Aug 19 14:16:03 2015 -0400

    Default to unicast discovery, with default host list of 127.0.0.1, [::1]
2015-08-20 14:26:40 -04:00
David Pilato dba1b52e81 [build] rpm module should be build on machine with /usr/bin/rpmbuild
We build the rpm module automatically if you have either:

* `/usr/bin/rpmbuild`
* `/usr/local/bin/rpmbuild`

available.

If your `rpmbuild` is in another location and available in your path, then run maven with `rpm` profile:

```sh
mvn deploy -Prpm
```

Closes #12984.
2015-08-19 15:37:28 +02:00
Simon Willnauer 0ffd99cca3 Drop commons-lang dependency
commons-lang really is only used by some core classes to join strings or modiy arrays.
It's not worth carrying the dependency. This commit removes the dependency on commons-lang
entirely.
2015-08-18 22:59:31 +02:00
David Pilato 692cc80523 [maven] also rename parent project artifactId
Also fixed bad scm links
2015-08-18 13:41:12 +02:00
David Pilato b5eb78875f [maven] rename maven names / ids for distribution modules 2015-08-18 13:38:49 +02:00
Robert Muir 68307aa9f3 Fix network binding for ipv4/ipv6
When elasticsearch is configured by interface (or default: loopback interfaces),
bind to all addresses on the interface rather than an arbitrary one.

If the publish address is not specified, default it from the bound addresses
based on the following sort ordering:

* ipv4/ipv6 (java.net.preferIPv4Stack, defaults to true)
* ordinary addresses
* site-local addresses
* link local addresses
* loopback addresses

One one address is published, and multicast is still always over ipv4: these
need to be future improvements.

Closes #12906
Closes #12915

Squashed commit of the following:

commit 7e60833312f329a5749f9a256b9c1331a956d98f
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 14:45:33 2015 -0400

    fix java 7 compilation oops

commit c7b9f3a42058beb061b05c6dd67fd91477fd258a
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 14:24:16 2015 -0400

    Cleanup/fix logic around custom resolvers

commit bd7065f1936e14a29c9eb8fe4ecab0ce512ac08e
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 13:29:42 2015 -0400

    Add some unit tests for utility methods

commit 0faf71cb0ee9a45462d58af3d1bf214e8a79347c
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 12:11:48 2015 -0400

    localhost all the way down

commit e198bb2bc0d1673288b96e07e6e6ad842179978c
Merge: b55d092 b93a75f
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 12:05:02 2015 -0400

    Merge branch 'master' into network_cleanup

commit b55d092811d7832bae579c5586e171e9cc1ebe9d
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 12:03:03 2015 -0400

    fix docs, fix another bug in multicast (publish host = bad here!)

commit 88c462eb302b30a82585f95413927a5cbb7d54c4
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 11:50:49 2015 -0400

    remove nocommit

commit 89547d7b10d68b23d7f24362e1f4782f5e1ca03c
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 11:49:35 2015 -0400

    fix http too

commit 9b9413aca8a3f6397b5031831f910791b685e5be
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 11:06:02 2015 -0400

    Fix transport / interface code

    Next up: multicast and then http
2015-08-17 15:43:07 -04:00
Nik Everett a9b2b033e3 [Packaging] Fix variable substitution
Systemd looks to be a bit less tolerant about $VAR than bash is. Replace
$VAR with ${VAR} in places in the systemd configuration file to get the
substitutions working.
2015-08-15 08:53:43 -07:00
Simon Willnauer b447e2ae99 Move master to [2.1.0-SNAPSHOT] 2015-08-14 23:44:06 +02:00
Lee Hinman 6f5a25d98e [DOC] Use 127.0.0.1 instead of localhost in READMEs
Users with IPv6 preferred over IPv4 may have `localhost` resolve to
`::1` instead of `127.0.0.1`, so we should be explicit so they don't run
into issues.
2015-08-14 14:47:58 -06:00
David Pilato 4e23fe2657 [build] revert maven 3.3.x fix
We need to revert #12803 as it creates some troubles in IntelliJ and also might fail when using older elasticsearch versions.
2015-08-13 15:21:06 +02:00
David Pilato fbd8f69273 Build of QA: Smoke Test Shaded Jar fails under maven 3.3.x
Build fails with maven 3.3.1 and 3.3.3. To reproduce, install one of the 3.3.x versions of maven and run `mvn clean verify` in the root directory of the project. The build will fail in the QA: Smoke Test Shaded Jar module with the following error:

```
Started J0 PID(99979@flea.local).
Suite: org.elasticsearch.shaded.test.ShadedIT
  2> NOTE: reproduce with: ant test  -Dtestcase=ShadedIT -Dtests.method=testJodaIsNotOnTheCP -Dtests.seed=2F4D23A7462CF921 -Dtests.locale= -Dtests.timezone=Asia/Baku -Dtests.asserts=true -Dtests.file.encoding=UTF-8
FAILURE 0.06s | ShadedIT.testJodaIsNotOnTheCP <<<
  > Throwable #1: junit.framework.AssertionFailedError: Expected an exception but the test passed: java.lang.ClassNotFoundException
  > at __randomizedtesting.SeedInfo.seed([2F4D23A7462CF921:3A9404F1F69FD80]:0)
  > at junit.framework.Assert.fail(Assert.java:57)
  > at java.lang.Thread.run(Thread.java:745)
  2> NOTE: reproduce with: ant test  -Dtestcase=ShadedIT -Dtests.method=testGuavaIsNotOnTheCP -Dtests.seed=2F4D23A7462CF921 -Dtests.locale= -Dtests.timezone=Asia/Baku -Dtests.asserts=true -Dtests.file.encoding=UTF-8
FAILURE 0.01s | ShadedIT.testGuavaIsNotOnTheCP <<<
  > Throwable #1: junit.framework.AssertionFailedError: Expected an exception but the test passed: java.lang.ClassNotFoundException
  > at __randomizedtesting.SeedInfo.seed([2F4D23A7462CF921:C2502FD54D83433D]:0)
  > at junit.framework.Assert.fail(Assert.java:57)
  > at java.lang.Thread.run(Thread.java:745)
  2> NOTE: reproduce with: ant test  -Dtestcase=ShadedIT -Dtests.method=testjsr166eIsNotOnTheCP -Dtests.seed=2F4D23A7462CF921 -Dtests.locale= -Dtests.timezone=Asia/Baku -Dtests.asserts=true -Dtests.file.encoding=UTF-8
FAILURE 0.01s | ShadedIT.testjsr166eIsNotOnTheCP <<<
  > Throwable #1: junit.framework.AssertionFailedError: Expected an exception but the test passed: java.lang.ClassNotFoundException
  > at __randomizedtesting.SeedInfo.seed([2F4D23A7462CF921:35593286F4269392]:0)
  > at junit.framework.Assert.fail(Assert.java:57)
  > at java.lang.Thread.run(Thread.java:745)
  2> NOTE: leaving temporary files on disk at: /Users/Shared/Jenkins/Home/workspace/elasticsearch-master/qa/smoke-test-shaded/target/J0/temp/org.elasticsearch.shaded.test.ShadedIT_2F4D23A7462CF921-001
  2> NOTE: test params are: codec=CheapBastard, sim=DefaultSimilarity, locale=, timezone=Asia/Baku
  2> NOTE: Mac OS X 10.10.4 x86_64/Oracle Corporation 1.8.0_25 (64-bit)/cpus=8,threads=1,free=482137936,total=514850816
  2> NOTE: All tests run in this JVM: [ShadedIT]
Completed [1/1] in 6.61s, 5 tests, 3 failures <<< FAILURES!

Tests with failures:
  - org.elasticsearch.shaded.test.ShadedIT.testJodaIsNotOnTheCP
  - org.elasticsearch.shaded.test.ShadedIT.testGuavaIsNotOnTheCP
  - org.elasticsearch.shaded.test.ShadedIT.testjsr166eIsNotOnTheCP
```
Please note that build doesn't fail with maven 3.2.x and it doesn't fail if mvn command is executed inside the qa/smoke-test-shaded directory. Only when the build is started from the root directory the error above can be observed.

The reason is because of the shaded version which depends on elasticsearch core.
When Maven build the module only, then elasticsearch core is not added to the dependency tree.

```sh
mvn dependency:tree -pl :smoke-test-shaded
```

```
[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ smoke-test-shaded ---
[INFO] org.elasticsearch.qa:smoke-test-shaded:jar:2.0.0-beta1-SNAPSHOT
[INFO] +- org.elasticsearch.distribution.shaded:elasticsearch:jar:2.0.0-beta1-SNAPSHOT:compile
[INFO] |  +- org.apache.lucene:lucene-core:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-backward-codecs:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-analyzers-common:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-queries:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-memory:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-highlighter:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-queryparser:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-sandbox:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-suggest:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-misc:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-join:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-grouping:jar:5.2.1:compile
[INFO] |  +- org.apache.lucene:lucene-spatial:jar:5.2.1:compile
[INFO] |  \- com.spatial4j:spatial4j:jar:0.4.1:compile
[INFO] +- org.hamcrest:hamcrest-all:jar:1.3:test
[INFO] \- org.apache.lucene:lucene-test-framework:jar:5.2.1:test
[INFO]    +- org.apache.lucene:lucene-codecs:jar:5.2.1:test
[INFO]    +- com.carrotsearch.randomizedtesting:randomizedtesting-runner:jar:2.1.16:test
[INFO]    +- junit:junit:jar:4.11:test
[INFO]    \- org.apache.ant🐜jar:1.8.2:test
```

But if shaded plugin is involved during the build, it modifies the `projectArtifactMap`:

```sh
mvn dependency:tree -pl org.elasticsearch.distribution.shaded:elasticsearch,:smoke-test-shaded
```

```
[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ smoke-test-shaded ---
[INFO] org.elasticsearch.qa:smoke-test-shaded:jar:2.0.0-beta1-SNAPSHOT
[INFO] +- org.elasticsearch.distribution.shaded:elasticsearch:jar:2.0.0-beta1-SNAPSHOT:compile
[INFO] |  \- org.elasticsearch:elasticsearch:jar:2.0.0-beta1-SNAPSHOT:compile
[INFO] |     +- org.apache.lucene:lucene-backward-codecs:jar:5.2.1:compile
[INFO] |     +- org.apache.lucene:lucene-analyzers-common:jar:5.2.1:compile
[INFO] |     +- org.apache.lucene:lucene-queries:jar:5.2.1:compile
[INFO] |     +- org.apache.lucene:lucene-memory:jar:5.2.1:compile
[INFO] |     +- org.apache.lucene:lucene-highlighter:jar:5.2.1:compile
[INFO] |     +- org.apache.lucene:lucene-queryparser:jar:5.2.1:compile
[INFO] |     |  \- org.apache.lucene:lucene-sandbox:jar:5.2.1:compile
[INFO] |     +- org.apache.lucene:lucene-suggest:jar:5.2.1:compile
[INFO] |     |  \- org.apache.lucene:lucene-misc:jar:5.2.1:compile
[INFO] |     +- org.apache.lucene:lucene-join:jar:5.2.1:compile
[INFO] |     |  \- org.apache.lucene:lucene-grouping:jar:5.2.1:compile
[INFO] |     +- org.apache.lucene:lucene-spatial:jar:5.2.1:compile
[INFO] |     |  \- com.spatial4j:spatial4j:jar:0.4.1:compile
[INFO] |     +- com.google.guava:guava:jar:18.0:compile
[INFO] |     +- com.carrotsearch:hppc:jar:0.7.1:compile
[INFO] |     +- joda-time:joda-time:jar:2.8:compile
[INFO] |     +- org.joda:joda-convert:jar:1.2:compile
[INFO] |     +- com.fasterxml.jackson.core:jackson-core:jar:2.5.3:compile
[INFO] |     +- com.fasterxml.jackson.dataformat:jackson-dataformat-smile:jar:2.5.3:compile
[INFO] |     +- com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:jar:2.5.3:compile
[INFO] |     |  \- org.yaml:snakeyaml:jar:1.12:compile
[INFO] |     +- com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:jar:2.5.3:compile
[INFO] |     +- io.netty:netty:jar:3.10.3.Final:compile
[INFO] |     +- com.ning:compress-lzf:jar:1.0.2:compile
[INFO] |     +- com.tdunning:t-digest:jar:3.0:compile
[INFO] |     +- org.hdrhistogram:HdrHistogram:jar:2.1.6:compile
[INFO] |     +- org.apache.commons:commons-lang3:jar:3.3.2:compile
[INFO] |     +- commons-cli:commons-cli:jar:1.3.1:compile
[INFO] |     \- com.twitter:jsr166e:jar:1.1.0:compile
[INFO] +- org.hamcrest:hamcrest-all:jar:1.3:test
[INFO] \- org.apache.lucene:lucene-test-framework:jar:5.2.1:test
[INFO]    +- org.apache.lucene:lucene-codecs:jar:5.2.1:test
[INFO]    +- org.apache.lucene:lucene-core:jar:5.2.1:compile
[INFO]    +- com.carrotsearch.randomizedtesting:randomizedtesting-runner:jar:2.1.16:test
[INFO]    +- junit:junit:jar:4.11:test
[INFO]    \- org.apache.ant🐜jar:1.8.2:test
```

A fix could consist of fixing something on Maven side. Probably something changed in a recent version and introduced this "issue" but it might be not really an issue. More a fix.

There are two workarounds:

1) exclude manually elasticsearch core from shaded version in smoke-test-shaded module and add manually each lucene lib needed by elasticsearch

2) add a new `elasticsearch-lucene` (lucene) POM module which simply declares all needed lucene libs in subprojects (such as the smoke tester one).

I choose the later.

Closes #12791.
2015-08-13 12:15:19 +02:00
Simon Willnauer 605253a39f Cut over master to 2.0.0-SNAPSHOT 2015-08-12 21:16:08 +02:00
Robert Muir 0f61f56b60 use spaces liberally in integration tests and fix space handling
Closes #12710
2015-08-11 18:03:41 +02:00
Adrien Grand abe3880dd9 Merge pull request #12274 from smflorentino/smflorentino-fix-winjvm-detect
Improve JVM Arch Detection
2015-08-11 16:17:26 +02:00
Nik Everett 3c76e15aab Merge pull request #12785 from nik9000/remove_conf
Fix rpm -e removing /etc/elasticsearch
2015-08-11 05:08:07 -07:00
Nik Everett 7223f7c75a Packaging: fix rpm -e removing /etc/elasticsearch
When we fixed rpm creating the /etc/elasticsearch/scripts directory we
broke removing the rpm - it lef the /etc/elasticsearch directory behind.
This fixes that.
2015-08-10 14:32:41 -07:00
Ryan Ernst 40f119d85a This method on settings loaded a class, based on a setting value, using
the default classloader. It had all kinds of leniency in how the
classname was found, and simply cannot work with plugins having isolated
classloaders.

This change removes that method. Some of the uses of it were for custom
extension points, like custom repository or discovery types. A lot were
just there to plugin mock implementations for tests. For the settings
that were legitimate, all now support plugins adding the given setting
via onModule. For those that were specific to tests for mocks, they now
use Classes.loadClass (a helper around Class.forName). This is a
temporary measure until (in a future PR) tests can change the
implementation via package private statics.

I also removed a number of unnecessary intermediate modules, added a
"jvm-example" plugin that can be filled in in the future as a smoke test
for breaking plugins, and gave some documentation to "spawn" modules
interface.

closes #12643
closes #12656
2015-08-10 14:04:45 -07:00
Scott Florentino e7aa2a1fb2 Closing #12256 Update service.bat with a safer RAM value 2015-08-10 13:22:29 -07:00
Colin Goodheart-Smithe 6abc69c488 Packaging: Makes sure all POMs contain a description
Adds an explicit description the RPM package so it doesn't inherit the description from the POM.

Closes #12550

Also, modified descriptions for deb and rpm packages to be the same and to reference the documentation rather than listing features that are out of date.
2015-08-10 15:21:24 +01:00
Nik Everett cde2502732 Merge pull request #12706 from nik9000/osx_can_build_rpm
Let OSX build rpms for linux
2015-08-10 05:48:36 -04:00
Nik Everett f84552dc98 [Packaging] Run tests in vagrant
This creates a module in qa called vagrant that can be run if you have
vagrant and virtualbox installed and will run the packaging tests in trusty
and centos-7.0. You can ask it to run tests in other linuxes. This is the full
list:
* precise aka Ubuntu 12.04
* trusty aka Ubuntu 14.04
* vivid aka Ubuntun 15.04
* wheezy aka Debian 7, the current debian oldstable distribution
* jessie aka Debian 8, the current debina stable distribution
* centos-6
* centos-7
* fedora-22
* oel-7

There is lots of documentation on how to do this in the TESTING.asciidoc.

Closes #12611
2015-08-10 05:31:42 -04:00
Nik Everett d6087667bf Merge pull request #12704 from nik9000/create_scripts
rpm and deb create scripts directory
2015-08-10 05:27:45 -04:00
Tanguy Leroux b1fd0a61da Do not kill process on service shutdown
When installed as a service with a DEB or RPM package, we should gently wait for elasticsearch to stop (flushing indices on closing can take some time) and never kill the process.

Closes #11248
2015-08-10 10:04:56 +02:00
Clinton Gormley 205262c737 Enable the license checker over distribution/* and plugins/*
Moved the license checker config into the parent pom, and overrede
the license dir/target-to-check in distributions/pom.

Disabled the license checker explicitly for projects which run integration
tests but have no licenses dir:

* core
* distribution
* qa
* plugins/delete-by-query
* plugins/mapper-size
* plugins/site-example

Closes #12752
Closes #12754
2015-08-09 18:18:59 +02:00
Clinton Gormley 7efc419041 Revert "Revert "disable license checker on windows for now, jenkins boxes have perl problems""
This reverts commit e2cbec185b.
2015-08-07 16:47:52 +02:00
Alexander Reelsen b6016b6e1e Bats testing: Remove useless systemctl check
Bats testing uncovered a useless systemctl check, that resulted in an
error, because the systemctl file was uninstalled, but we hoped to
check for an explicetely configured SystemExitCode.

In addition we did not reload the systemctl configuration when uninstalling
elasticsearch, which now is fixed as well.

Closes #12682
2015-08-07 14:42:38 +02:00
Clinton Gormley e2cbec185b Revert "disable license checker on windows for now, jenkins boxes have perl problems"
This reverts commit 6e9d8671bc.
2015-08-07 14:27:24 +02:00
Nik Everett 8e6e8ff234 Let OSX build rpms for linux
Closes #12701
2015-08-07 08:11:34 -04:00
Nik Everett f1fc2efcc8 Packaging: rpm and deb create scripts directory
Elasticsearch will create this if it doesn't exist if it cant but because
it doesn't own /etc/elasticsearch when installed by rpm and deb it can't
create /etc/elasticsearch/scripts.

Closes #12702
2015-08-07 08:10:53 -04:00
Robert Muir f50531c403 Revert "Revert "disable license checker on windows for now, jenkins boxes have perl problems""
This reverts commit 633b4868d9.
2015-08-06 23:38:24 -04:00
Robert Muir 633b4868d9 Revert "disable license checker on windows for now, jenkins boxes have perl problems"
This reverts commit 6e9d8671bc.

Lets see if they are up for the task now.
2015-08-06 22:17:45 -04:00
Robert Muir 6e7d1a828d fix totally broken batch file. thank you jenkins. windows you still suck. 2015-08-06 18:27:06 -04:00
Robert Muir 6e9d8671bc disable license checker on windows for now, jenkins boxes have perl problems 2015-08-06 18:01:04 -04:00
Robert Muir 7414d19d28 Support jenkins randomization in integration tests 2015-08-06 14:48:27 -04:00
Robert Muir 354a32b05d remove annoying maven warning 2015-08-05 22:08:12 -04:00
Robert Muir 5fe49e4420 cleanup skipping everywhere 2015-08-05 21:45:09 -04:00
Alexander Reelsen 9028fb8fbe Release: Update build release script to reflect latest changes
As the script now deploys to S3 and several things in master have
changed, this script needs to reflect the latest changes

* An unsigned RPM is built by default, so that users of older
  RPM based distros can download and use that RPM by default
* In addition a signed RPM is built, that is used for the repositories
* Paths for the new distributions have been fixed
* The check for the number of jars has been removed, as this is done
  as part of the license checking in `mvn verify`
* Checksum generation has been removed, as this is done as part of the
  mvn build
* Publishing artifacts of S3 has been removed
* Repostitory creation script has been updated
2015-08-05 09:38:42 +02:00
Robert Muir 6f9a067197 Change master branch back to 2.0-beta1 2015-08-04 15:38:21 -04:00
Robert Muir 29d73accdd Merge pull request #12639 from rmuir/improve_integ
Don't use port 9200/9300 for integration tests
2015-08-04 13:09:47 -04:00
David Pilato c61dccd189 Packaging: mvn install renames artifacts when copying
This PR:

* renames all distribution artifacts to `elasticsearch` so maven plugins will pick up the correct finalName without needing any hack.
* changes the groupId for every single distribution module as we can't have more than one module using the same groupId:artifactId
* does not attach anymore empty jar files for tar/zip/... modules as they don't contain any `src/main/java` stuff.

When you build it, you end up with:

```
$ tree ~/.m2/repository/org/elasticsearch/distribution
distribution
├── deb
│   └── elasticsearch
│       ├── 2.0.0-beta1-SNAPSHOT
│       │   ├── elasticsearch-2.0.0-beta1-SNAPSHOT.deb
│       │   ├── elasticsearch-2.0.0-beta1-SNAPSHOT.deb.md5
│       │   ├── elasticsearch-2.0.0-beta1-SNAPSHOT.deb.sha1
│       │   ├── elasticsearch-2.0.0-beta1-SNAPSHOT.pom
│       │   ├── elasticsearch-2.0.0-beta1-SNAPSHOT.pom.md5
│       │   └── elasticsearch-2.0.0-beta1-SNAPSHOT.pom.sha1
├── elasticsearch-distribution
│   ├── 2.0.0-beta1-SNAPSHOT
│   │   ├── elasticsearch-distribution-2.0.0-beta1-SNAPSHOT.pom
│   │   ├── elasticsearch-distribution-2.0.0-beta1-SNAPSHOT.pom.md5
│   │   └── elasticsearch-distribution-2.0.0-beta1-SNAPSHOT.pom.sha1
├── fully-loaded
│   └── elasticsearch
│       ├── 2.0.0-beta1-SNAPSHOT
│       │   ├── elasticsearch-2.0.0-beta1-SNAPSHOT.pom
│       │   ├── elasticsearch-2.0.0-beta1-SNAPSHOT.pom.md5
│       │   └── elasticsearch-2.0.0-beta1-SNAPSHOT.pom.sha1
├── rpm
│   └── elasticsearch
│       ├── 2.0.0-beta1-SNAPSHOT
│       │   ├── elasticsearch-2.0.0-beta1-SNAPSHOT.pom
│       │   ├── elasticsearch-2.0.0-beta1-SNAPSHOT.pom.md5
│       │   ├── elasticsearch-2.0.0-beta1-SNAPSHOT.pom.sha1
│       │   ├── elasticsearch-2.0.0-beta1-SNAPSHOT.rpm
│       │   ├── elasticsearch-2.0.0-beta1-SNAPSHOT.rpm.md5
│       │   └── elasticsearch-2.0.0-beta1-SNAPSHOT.rpm.sha1
├── shaded
│   └── elasticsearch
│       ├── 2.0.0-beta1-SNAPSHOT
│       │   ├── elasticsearch-2.0.0-beta1-SNAPSHOT.jar
│       │   ├── elasticsearch-2.0.0-beta1-SNAPSHOT.jar.md5
│       │   ├── elasticsearch-2.0.0-beta1-SNAPSHOT.jar.sha1
│       │   ├── elasticsearch-2.0.0-beta1-SNAPSHOT.pom
│       │   ├── elasticsearch-2.0.0-beta1-SNAPSHOT.pom.md5
│       │   └── elasticsearch-2.0.0-beta1-SNAPSHOT.pom.sha1
├── tar
│   └── elasticsearch
│       ├── 2.0.0-beta1-SNAPSHOT
│       │   ├── elasticsearch-2.0.0-beta1-SNAPSHOT.pom
│       │   ├── elasticsearch-2.0.0-beta1-SNAPSHOT.pom.md5
│       │   ├── elasticsearch-2.0.0-beta1-SNAPSHOT.pom.sha1
│       │   ├── elasticsearch-2.0.0-beta1-SNAPSHOT.tar.gz
│       │   ├── elasticsearch-2.0.0-beta1-SNAPSHOT.tar.gz.md5
│       │   └── elasticsearch-2.0.0-beta1-SNAPSHOT.tar.gz.sha1
└── zip
    └── elasticsearch
        └── 2.0.0-beta1-SNAPSHOT
            ├── elasticsearch-2.0.0-beta1-SNAPSHOT.pom
            ├── elasticsearch-2.0.0-beta1-SNAPSHOT.pom.md5
            ├── elasticsearch-2.0.0-beta1-SNAPSHOT.pom.sha1
            ├── elasticsearch-2.0.0-beta1-SNAPSHOT.zip
            ├── elasticsearch-2.0.0-beta1-SNAPSHOT.zip.md5
            ├── elasticsearch-2.0.0-beta1-SNAPSHOT.zip.sha1
            └──
```

Closes #12536
2015-08-04 15:34:11 +02:00
Robert Muir cf6fe6cdcd Don't use port 9200/9300 for integration tests 2015-08-04 09:25:44 -04:00
Nik Everett e8ccdf91fc Merge pull request #12610 from nik9000/bug/12504_2.0
Plugin script: Fix spaces
2015-08-04 09:05:38 -04:00
Robert Muir 5059c978e3 Fix distribution checks to pass unzipped directory to license checker 2015-08-04 07:54:55 -04:00
Simon Willnauer 6753f7f03e Cut over master to 2.0.0-SNAPSHOT 2015-08-04 10:54:12 +02:00
Robert Muir c059101fbb Fix missed test class 2015-08-03 21:56:43 -04:00
Ryan Ernst 1e12d03252 Tests: Rename base tests cases to use "TestCase" suffix
Most of the abstract base test classes we have were previously @Ignored.
However, there were also some other tests ignored. Having two ways to
quiet tests is confusing, and clearly it has caused some tests
to get lost in the fold.

This change moves all base test classes to use the "TestCase" suffix,
which is not picked up by the test class name pattern. It also removes
@Ignore from (almost) all tests, and adds it to forbidden apis.
And since we were renaming, I shorted base test class names to use
"ES" instead of "Elasticsearch". I type this a lot of types a day,
and I have heard others express a similar desire for a shorter name.

closes #10659
2015-08-03 17:43:00 -07:00
Scott Florentino 21a7518271 Merge branch 'master' into smflorentino-fix-winjvm-detect 2015-08-03 17:26:25 -07:00
Nik Everett 1b5bf17ae0 Plugin script: Fix spaces
Fixes ES_HOME with spaces and installing plugins from a local directory
with spaces.

Closes #12504
2015-08-03 15:40:18 -04:00
Robert Muir 0e027d82c9 Merge pull request #12602 from rmuir/core_integ_tests
Add integration test phase for elasticsearch core/
2015-08-03 10:15:37 -04:00
Robert Muir 379ffdb8ee Add integration test phase for elasticsearch core/ 2015-08-03 09:43:11 -04:00
Robert Muir e1886a9b12 Test that shaded jar contains only shaded classes 2015-08-03 08:56:45 -04:00
Robert Muir a5a27a5cd9 Add jar hell check test for the shaded jar 2015-08-03 05:54:55 -04:00
Robert Muir aaf0976cb7 clear up dependency hell a bit more 2015-08-01 14:02:14 -04:00
Robert Muir eaaa06b6b3 fix shading to create correct POM and not drag in wacky dependencies 2015-08-01 12:44:00 -04:00
Robert Muir 6f33ea4f62 move shading to distribution/shaded 2015-08-01 11:37:43 -04:00
Robert Muir fef55348f9 clean up .zip handling so its not uselessly downloaded in the build 2015-08-01 09:10:03 -04:00
Robert Muir 8d5b5ad862 restore my lost code changes and add measures to detect/prevent this in the future 2015-07-31 13:00:24 -04:00
Robert Muir 3d7da5a826 remove unused properties sections from distribution modules 2015-07-31 10:20:50 -04:00
Robert Muir f804ec3cd9 cleanup 2015-07-31 10:05:11 -04:00
Robert Muir 2068fbfd1e Merge pull request #12549 from rmuir/distribution_tests_1
add integration tests for tar/deb/rpm packaging
2015-07-30 09:18:09 -04:00
Alexander Reelsen 69d7f1a78a Startup: Remove getopt parsing in shell script, use java CLITool
In order to ensure, we have the same experience across operating systems
and shells, this commit uses the java CLI parser instead of the shell
getopt parsing to parse arguments.

This also allows for support for paths, which contain spaces.

Also commons-cli depdency was upgraded to 1.3.1 and tests have been added.

Changes

* new exit code, OK_AND_EXIT, allowing to tell the caller to exit, as everything
  went as expected (e.g. when running a version output)

BWC breaking:

* execute() returns an ExitStatus instead of an integer, otherwise there is no
  possibility to signal by a command, if the JVM should be exited after a run.
  This affects plugins, that have command line tools
* -v used to be version, but is a verbose flag by default in the current CLI infra,
  must be -V or --version now
* -X has been removed - the current implementation was useless anyway, as
  it prefixed those properties with "es.". You should use
  ES_JAVA_OPTS/JAVA_OPTS for JVM configuration
2015-07-30 13:20:29 +02:00
Adrien Grand 6893643bef Build: Set the correct sha1 for HDRHistogram. 2015-07-30 10:24:55 +02:00
Robert Muir ddfea366de Add integration tests for distribution/rpm 2015-07-29 22:26:45 -04:00
Robert Muir 2ed8e697d5 Add integration tests for distribution/deb 2015-07-29 21:34:46 -04:00
Robert Muir 1c5a558164 Add integration tests for distribution/tar 2015-07-29 15:33:29 -04:00
Robert Muir c6d6eaef4d Add a run.sh to run from current source code with debugger
The maven magic needed for this is now heavier
2015-07-29 14:21:43 -04:00
Alexander Reelsen c423319dda Build: Remove profile to create attached RPM
This is no longer necessary, as the RPM is now built as
a primary artifact, so no need to deploy it as a secondary
artifact anymore.

Closes #12529
2015-07-29 15:54:26 +02:00
Robert Muir c6abf28273 Disable license checker temporarily to get regression builds working again
x
2015-07-29 08:39:25 -04:00
Chris Earle e1c99bc8b4 Add -XX:+PrintGCDateStamps when using GC Logs
Now that Elasticsearch requires Java 7 or later, it's safe to add `-XX:+PrintGCDateStamps` to get human readable times alongside JVM times.
2015-07-27 15:17:54 -04:00
Alexander Reelsen 9628d2632f Build: Split packages into submodules
This change creates a proper `distribution` modules in which we have today packaging for
all of our four current packages:

* zip
* tar.gz
* rpm
* deb

Licenes have moved into the distribution project as well. So have the config/ and the bin/ directory
from the core/ project.

The RPM package is now built, if rpmbuild exists.

The bats tests have been moved as well.

Also the zip distribution now executes the REST integration tests.
2015-07-27 17:50:54 +02:00