Commit Graph

409 Commits

Author SHA1 Message Date
Alexander Reelsen 6436cd52f2 Smoke tester: Remove deprecated verbose flag for run() method 2015-10-01 15:14:23 +02:00
Nik Everett 8a7f9edf02 Merge pull request #13820 from nik9000/immutable_set_be_gone_2
Finish banning ImmutableSet
2015-09-29 15:34:34 +02:00
Robert Muir e0d42739dd Factor groovy out of core into lang-groovy 2015-09-28 20:17:45 -04:00
Nik Everett 85b99d2011 Finish banning ImmutableSet
Ban ImmutableSet$Builder because that let you sneak some `ImmutableSet`s in.

Remove all remaining imports of ImmutableSet.
2015-09-28 01:49:46 +02:00
Nik Everett 65041a8121 Entirely remove and ban ImmutableSet
The last usage was ImmutableMap#keySet
2015-09-23 16:26:16 -04:00
Nik Everett 04c570461e Remove and ban ImmutableSet#of 2015-09-23 16:14:38 -04:00
Nik Everett b0ab02e35c Remove and ban ImmutableSet#copyOf
It was used heavily in the Guice we've embedded.
2015-09-23 15:37:49 -04:00
Nik Everett 6ecda41485 Remove and ban ImmutableSet#builder 2015-09-23 13:55:34 -04:00
Nik Everett 52f3c89c3b Remove and ban ImmutableMap#entrySet
Banning `ImmutableSet` outright is too much to do all at once - this starts
the process by banning `ImmutableMap#entrySet` - one of the more common ways
that `ImmutableSet`s come up. It then starts to remove calls to
`ImmutableMap#entrySet` by changing declarations from `ImmutableMap` to `Map`.

Unfortunately this process is like pulling on a long, windy string and one
declaration change requires another which requires 5 more which in turn
require another few. So this change is rather large.

As such, to keep the changes manageable they only remove `ImmutableMap` from
the signatures that are needed for `entrySet` and make little effort to stop
using `ImmutableMap` internally. Removing the usages of `ImmutableMap`
complicates immutability guarantees and will be done separately.
2015-09-23 11:07:28 -04:00
Robert Muir 689af1a6d6 Factor expressions scripts out to lang-expression plugin 2015-09-22 20:33:47 -04:00
Nik Everett e284210652 [core] Forbid ForwardingSet
Removes CopyOnWriteHashSet, our only usage of ForwardingSet. We weren't
using it.

Related to #13224
2015-09-22 12:31:08 -04:00
Chris Earle 8a32891f1d Adding the actual method name of the replacement method for FileSystem.getDefault 2015-09-22 10:42:14 -04:00
Chris Earle eb266cdde6 Add packages to the 'Use xyz instead' comments 2015-09-22 10:42:14 -04:00
David Pilato f230eabc15 [cloud-azure] Split azure plugin in 3 plugins
Until now we had a cloud-azure plugin which is providing 3 distinct features:

* discovery on Azure
* snapshot/restore on Aure
* SMB store

This commit splits the plugin by feature so people can use either one or the other or both features.

Doc is updated accordingly.
2015-09-21 17:55:23 +02:00
Jason Tedor c281826702 Remove and forbid use of com.google.common.primitives.Ints
This commit removes and now forbids all uses of
com.google.common.primitives.Ints across the codebase. This is one of
many steps in the eventual removal of Guava as a dependency.

Relates #13224
2015-09-18 08:43:17 -04:00
David Pilato 422cfa27c0 Merge branch 'maven/elasticsearch-already-excluded' 2015-09-17 18:10:04 +02:00
Jason Tedor 257f3e862f Add missing forbidden entries for removed Guava classes 2015-09-15 11:15:36 -04:00
Alexander Reelsen dea0243193 Release script: Major Improvements
In order to be able to clean up broken steps in the release, one should
be able to run each step individually. So now one needs to specifiy each
step to run via cmdline argument

* --deploy-sonatype: runs `mvn deploy` and pushes to the staging mvn repo
* --deploy-s3: copies all artifacts over to the s3bucket
* --deploy-s3-repos: Creates the s3 repositories
* --no-install option, so that an existing mvn repo can be used for s3 operations

Also, several minor changes have been added

* Fixed typo in smoke test reference for email
* Checking for correct AWS environment variables
* Only create releases directory if it does not exist
* RPM sign check is only executed after `mvn install`
* Various path fixes for deb/rpm-s3 uploads
* Added output of deb/rpm-s3 commands for easier debugging
* Add configurable destination s3 bucket
* Removed verbosity, always be verbose
* Added color to the command which is being running right now, to differentiate from console output
2015-09-15 17:09:35 +02:00
Jason Tedor b912ad7d56 Remove file that was inadvertently committed 2015-09-15 10:42:11 -04:00
Jason Tedor b3c6327caf Remove and forbid use of com.google.common.base.Joiner
This commit removes and now forbids all uses of
com.google.common.base.Joiner across the codebase. This is one of many
steps in the eventual removal of Guava as a dependency.

Relates #13224
2015-09-15 10:30:42 -04:00
Jason Tedor 7a29febd72 Remove and forbid use of com.google.common.math.LongMath
This commit removes and now forbids all uses of
com.google.common.math.LongMath across the codebase. This is one step
of many in the eventual removal of Guava as a dependency.
2015-09-15 10:28:02 -04:00
Simon Willnauer ff4a11aa32 Replace and ban next batch of Guava classes
This commit replaces and bans:
 * com.google.common.util.concurrent.UncheckedExecutionException
 * com.google.common.util.concurrent.AtomicLongMap
 * com.google.common.primitives.Longs
 * com.google.common.io.ByteStreams
 * com.google.common.collect.UnmodifiableIterator
 * com.google.common.collect.ObjectArrays
 * com.google.common.collect.Multimap
 * com.google.common.collect.MultimapBuilder

Relates to #13224
2015-09-15 14:44:41 +02:00
Jason Tedor 527ab95c39 Remove and forbid use of com.google.common.collect.Iterables
This commit removes and now forbids all uses of
com.google.common.collect.Iterables across the codebase. This is one of
many steps in the eventual removal of Guava as a dependency.

Relates #13224
2015-09-15 07:47:36 -04:00
Jason Tedor f5c408535d Remove and forbid use of com.google.common.base.Preconditions
This commit removes and now forbids all uses of
com.google.common.base.Preconditions across the codebase. This is one
of many steps in the eventual removal of Guava as a dependency.

Relates #13224
2015-09-15 07:23:34 -04:00
Alexander Reelsen 5f6aeb982d Release script: Improve automation for package repositories
* Automatic package repository creation for debian and rpm repositories using deb-s3 and rpm-s3 tools
* Fixing paths in email for repositories
* Add manual verification step for maven staging repo
* Do not create release directory in /tmp, because we might loose it on VMs
* Removed unused download-s3 script
* Add signage check for RPM
* Removed download-s3.py/upload-s3.py, as they are unused

Closes #13209
2015-09-14 15:03:30 +02:00
Simon Willnauer 30bd46ab25 Replace LoadingCache usage with a simple ConcurrentHashMap
This commit replaces the usage of LoadedCache with a simple CHM and calls
to computeIfAbsent and adds LoadingCache and CacheLoader to forbidden APIs

Relates to #13224
2015-09-14 13:25:58 +02:00
Alexander Reelsen a96350d785 Release: Add 1.7.2 bwc indices/versions
* Added BWC indices
* Added snapshot version to Version.java
* Fixed create_bwc_index to use localhost instead of localhost and 127.0.0.1 (problem with ipv4/6 setup)
2015-09-14 12:58:46 +02:00
Simon Willnauer 40959068d5 Remove and forbid use of guava Function, Charsets, Collections2
This commit removes and now forbids all uses of
Function, Charsets, Collections2  across the codebase. This
is one of many steps in the eventual removal of Guava as a dependency.

Relates #13224
2015-09-14 10:27:12 +02:00
Robert Muir b16e1569fe Remove all setAccessible in tests and forbid 2015-09-12 19:46:39 -04:00
Robert Muir d6f56030d8 ban setAccessible from core code.
In addition to being a big security problem, setAccessible is a risk
for java 9 migration. We need to clean up our code so we can ban it
and eventually enforce this with security manager for third-party code, too,
or we may have problems.

Instead of using setAccessible, use the correct modifier (e.g. public).

TODO: ban in tests
TODO: ban in security manager at runtime
2015-09-12 02:11:06 -04:00
Jason Tedor d6d8d30d47 Remove and forbid use of com.google.common.collect.ImmutableSortedMap
This commit removes and now forbids all uses of
com.google.common.collect.ImmutableSortedMap across the codebase. This
is one of many steps in the eventual removal of Guava as a dependency.

Relates #13224
2015-09-11 17:42:19 -04:00
Simon Willnauer 7b0f086946 Remove and forbid use of several com.google.common.util. classes
This commit replaces:
 * com.google.common.util.concurrent.ListenableFuture
 * com.google.common.util.concurrent.SettableFuture
 * com.google.common.util.concurrent.Futures
 * com.google.common.util.concurrent.MoreExecutors

And forbits its usage via forbidden APIs. This is one of
many steps in the eventual removal of Guava as a dependency.

Relates to #13224
2015-09-11 21:15:12 +02:00
Jason Tedor 722a08d4d1 Forbid Guava in all instead of core
Now that Guava is no longer shaded, it can be forbidden in all instead
of core.

Relates #13224
2015-09-11 14:53:15 -04:00
Adrien Grand e988a9edc6 Merge pull request #13439 from rmuir/1702090
upgrade lucene to r1702265
2015-09-11 09:48:43 +02:00
Jason Tedor b5b22d4b01 Remove and forbid use of com.google.common.collect.Queues
This commit removes and now forbids all uses of
com.google.common.collect.Queues across the codebase. This is one of
many steps in the eventual removal of Guava as a dependency.

Relates #13224
2015-09-10 23:59:59 -04:00
Jason Tedor 65353b8a32 Remove and forbid use of com.google.common.collect.ImmutableSortedSet
This commit removes and now forbids all uses of
com.google.common.collect.ImmutableSortedSet across the codebase. This
is one of many steps in the eventual removal of Guava as a dependency.

Relates #13224
2015-09-10 23:33:55 -04:00
Robert Muir dd208002c9 Merge branch 'master' into 1702090 2015-09-10 21:41:07 -04:00
Jason Tedor 5d4d34ab16 Remove and forbid use of c.g.c.b.Preconditions#checkNotNull
This commit removes and now forbids all uses of
com.google.common.base.Preconditions#checkNotNull across the codebase.
This is one of many steps in the eventual removal of Guava as a
dependency.

Relates #13224
2015-09-10 17:57:00 -04:00
Robert Muir c1f2fc76c2 Upgrade lucene to r1702090
The semantics of the `boost` parameter for `function_score` changed. This is
due to the fact that Lucene now requires that query boosts and top-level boosts
are applied the same way.
2015-09-10 23:36:43 +02:00
Ryan Ernst 47de1bd923 Merge pull request #13443 from rjernst/simplify_bwc_path
Move static bwc indexes to a shared location
2015-09-10 10:31:50 -07:00
Jason Tedor 3bd1d38176 Remove and forbid use of com.google.common.collect.Sets
This commit removes and now forbids all uses of
com.google.common.collect.Sets across the codebase. This is one of many
steps in the eventual removal of Guava as a dependency.

Relates #13224
2015-09-10 11:22:09 -04:00
Nik Everett 56c3471851 Fix test for _cat/nodeattrs
Adds a node attribute to all test runs and uses the attribute to test
`_cat/nodeattrs`.

Note that its quite possible create an impressively slow regex while doing
this and you have to be careful. See comment in commit for more if curious.

Closes #12558
2015-09-10 10:50:51 -04:00
Alexander Reelsen 9421aaea4d Release: Remove deprecated method of repository creation
This removes the old shell script for repository creation.
Small fix for gpg key name as well.
2015-09-10 11:14:31 +02:00
Ryan Ernst 8f75c2b3a8 Tests: Move static bwc indexes to a shared location
There are a few tests that currently use the statically generated
backcompat indexes. This change moves them to a shared location, so they
no longer have to build a path based on the package name of the old
index tests.
2015-09-09 15:42:26 -07:00
Jason Tedor 2a5412ebf0 Remove and forbid use of com.google.common.collect.Maps
This commit removes and now forbids all uses of
com.google.common.collect.Maps across the codebase. This is one of many
steps in the eventual removal of Guava as a dependency.

Relates #13224
2015-09-09 17:41:41 -04:00
Jason Tedor 1806c1e0c7 Remove and forbid use of com.google.common.base.Throwables
This commit removes and now forbids all uses of
com.google.common.base.Throwables across the codebase.

For uses of com.google.common.base.Throwables#getStackTraceAsString,
use org.elasticsearch.ExceptionsHelper#stackTrace.

Relates #13224
2015-09-08 21:56:23 -04:00
Alexander Reelsen 28970e7d54 Release: Add script to validate mvn repositories
This script allows to ensure that artifacts have been pushed to
a repository after running `mvn deploy`. This will allow us to
check that all of our artifacts have been deployed to sonatype and
the S3 bucket.

Basically it takes the contents of a local mvn repository and
runs HTTP HEAD requests against all artifacts. It also compares if
the length returned by the Content-Length header is the same as the
size of the artifact locally.
2015-09-08 14:12:33 +02:00
David Pilato b89efbed69 [maven] assembly: elasticsearch is already excluded
As elasticsearch is marked as provided we don't need to explicitly exclude it from the assembly descriptor.

We get a warning today for all plugins, the following:

```
[INFO] --- maven-assembly-plugin:2.5.5:single (default) @ repository-s3 ---
[INFO] Reading assembly descriptor: /path/to/plugin-assembly.xml
[WARNING] The following patterns were never triggered in this artifact exclusion filter:
o  'org.elasticsearch:elasticsearch'
[INFO] Building zip: /path/to/target/releases/repository-s3-3.0.0-SNAPSHOT.zip
[INFO]
```

It now gives:

```
[INFO] --- maven-assembly-plugin:2.5.5:single (default) @ repository-s3 ---
[INFO] Reading assembly descriptor: /path/to/plugin-assembly.xml
[INFO] Building zip: /path/to/target/releases/repository-s3-3.0.0-SNAPSHOT.zip
[INFO]
```
2015-09-08 08:55:31 +02:00
Jason Tedor a6ffe8f6d5 Remove and forbid use of com.google.common.base.Strings
This commit removes and now forbids all uses of
com.google.common.base.Strings across the codebase.

For uses of com.google.common.base.Strings.isNullOrEmpty, use
org.elasticsearch.common.Strings.isNullOrEmpty.

For uses of com.google.common.base.Strings.padStart use
org.elasticsearch.common.Strings.padStart.

For uses of com.google.common.base.Strings.nullToEmpty use
org.elasticsearch.common.Strings.coalesceToEmpty.

Relates #13224
2015-09-07 09:40:14 -04:00
Jason Tedor 8a3411e5e4 Remove and forbid the use of com.google.common.base.Predicate(s)?
This commit removes and now forbids all uses of
com.google.common.base.Predicate and com.google.common.base.Predicates
across the codebase. This is one of the many steps in the eventual
removal of Guava as a dependency. This was enabled by #13314.

Relates #13224
2015-09-06 07:20:24 -04:00
Jason Tedor 14e4882425 Remove and forbid use of com.google.common.base.Objects
This commit removes and now forbids all uses of
com.google.common.base.Objects across the codebase. This is a small
step in the eventual removal of Guava as a dependency.

Relates #13224
2015-09-05 08:52:30 -04:00
Clinton Gormley 877589e153 In the license checker, include the "ignore prefix" parameter in the output
Closes #13322
2015-09-03 21:00:11 +02:00
Simon Willnauer 796701d52e Move version to 3.0.0-SNAPSHOT 2015-09-03 10:43:28 +02:00
David Pilato 1a8a2c9bc2 [qa] Add smoke test client module
This commit adds a new smoke test for testing client as a end Java user.

It starts a cluster in `pre-integration-test` phase, then execute the client operations defined as JUnit tests within `integration-test` phase and then stop the external cluster in `post-integration-test` phase.

You can also run test classes from your IDE.

* Start an external node on your machine with `bin/elasticsearch` (note that you can test Java API regressions if you run an older or newer node version)
* Run the JUnit test. By default, it will run tests on `localhost:9300` but you can change this setting using system property `tests.cluster`. It also expects the default `cluster.name` (`elasticsearch`).

This commit also starts adding [snippets as defined by Maven](https://maven.apache.org/guides/mini/guide-snippet-macro.html) to help keeping automatically synchronized the Java reference guide with the current code.

Our documentation builder tool does not support snippets though but we will most likely support it at some point.
2015-09-02 17:34:54 +02:00
David Pilato 34ee4c2d66 [build] remove shaded elasticsearch version
The shaded version of elasticsearch was built at the very beginning to avoid dependency conflicts in a specific case where:

* People use elasticsearch from Java
* People needs to embed elasticsearch jar within their own application (as it's today the only way to get a `TransportClient`)
* People also embed in their application another (most of the time older) version of dependency we are using for elasticsearch, such as: Guava, Joda, Jackson...

This conflict issue can be solved within the projects themselves by either upgrade the dependency version and use the one provided by elasticsearch or by shading elasticsearch project and relocating some conflicting packages.

Example
-------

As an example, let's say you want to use within your project `Joda 2.1` but elasticsearch `2.0.0-beta1` provides `Joda 2.8`.
Let's say you also want to run all that with shield plugin.

Create a new maven project or module with:

```xml
<groupId>fr.pilato.elasticsearch.test</groupId>
<artifactId>es-shaded</artifactId>
<version>1.0-SNAPSHOT</version>

<properties>
    <elasticsearch.version>2.0.0-beta1</elasticsearch.version>
</properties>

<dependencies>
    <dependency>
        <groupId>org.elasticsearch</groupId>
        <artifactId>elasticsearch</artifactId>
        <version>${elasticsearch.version}</version>
    </dependency>
    <dependency>
        <groupId>org.elasticsearch.plugin</groupId>
        <artifactId>shield</artifactId>
        <version>${elasticsearch.version}</version>
    </dependency>
</dependencies>
```

And now shade and relocate all packages which conflicts with your own application:

```xml
<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>2.4.1</version>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                    <configuration>
                        <relocations>
                            <relocation>
                                <pattern>org.joda</pattern>
                                <shadedPattern>fr.pilato.thirdparty.joda</shadedPattern>
                            </relocation>
                        </relocations>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>
```

You can create now a shaded version of elasticsearch + shield by running `mvn clean install`.

In your project, you can now depend on:

```xml
<dependency>
    <groupId>fr.pilato.elasticsearch.test</groupId>
    <artifactId>es-shaded</artifactId>
    <version>1.0-SNAPSHOT</version>
</dependency>
<dependency>
    <groupId>joda-time</groupId>
    <artifactId>joda-time</artifactId>
    <version>2.1</version>
</dependency>
```

Build then your TransportClient as usual:

```java
TransportClient client = TransportClient.builder()
        .settings(Settings.builder()
                        .put("path.home", ".")
                        .put("shield.user", "username:password")
                        .put("plugin.types", "org.elasticsearch.shield.ShieldPlugin")
        )
        .build();
client.addTransportAddress(new InetSocketTransportAddress(new InetSocketAddress("localhost", 9300)));

// Index some data
client.prepareIndex("test", "doc", "1").setSource("foo", "bar").setRefresh(true).get();
SearchResponse searchResponse = client.prepareSearch("test").get();
```

If you want to use your own version of Joda, then import for example `org.joda.time.DateTime`. If you want to access to the shaded version (not recommended though), import `fr.pilato.thirdparty.joda.time.DateTime`.

You can run a simple test to make sure that both classes can live together within the same JVM:

```java
CodeSource codeSource = new org.joda.time.DateTime().getClass().getProtectionDomain().getCodeSource();
System.out.println("unshaded = " + codeSource);

codeSource = new fr.pilato.thirdparty.joda.time.DateTime().getClass().getProtectionDomain().getCodeSource();
System.out.println("shaded = " + codeSource);
```

It will print:

```
unshaded = (file:/path/to/joda-time-2.1.jar <no signer certificates>)
shaded = (file:/path/to/es-shaded-1.0-SNAPSHOT.jar <no signer certificates>)
```

This PR also removes fully-loaded module.

By the way, the project can now build with Maven 3.3.3 so we can relax a bit our maven policy.
2015-09-02 11:57:10 +02:00
Jason Tedor a8bace9f97 Remove and forbid final uses of ImmutableList 2015-08-31 14:35:23 -04:00
Martijn Laarman a80317c4b3 cmd /C needs to be quoted as a whole
To support spaces in both the command as well as its arguments cmd needs
be called like this:

cmd /C ""c:\a b\c.bat" "argument 1" "argument2""

ant was running

cmd /C "c:\a b\c.bat" "argument 1" "argument2"

which in windows causes to be preprocessed to

cmd /C c:\a b\c.bat" "argument 1" "argument2

Which would make it appear as though ant was not properly quoting (which
it did sort of).
2015-08-31 13:33:10 +02:00
Jason Tedor 3067cacb66 Remove and forbid use of com.google.common.collect.Lists
This commit removes and now forbids all uses of
com.google.common.collect.Lists across the codebase. This is the first
of many steps in the eventual removal of Guava as a dependency.
2015-08-27 22:14:33 -04:00
Simon Willnauer f64a875e03 use provided version in smoke test file paths 2015-08-27 23:20:01 +02:00
Ryan Ernst 38b8f20cc5 Make 0.x and 1.x indexes still work with get-bwc-version 2015-08-27 10:19:59 -07:00
Ryan Ernst 1029b119d9 Tests: Fix generation scripts for bwc indexes, and add 2.0 beta1 index 2015-08-26 19:36:26 -07:00
Simon Willnauer 5579b8fad9 [BUILD] Add publish instructions to the release script 2015-08-26 16:22:14 +02:00
Simon Willnauer d4a9089253 [BUILD] Allow smoke-testing other artifact but staging artifacts 2015-08-26 16:07:50 +02:00
Britta Weber 12aeafcbed Merge pull request #12961 from brwe/jps-check
check with jps that the pid file contains a pid that actually is an e…
2015-08-26 14:29:13 +02:00
Clinton Gormley 13727ba6fd Release notes: Add upgrades category 2015-08-25 17:16:14 +02:00
Simon Willnauer 3175be7a25 Add smoke-tester to mail template 2015-08-25 12:19:24 +02:00
Simon Willnauer aaca188571 Merge pull request #13078 from s1monw/add_smoke_tester
Add RC smoke tester that checks basic functionality
2015-08-25 10:53:43 +02:00
Nik Everett 0de493e35d Merge pull request #13070 from nik9000/eclipse_fix
Fix some issues developing Elasticsearch with Eclipse
2015-08-24 15:35:50 -04:00
Simon Willnauer 3a01f7b3b4 [BUILD] Python3 doesn't have dict.iter_items - it's replaced by dict.items() 2015-08-24 12:52:09 +02:00
Alexander Reelsen cb68a2196d Release: Prevent running whole lifecycle twice 2015-08-24 10:26:49 +02:00
Nik Everett bb9e77f876 Eclipse: Fix launch configuration
We had a file in dev-tools/ElasticSearch.launch which tried to launch
elasticsearch but failed somewhat epically because of the security manager
and files having moved. This recreates it with
`-Des.security.manager.enabled=false` to get it working again. Its not as nice
as testing with the security manager in place but its better than waiting
minutes for maven to package and startup elasticsearch.
2015-08-21 19:07:03 -07:00
Simon Willnauer 5458d07ea2 Refactor script for RC creation
The script now allows to run all required steps at once and alternatively
prints out manual instructions to run the steps individually. It also has
flags and options to run debug builds from a local checkout.
2015-08-21 14:36:59 +02:00
Jason Tedor 31b80e4f3f Merge pull request #13010 from jasontedor/feature/improve-java-version-comparison
Improve java version comparison and explicitly enforce a version format
2015-08-21 08:30:50 -04:00
Jason Tedor 5ff1f8a058 Enforce version format for java.version for plugins
Currently we implicitly enforce a version format on the java.version
property for plugins via JarHell.checkJavaVersion. We should explicitly
enforce this version format and specify it in the documentation.

This commit adds an explicit enforcement of the version format on the
java.version property for plugins and updates the documentation for
plugin-descriptor.properties accordingly.

Closes #13009
2015-08-21 08:28:41 -04:00
Ryan Ernst 164efaecbe Networking: Move multicast discovery to a plugin
Multicast has known issues (see #12999 and #12993). This change moves
multicast into a plugin, and deprecates it in the docs.  It also allows
for plugging in multiple zen ping implementations.

closes #13019
2015-08-20 16:43:25 -07:00
Robert Muir e2ab62596f Default to unicast discovery, with default host list of 127.0.0.1, [::1]
Fix unicast discovery to work when a host has multiple addresses.
Ban dangerous methods in java.net with forbidden APIs.
Fix ipv6 bugs and formatting of network addresses everywhere.

Closes #12999
Closes #12993

Squashed commit of the following:

commit 6c1aa001d091c5cf25212a53dc701fb704337f1e
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 14:25:43 2015 -0400

    Fix these to be correct with addresses just in case

commit 648215627e84abf58a71400e7dc9ae775efb71d6
Merge: d00561b 41d8fbe
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 13:23:09 2015 -0400

    Merge branch 'master' into unicast_all_the_way_down

commit d00561b76fd1aa5850699f7901f3dae3d4d402b7
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 16:38:50 2015 +0200

    limit local ports to 5 in UnicastZenPing

commit e2e15c594006746cbe24432694294a71cc99deb8
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 10:32:47 2015 -0400

    fix port limiting

commit 10153cb7adadda81a1f482445e703836b65cf5e2
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 10:18:37 2015 -0400

    don't serialize scopeids: that's broken

commit 2aa63d43db2baec68a2e9bc227cfeb85dfeb4f83
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 16:06:51 2015 +0200

    restore @Network

commit c840f1d1ef438826ae1ecfd5e45942a0e30dc9c0
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 16:02:30 2015 +0200

     Use NetworkAddress.formatAddress where applicable in plugins

commit 374ce878852b35d626b7a29c8c4773545b0e9ddd
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 15:34:06 2015 +0200

    Use NetworkAddress.formatAddress where applicable

commit e7a606d63f1bc43c1b62b6e17adf707c76d43a15
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 10:17:57 2015 +0200

    Add @Multicast annotation to disable multicast tests by default.

    We only run multicast tests now when we explicitly state it. A working
    multicast env is required which is not always the case.

commit 2d7d2d0347179696ab41f71f048b13305014c85b
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 09:51:28 2015 +0200

    Remove extra check for local mode in InternalTestCluster

commit dda59ac39aa136d4687b9274c2692cd77f8b8f66
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 09:37:03 2015 +0200

    Handle node mode across entire test cluster

    We used static methods reading sys properties to define the node mode
    per cluster. this had lots of problems when tests couldn't cope with
    mixed or only local mode. Now we are passing it down to the cluster from the test
    which allows to @SuppressNetworkMode / @SupressLocalMode on the test to force
    consistent node configurations.

commit 058197b7a408318995c88ce7f6762e32348de0de
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 03:19:14 2015 -0400

    really ban InetSocketAddress's trappy method and break build and go to sleep, sorry

commit ac8779185aee1e17e6f5a81766290fdfc9c603ba
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 03:16:52 2015 -0400

    Ban methods that might surprisingly cause DNS lookups

commit e64fe3dff2b11503e5f2831eb9863d64f56c5538
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 02:59:05 2015 -0400

    Add unit test

commit f15434f20fb1a3691b1cc16028597d8fae937e05
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 02:39:02 2015 -0400

    fix ipv6 formatting bugs

commit 05c2c74098052c75fbb79ea1818a295ef2e03e30
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 02:12:05 2015 -0400

    format addresses correctly so I can actually read what comes out of our logs and stats apis

commit 4f9389dcf1e8925f23153c5eb271b4ce2294dbaf
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Aug 19 21:26:52 2015 -0400

    ban dangerous methods in java.net

commit 6aacd4d9925f324903d1d099a6cf5f862aeaf677
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Aug 19 20:59:24 2015 -0400

    ban lenient method

commit f466a842c60163d1f4554bdce8a4163edb534c2c
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 00:29:00 2015 +0200

    fix tests to not mix local transport and zen unicast disco

commit 0de007a33b33fb68cf85cd86db4ca4f8ce10bbc9
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 00:10:07 2015 +0200

    fix tests to not mix local transport and zen unicast disco

commit 539f6ca6e5137e0d496239adc8684688dedcc824
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 00:02:01 2015 +0200

    fix tests to not mix local transport and zen unicast disco

commit 004c2881b25467f332acc8c9f9e92b1f0f9d314e
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Aug 19 17:51:45 2015 -0400

    Fix multinode

commit 54113af325ce31571811c49fdaae89d5687be4ba
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Aug 19 17:36:45 2015 -0400

    fix integration tests

commit 0156a77a56319d6b9737ec6a531992052e50bd59
Author: Simon Willnauer <simonw@apache.org>
Date:   Wed Aug 19 23:32:18 2015 +0200

    enable multicast in MulticastZenPingIT.java

commit 1791caa35da853ce0122485fa3fd4674c671ec6e
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Aug 19 17:23:16 2015 -0400

    Fix constant

commit 22820b53e0b2dc9fd47145c2bc29ce912a8fd484
Author: Simon Willnauer <simonw@apache.org>
Date:   Wed Aug 19 22:59:09 2015 +0200

    give it some extra ids for local transport crazyness

commit b2138fafa94a8a085813fd48356df63e57ade5b3
Author: Simon Willnauer <simonw@apache.org>
Date:   Wed Aug 19 22:51:42 2015 +0200

    pass on local addresses from configured transport rather than hard code IP addresses

commit 1bf5de1f457b081e0ce262b57d2b55d39c434156
Author: Simon Willnauer <simonw@apache.org>
Date:   Wed Aug 19 22:04:31 2015 +0200

    fix PluggableTransportModuleIT.java to use local disco and detach port limit for node local disco

commit b6706eddfa04c43947c16551359ae98a463d34aa
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Aug 19 14:16:03 2015 -0400

    Default to unicast discovery, with default host list of 127.0.0.1, [::1]
2015-08-20 14:26:40 -04:00
Britta Weber c0d31a2c9e Merge pull request #12965 from brwe/es-running-already
fail with better error message if elasticsearch was started already
2015-08-19 17:56:56 +02:00
Britta Weber 4cc6359756 fail with better error message if elasticsearch was started already
Before it failed with
.../es/qa/smoke-test-plugins/target/integ-tests/es.pid doesn't exist
which was confusing.
2015-08-19 17:56:05 +02:00
Christoph Büscher 8454d49552 Plugins: Add 'name' property to plugin descriptor file to determine plugin name
At the moment, when installing from an url, a user provides the plugin name on
the command line like:

* bin/plugin install [plugin-name] --url [url]

This can lead to problems when picking an already existing name from another
plugin, and can potentially overwrite plugins already installed with that name.

This, this PR introduces a mandatory `name` property to the plugin descriptor
file which replaces the name formerly provided by the user.

With the addition of the `name` property to the plugin descriptor file, the user
does not need to specify the plugin name any longer when installing from a file
or url. Because of this, all arguments to `plugin install` command are now
either treated as a symbolic name, a URL or a file without the need to specify
this with an explicit option.

The new syntax for `plugin install` is now:

bin/plugin install [name or url]

* downloads official plugin
bin/plugin install analysis-kuromoji

* downloads github plugin
bin/plugin install lmenezes/elasticsearch-kopf

* install from URL or file
bin/plugin install http://link.to/foo.zip
bin/plugin install file:/path/to/foo.zip

If the argument does not parse to a valid URL, it is assumed to be a name and the
download location is resolved like before. Regardless of the source location of
the plugin, it is extracted to a temporary directory and the `name` property from
the descriptor file is used to determine the final install location.

Relates to #12715
2015-08-19 12:13:01 +02:00
Britta Weber 04141b61c4 make property local and use java home to find jps 2015-08-19 01:32:55 +02:00
Robert Muir e07f039659 Workaround JDK bug 8034057
This causes a FileSystemException when trying to retrieve FileStore for a path,
and falsely returns false for Files.isWritable

Squashed commit of the following:

commit d2cc0d966f3bc94aa836b316a42b3c5724bc01ef
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Aug 18 15:49:48 2015 -0400

    fixes for the non-bogus comments

commit 6e0a272f5f8ef7358654ded8ff4ffc31831fa5c7
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Aug 18 15:30:43 2015 -0400

    Fix isWritable too

commit 2a8764ca118fc4c950bfc60d0b97de873e0e82ad
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Aug 18 14:49:50 2015 -0400

    try to workaround filestore bug
2015-08-18 16:44:57 -04:00
David Pilato 0599f85d2d [build] simplify ant script for plugins
Now we are using short names for artifactId (see #12879) so we don't need anymore to transform long names `elasticsearch-pluginname` to short names `pluginname` in ant script when we install a plugin.
Modify also convert-plugin-name
Clean up remaining plugins with old format
And fix vagrant tests
2015-08-18 19:41:37 +02:00
Simon Willnauer 1e511eda28 Remove usage or `InetAddress#getLocalHost`
this method is very confusing and if it's used it's likely the wrong thing
with respect to the actual bound / published address. This change discourages
it's use and removes all useage. It's replaced with the actual published address
most of the time.
2015-08-18 16:48:34 +02:00
Britta Weber f031c17681 check with jps that the pid file contains a pid that actually is an elasticsearch process
relates to #12063
2015-08-18 16:24:40 +02:00
David Pilato c189c1a1ec [maven] also rename artifactId for dev-tools and rest-api-spec 2015-08-18 13:41:12 +02:00
David Pilato 0014728a0d [maven] rename maven names for parent, dev-tools and rest spec
Follow up for previous commit:

```
[INFO] Build Tools and Resources .......................... SUCCESS [  0.158 s]
[INFO] Rest API Specification ............................. SUCCESS [  0.002 s]
[INFO] Elasticsearch Parent POM ........................... SUCCESS [  0.122 s]
```
2015-08-18 13:38:49 +02:00
David Pilato 2b6c5762f4 [build] display ignored artifact when checking licenses 2015-08-18 13:38:48 +02:00
Robert Muir 68307aa9f3 Fix network binding for ipv4/ipv6
When elasticsearch is configured by interface (or default: loopback interfaces),
bind to all addresses on the interface rather than an arbitrary one.

If the publish address is not specified, default it from the bound addresses
based on the following sort ordering:

* ipv4/ipv6 (java.net.preferIPv4Stack, defaults to true)
* ordinary addresses
* site-local addresses
* link local addresses
* loopback addresses

One one address is published, and multicast is still always over ipv4: these
need to be future improvements.

Closes #12906
Closes #12915

Squashed commit of the following:

commit 7e60833312f329a5749f9a256b9c1331a956d98f
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 14:45:33 2015 -0400

    fix java 7 compilation oops

commit c7b9f3a42058beb061b05c6dd67fd91477fd258a
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 14:24:16 2015 -0400

    Cleanup/fix logic around custom resolvers

commit bd7065f1936e14a29c9eb8fe4ecab0ce512ac08e
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 13:29:42 2015 -0400

    Add some unit tests for utility methods

commit 0faf71cb0ee9a45462d58af3d1bf214e8a79347c
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 12:11:48 2015 -0400

    localhost all the way down

commit e198bb2bc0d1673288b96e07e6e6ad842179978c
Merge: b55d092 b93a75f
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 12:05:02 2015 -0400

    Merge branch 'master' into network_cleanup

commit b55d092811d7832bae579c5586e171e9cc1ebe9d
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 12:03:03 2015 -0400

    fix docs, fix another bug in multicast (publish host = bad here!)

commit 88c462eb302b30a82585f95413927a5cbb7d54c4
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 11:50:49 2015 -0400

    remove nocommit

commit 89547d7b10d68b23d7f24362e1f4782f5e1ca03c
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 11:49:35 2015 -0400

    fix http too

commit 9b9413aca8a3f6397b5031831f910791b685e5be
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 11:06:02 2015 -0400

    Fix transport / interface code

    Next up: multicast and then http
2015-08-17 15:43:07 -04:00
Alexander Reelsen f95a509538 Release script: Set versions for non inherited projects
rest-api-spec and dev-tools dont have the elasticsearch-parent
set as a parent and thus need a separate mvn run to change the
plugin version.
2015-08-17 16:40:29 +02:00
Alexander Reelsen a75adaaaaa Prepare release script: fix python compilation error 2015-08-17 15:52:36 +02:00
Alexander Reelsen d1c93fb573 Release: Remove aws-maven plugin/improve release docs
In order to have consistent deploys across several repositories,
we should deploy to sonatype first, then mirror those contents,
and then upload to s3.

This means, the aws wagon is not needed anymore.
2015-08-17 15:39:22 +02:00
Alexander Reelsen c3b3b0e6f8 Release: Replace python search/replace with mvn versions:set plugin
mvn has a versions:set plugin, that can be easily invoked and does not
require the python script to parse the files and hope that there are no
other snapshot mentions.
2015-08-17 11:47:24 +02:00
Simon Willnauer e7d075f6ae Add elasticsearch version as a prefix for the staging URL
This is purely for maintainance reasons since it easier to see if we can drop
certain stageing urls if we have the version next to the hash.
I also removed the gpg passphrase from the example URL since it's better to get prompted?
2015-08-17 11:05:49 +02:00
Simon Willnauer b447e2ae99 Move master to [2.1.0-SNAPSHOT] 2015-08-14 23:44:06 +02:00
Lee Hinman 178dc7ebaf Revert "Add no-Charset version of Files.readAllLines to forbidden apis"
This reverts commit 88765320a9.
2015-08-14 15:05:24 -06:00
Lee Hinman 88765320a9 Add no-Charset version of Files.readAllLines to forbidden apis 2015-08-14 14:55:22 -06:00
Alexander Reelsen e869c87119 Build: Adding aws-maven extension to dev-tools and rest-api-spec 2015-08-14 17:30:00 +02:00
Alexander Reelsen 3893544f98 Release scripts: Split prepare_release into two scripts
In order to reflect our RC release process, we need to separate
the prepare_release script into two separate scripts.

One script now updates the documentation. That one can be executed
anytime and needs to be pushed after that.

The other script updates the version in Version.java and all pom.xml
files, but does not commit anymore. This allows to create a non snapshot
version locally, run mvn deploy, push the artifacts into S3 and, upon
successful tests, simply release them on sonatype. This also allows for
updates, because the S3 snapshot will include the commitId in their repo
as already pushed before.
2015-08-14 16:44:39 +02:00
Clinton Gormley 1eaa8ee7b3 Add an optional "prefix_to_ignore" parameter to the license checker
If core plugins are to be renamed to not include the "elasticsearch-"
prefix, then we need a way of telling the license checker which
JAR files to ignore.
2015-08-14 15:41:05 +02:00
Simon Willnauer c1405d26b5 fixtypo 2015-08-14 15:27:23 +02:00