Commit Graph

704 Commits

Author SHA1 Message Date
Ryan Ernst 606cc456d2 disable unit tests until they work 2015-09-16 10:26:01 -07:00
Ryan Ernst 7eef7e42d4 quiet flakey gce tests 2015-09-16 09:59:38 -07:00
Christoph Büscher ff74e94260 Merge branch 'master' into feature/query-refactoring
Conflicts:
    core/src/main/java/org/elasticsearch/common/unit/Fuzziness.java
    core/src/main/java/org/elasticsearch/index/mapper/internal/IdFieldMapper.java
    core/src/main/java/org/elasticsearch/index/query/IdsQueryParser.java
    core/src/main/java/org/elasticsearch/index/query/functionscore/random/RandomScoreFunctionParser.java
    core/src/test/java/org/elasticsearch/search/highlight/HighlighterSearchIT.java
2015-09-16 13:04:18 +02:00
Ryan Ernst 45f757de6d Test: Move rest-api-spec for plugins into test resources
Plugin tests require having rest-api tests, and currently copy that spec
from a directory in the root of the plugin source into the test
resources. This change moves the rest-api-spec dir into test resources
so it is like any other test resources. It also removes unnecessary
configuration for resources from the shared plugin pom.
2015-09-16 03:04:53 -07:00
Robert Muir 01e6d8e3dc Remove java.lang.reflect.ReflectPermission "suppressAccessChecks"
Closes #13603

Squashed commit of the following:

commit 8799fb42d80297a79285beaf407b1bbecdb5854d
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Sep 16 03:32:29 2015 -0400

    Add randomizedtesting snapshot note

commit 0d874d9f0f5fddaeab8f48f9816a052dcaa691be
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Sep 16 03:11:01 2015 -0400

    Add a mechanism for insecure plugins and get all tests passing

commit 80540aeb9a264f6f299aaa3bc89df7f9b7923a60
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Sep 15 22:59:29 2015 -0400

    Really remove, we are killing this

commit 884818c1ad44ca2e7572a6998c086580be919657
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Sep 15 22:57:22 2015 -0400

    fill in TODOs

commit 34f4cb81f249edfec4d8d211da892f8c987e5948
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Sep 15 22:31:43 2015 -0400

    Publish snapshots of RR and lucene and cutover

commit d68eb9d66ce059761805c64d67e41a29098c9afa
Merge: f27e208 f62da59
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Sep 15 12:32:41 2015 -0400

    Merge branch 'master' into kill-setaccessible

commit f27e20855216dab6a6ad035d41018d8c67f3144c
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Sep 15 12:32:21 2015 -0400

    make a real lucene snapshot
2015-09-16 04:08:31 -04:00
Simon Willnauer bfa984e1c2 Prevent losing stacktraces when exceptions occur
This commit removes unnecesssary use of ExceptionHelpers where we actually
should serialize / deserialize the actual exception. This commit also
fixes one of the oddest problems where the actual exception was never
rendered / printed if `all shards failed` due to a missing cause.

This commit unfortunately doesn't fix Snapshot/Restore which is almost
unfixable since it has to serialize XContent and read from it which can't
transport exceptions.
2015-09-15 23:20:54 +02:00
Pedro Andujar 3961de3e56 Enable SSL for blob storage
In order to increase Security.

The Microsoft Azure storage services support both HTTP and HTTPS; however, using HTTPS is highly recommended.
2015-09-15 14:25:56 +01:00
David Pilato a38bcc5d62 [test] plugins simple RestIT tests don't work from IDE
When running a RestIT test from the IDE, you actually start an internal node which does not automatically load the plugin you would like to test.

We need to add:

```java
    @Override
    protected Collection<Class<? extends Plugin>> nodePlugins() {
        return pluginList(PLUGIN_HERE.class);
    }
```

Everything works fine when running from maven because each test basically:

* installs elasticsearch
* installs one plugin
* starts elasticsearch with this plugin loaded
* runs the test

Note that this PR only fixes the fact we run an internal cluster with the expected plugin.

Cloud tests will still fail when run from the IDE because is such a case you actually start an internal node with many mock plugins.
And REST test suite for cloud plugins basically checks if the plugin is running by checking the output of NodesInfo API.

And we check:

```yml
- match:  { nodes.$master.plugins.0.name: cloud-azure  }
- match:  { nodes.$master.plugins.0.jvm: true  }
```

But in that case, this condition is certainly false as we started also `mock-transport-service`, `mock-index-store`, `mock-engine-factory`, `node-mocks`, `asserting-local-transport`, `mock-search-service`.

Closes #13479
2015-09-15 10:10:05 +02:00
Christoph Büscher 90f24c1a79 Merge branch 'master' into feature/query-refactoring
Conflicts:
	core/src/main/java/org/elasticsearch/index/query/GeoDistanceRangeQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/GeoPolygonQueryBuilder.java
	core/src/main/java/org/elasticsearch/index/query/GeohashCellQuery.java
	core/src/main/java/org/elasticsearch/index/query/WrapperQueryBuilder.java
	core/src/test/java/org/elasticsearch/search/child/ChildQuerySearchIT.java
2015-09-14 14:31:01 +02:00
Simon Willnauer 40959068d5 Remove and forbid use of guava Function, Charsets, Collections2
This commit removes and now forbids all uses of
Function, Charsets, Collections2  across the codebase. This
is one of many steps in the eventual removal of Guava as a dependency.

Relates #13224
2015-09-14 10:27:12 +02:00
Simon Willnauer 95c83ba2b8 Fix compilation error 2015-09-14 09:29:41 +02:00
Simon Willnauer a0c10850d9 Use Supplier instead of Reflection
Java 8 allows for method references which in-turn will cause
compile errors if a method is not visible while reflection fails late
and maybe too late. We can now register Request instances via FooRequest::new
instead of passing FooRequest.class and call it's ctor via reflection.
2015-09-14 09:11:32 +02:00
Robert Muir 0b16552367 Revert "Disable this test (we should fix rest framework to disable based on java versions)"
This reverts commit 6a0381b582.
2015-09-12 12:05:47 -04:00
Robert Muir d6f56030d8 ban setAccessible from core code.
In addition to being a big security problem, setAccessible is a risk
for java 9 migration. We need to clean up our code so we can ban it
and eventually enforce this with security manager for third-party code, too,
or we may have problems.

Instead of using setAccessible, use the correct modifier (e.g. public).

TODO: ban in tests
TODO: ban in security manager at runtime
2015-09-12 02:11:06 -04:00
Igor Motov 39ca45050a Simplify the BlobContainer blob writing interface
Instead of asking blob store to create output for posting blob content, this change provides that content of the blob to the blob store for writing. This will significantly simplify the  interface for S3 and Azure plugins.
2015-09-11 16:39:58 -04:00
javanna 73f7df510e Merge branch 'master' into feature/query-refactoring
Conflicts:
	core/src/main/java/org/elasticsearch/index/query/HasChildQueryBuilder.java
	core/src/main/java/org/elasticsearch/index/query/HasChildQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/HasParentQueryBuilder.java
	core/src/main/java/org/elasticsearch/index/query/HasParentQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/functionscore/FunctionScoreQueryBuilder.java
	core/src/main/java/org/elasticsearch/index/query/functionscore/FunctionScoreQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/functionscore/factor/FactorParser.java
	core/src/test/java/org/elasticsearch/ExceptionSerializationTests.java
	core/src/test/java/org/elasticsearch/benchmark/search/child/ChildSearchBenchmark.java
	core/src/test/java/org/elasticsearch/benchmark/search/child/ChildSearchShortCircuitBenchmark.java
	core/src/test/java/org/elasticsearch/index/query/SimpleIndexQueryParserTests.java
	core/src/test/java/org/elasticsearch/percolator/PercolatorIT.java
	core/src/test/java/org/elasticsearch/search/child/ChildQuerySearchIT.java
	docs/reference/query-dsl/has-parent-query.asciidoc
2015-09-11 14:15:12 +02:00
Simon Willnauer 6b4699fbf7 Only add unicast provider if all settings are present
Closes #13492
2015-09-11 13:21:03 +02:00
David Pilato 163c34127f [azure] Revert discovery plugins must declare their UnicastHostsProvider
Related to #13492

The previous fix is not working for Azure Plugin.
I will work on a new fix for it but for now, I'm reverting the change for azure.
2015-09-11 10:25:32 +02:00
Adrien Grand e988a9edc6 Merge pull request #13439 from rmuir/1702090
upgrade lucene to r1702265
2015-09-11 09:48:43 +02:00
David Pilato e8cad378f3 [ec2/azure] discovery plugins must declare their UnicastHostsProvider
Closes #13492
(cherry picked from commit 1e233b4)
(cherry picked from commit 0fe4131)
2015-09-11 09:32:39 +02:00
Robert Muir 6a0381b582 Disable this test (we should fix rest framework to disable based on java versions) 2015-09-11 00:13:42 -04:00
Robert Muir c1f2fc76c2 Upgrade lucene to r1702090
The semantics of the `boost` parameter for `function_score` changed. This is
due to the fact that Lucene now requires that query boosts and top-level boosts
are applied the same way.
2015-09-10 23:36:43 +02:00
javanna 56b3db6ba3 Merge branch 'master' into feature/query-refactoring
Conflicts:
	core/src/main/java/org/elasticsearch/index/query/AndQueryBuilder.java
	core/src/main/java/org/elasticsearch/index/query/AndQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/FQueryFilterParser.java
	core/src/main/java/org/elasticsearch/index/query/FilteredQueryBuilder.java
	core/src/main/java/org/elasticsearch/index/query/FilteredQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/HasChildQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/IndicesQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/LimitQueryBuilder.java
	core/src/main/java/org/elasticsearch/index/query/LimitQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/MultiMatchQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/OrQueryBuilder.java
	core/src/main/java/org/elasticsearch/index/query/OrQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/QueryFilterBuilder.java
	core/src/main/java/org/elasticsearch/index/query/QueryFilterParser.java
	core/src/main/java/org/elasticsearch/index/query/QueryParseContext.java
	core/src/main/java/org/elasticsearch/index/query/ScriptQueryParser.java
	core/src/main/java/org/elasticsearch/indices/query/IndicesQueriesRegistry.java
	core/src/main/java/org/elasticsearch/search/sort/GeoDistanceSortParser.java
	core/src/main/java/org/elasticsearch/search/sort/SortParseElement.java
	core/src/test/java/org/elasticsearch/benchmark/search/child/ChildSearchShortCircuitBenchmark.java
	core/src/test/java/org/elasticsearch/index/query/SimpleIndexQueryParserTests.java
2015-09-10 15:52:28 +02:00
Jason Tedor 6b19aebab0 Merge pull request #13438 from jasontedor/maps-be-gone
Remove and forbid use of com.google.common.collect.Maps
2015-09-09 17:58:44 -04:00
Jason Tedor 2a5412ebf0 Remove and forbid use of com.google.common.collect.Maps
This commit removes and now forbids all uses of
com.google.common.collect.Maps across the codebase. This is one of many
steps in the eventual removal of Guava as a dependency.

Relates #13224
2015-09-09 17:41:41 -04:00
Ryan Ernst 9e8a90a657 Add xlint ignores for warning classes, where appropriate. 2015-09-09 12:47:07 -07:00
David Pilato b89efbed69 [maven] assembly: elasticsearch is already excluded
As elasticsearch is marked as provided we don't need to explicitly exclude it from the assembly descriptor.

We get a warning today for all plugins, the following:

```
[INFO] --- maven-assembly-plugin:2.5.5:single (default) @ repository-s3 ---
[INFO] Reading assembly descriptor: /path/to/plugin-assembly.xml
[WARNING] The following patterns were never triggered in this artifact exclusion filter:
o  'org.elasticsearch:elasticsearch'
[INFO] Building zip: /path/to/target/releases/repository-s3-3.0.0-SNAPSHOT.zip
[INFO]
```

It now gives:

```
[INFO] --- maven-assembly-plugin:2.5.5:single (default) @ repository-s3 ---
[INFO] Reading assembly descriptor: /path/to/plugin-assembly.xml
[INFO] Building zip: /path/to/target/releases/repository-s3-3.0.0-SNAPSHOT.zip
[INFO]
```
2015-09-08 08:55:31 +02:00
Adrien Grand 0c26e7cd83 Remove the scan and count search types.
These search types have been deprecated in 2.1 and 2.0 respectively, and will
be removed in 3.0.
2015-09-07 15:18:45 +02:00
Jason Tedor 8a3411e5e4 Remove and forbid the use of com.google.common.base.Predicate(s)?
This commit removes and now forbids all uses of
com.google.common.base.Predicate and com.google.common.base.Predicates
across the codebase. This is one of the many steps in the eventual
removal of Guava as a dependency. This was enabled by #13314.

Relates #13224
2015-09-06 07:20:24 -04:00
xuzha e69551d8df This commit remove the block for user to set invalid Signer Type.
Instead of throw an exception, this change log a warning message
and accept any signer type.
2015-09-05 22:57:05 -07:00
Robert Muir f216d92d19 Upgrade to lucene 5.4-snapshot r1701068 2015-09-03 15:13:33 -04:00
David Pilato 6953193f5b [test] fix test names
For aws s3 and ec2
2015-09-03 12:21:26 +02:00
David Pilato 30aa231f8e [plugin] split cloud-aws in repository-s3 and discovery-ec2
Until now we had a cloud-aws plugin which is providing 2 disctinct features:

* discovery on EC2
* snapshot/restore on S3

This commit splits the plugin by feature so people can use either one or the other or both features.

Doc is updated accordingly.
2015-09-03 11:12:20 +02:00
Simon Willnauer 796701d52e Move version to 3.0.0-SNAPSHOT 2015-09-03 10:43:28 +02:00
Ryan Ernst 06d5e24548 Tests: Rename test suffix so we only use "Tests"
We currently have a small number of test classes with the suffix "Test",
yet most use the suffix "Tests". This change renames all the "Test"
classes, so that we have a simple rule: "Non-inner classes ending with
Tests".
2015-09-02 14:38:43 -07:00
Ryan Ernst fabadc6a67 Fix naming for plugins abstract test cases
See #13282
2015-09-02 12:33:53 -07:00
Adrien Grand c6d282f9f6 Remove extra licenses 2015-09-01 17:44:57 +02:00
Adrien Grand 3619cce53b Update licenses for analysis plugins. 2015-09-01 15:21:49 +02:00
Ryan Ernst 6295f8e795 Merge branch 'master' into tell_me_your_plugins 2015-08-30 14:20:54 -07:00
Jason Tedor 3067cacb66 Remove and forbid use of com.google.common.collect.Lists
This commit removes and now forbids all uses of
com.google.common.collect.Lists across the codebase. This is the first
of many steps in the eventual removal of Guava as a dependency.
2015-08-27 22:14:33 -04:00
Clinton Gormley 3599d55637 REST spec: Fixed link to delete-by-query docs
Closes #13089
2015-08-25 11:57:55 +02:00
David Pilato 8c4cecb41b [cloud-aws] Update AWS SDK to 1.10.12
Release notes:

* [1.10.11](http://aws.amazon.com/releasenotes/Java/5199590350929641)
* [1.10.12](http://aws.amazon.com/releasenotes/Java/6635368276326731)
2015-08-24 23:06:28 +02:00
David Pilato 528b4389ac [cloud-aws] Update AWS SDK to 1.10.10
Update SHA1 files.

Closes #12859.
2015-08-24 22:58:10 +02:00
Fabian Selles Rosa 3efa6419f1 [cloud-aws] Update AWS SDK to 1.10.10
Closes #12859.
2015-08-24 22:50:57 +02:00
Ryan Ernst 89e1a0fb7d Plugins: Removed plugin.types
The setting `plugin.types` is currently used to load plugins from the
classpath. This is necessary in tests, as well as the transport client.

This change removes the setting, and replaces it with the ability to
directly add plugins when building a transport client, as well as
infrastructure in the integration tests to specify which plugin classes
should be loaded on each node.
2015-08-22 01:37:43 -07:00
xuzha 2276ff6283 Remove `cloud.account` and `cloud.key` settings
closes #12809
2015-08-21 00:56:15 -07:00
Simon Willnauer 47a0296211 Clear ScrollID / NextScrollID even if the first query returned no docs 2015-08-21 09:51:30 +02:00
Ryan Ernst 0c09bf2b18 Remove outdated comment 2015-08-20 22:30:26 -07:00
Ryan Ernst 164efaecbe Networking: Move multicast discovery to a plugin
Multicast has known issues (see #12999 and #12993). This change moves
multicast into a plugin, and deprecates it in the docs.  It also allows
for plugging in multiple zen ping implementations.

closes #13019
2015-08-20 16:43:25 -07:00
Robert Muir e2ab62596f Default to unicast discovery, with default host list of 127.0.0.1, [::1]
Fix unicast discovery to work when a host has multiple addresses.
Ban dangerous methods in java.net with forbidden APIs.
Fix ipv6 bugs and formatting of network addresses everywhere.

Closes #12999
Closes #12993

Squashed commit of the following:

commit 6c1aa001d091c5cf25212a53dc701fb704337f1e
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 14:25:43 2015 -0400

    Fix these to be correct with addresses just in case

commit 648215627e84abf58a71400e7dc9ae775efb71d6
Merge: d00561b 41d8fbe
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 13:23:09 2015 -0400

    Merge branch 'master' into unicast_all_the_way_down

commit d00561b76fd1aa5850699f7901f3dae3d4d402b7
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 16:38:50 2015 +0200

    limit local ports to 5 in UnicastZenPing

commit e2e15c594006746cbe24432694294a71cc99deb8
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 10:32:47 2015 -0400

    fix port limiting

commit 10153cb7adadda81a1f482445e703836b65cf5e2
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 10:18:37 2015 -0400

    don't serialize scopeids: that's broken

commit 2aa63d43db2baec68a2e9bc227cfeb85dfeb4f83
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 16:06:51 2015 +0200

    restore @Network

commit c840f1d1ef438826ae1ecfd5e45942a0e30dc9c0
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 16:02:30 2015 +0200

     Use NetworkAddress.formatAddress where applicable in plugins

commit 374ce878852b35d626b7a29c8c4773545b0e9ddd
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 15:34:06 2015 +0200

    Use NetworkAddress.formatAddress where applicable

commit e7a606d63f1bc43c1b62b6e17adf707c76d43a15
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 10:17:57 2015 +0200

    Add @Multicast annotation to disable multicast tests by default.

    We only run multicast tests now when we explicitly state it. A working
    multicast env is required which is not always the case.

commit 2d7d2d0347179696ab41f71f048b13305014c85b
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 09:51:28 2015 +0200

    Remove extra check for local mode in InternalTestCluster

commit dda59ac39aa136d4687b9274c2692cd77f8b8f66
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 09:37:03 2015 +0200

    Handle node mode across entire test cluster

    We used static methods reading sys properties to define the node mode
    per cluster. this had lots of problems when tests couldn't cope with
    mixed or only local mode. Now we are passing it down to the cluster from the test
    which allows to @SuppressNetworkMode / @SupressLocalMode on the test to force
    consistent node configurations.

commit 058197b7a408318995c88ce7f6762e32348de0de
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 03:19:14 2015 -0400

    really ban InetSocketAddress's trappy method and break build and go to sleep, sorry

commit ac8779185aee1e17e6f5a81766290fdfc9c603ba
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 03:16:52 2015 -0400

    Ban methods that might surprisingly cause DNS lookups

commit e64fe3dff2b11503e5f2831eb9863d64f56c5538
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 02:59:05 2015 -0400

    Add unit test

commit f15434f20fb1a3691b1cc16028597d8fae937e05
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 02:39:02 2015 -0400

    fix ipv6 formatting bugs

commit 05c2c74098052c75fbb79ea1818a295ef2e03e30
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Aug 20 02:12:05 2015 -0400

    format addresses correctly so I can actually read what comes out of our logs and stats apis

commit 4f9389dcf1e8925f23153c5eb271b4ce2294dbaf
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Aug 19 21:26:52 2015 -0400

    ban dangerous methods in java.net

commit 6aacd4d9925f324903d1d099a6cf5f862aeaf677
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Aug 19 20:59:24 2015 -0400

    ban lenient method

commit f466a842c60163d1f4554bdce8a4163edb534c2c
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 00:29:00 2015 +0200

    fix tests to not mix local transport and zen unicast disco

commit 0de007a33b33fb68cf85cd86db4ca4f8ce10bbc9
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 00:10:07 2015 +0200

    fix tests to not mix local transport and zen unicast disco

commit 539f6ca6e5137e0d496239adc8684688dedcc824
Author: Simon Willnauer <simonw@apache.org>
Date:   Thu Aug 20 00:02:01 2015 +0200

    fix tests to not mix local transport and zen unicast disco

commit 004c2881b25467f332acc8c9f9e92b1f0f9d314e
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Aug 19 17:51:45 2015 -0400

    Fix multinode

commit 54113af325ce31571811c49fdaae89d5687be4ba
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Aug 19 17:36:45 2015 -0400

    fix integration tests

commit 0156a77a56319d6b9737ec6a531992052e50bd59
Author: Simon Willnauer <simonw@apache.org>
Date:   Wed Aug 19 23:32:18 2015 +0200

    enable multicast in MulticastZenPingIT.java

commit 1791caa35da853ce0122485fa3fd4674c671ec6e
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Aug 19 17:23:16 2015 -0400

    Fix constant

commit 22820b53e0b2dc9fd47145c2bc29ce912a8fd484
Author: Simon Willnauer <simonw@apache.org>
Date:   Wed Aug 19 22:59:09 2015 +0200

    give it some extra ids for local transport crazyness

commit b2138fafa94a8a085813fd48356df63e57ade5b3
Author: Simon Willnauer <simonw@apache.org>
Date:   Wed Aug 19 22:51:42 2015 +0200

    pass on local addresses from configured transport rather than hard code IP addresses

commit 1bf5de1f457b081e0ce262b57d2b55d39c434156
Author: Simon Willnauer <simonw@apache.org>
Date:   Wed Aug 19 22:04:31 2015 +0200

    fix PluggableTransportModuleIT.java to use local disco and detach port limit for node local disco

commit b6706eddfa04c43947c16551359ae98a463d34aa
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Aug 19 14:16:03 2015 -0400

    Default to unicast discovery, with default host list of 127.0.0.1, [::1]
2015-08-20 14:26:40 -04:00
Adrien Grand 6fa258b8fa Deprecate the `scan` search type.
This commit deprecates the `scan` search type in favour of regular scroll
requests sorted by `_doc`.

Related to #12983
2015-08-20 12:47:23 +02:00
Igor Motov dc82262db6 Don't show access_key and filter_key in S3 repository settings
In #11265 we added an ability to filter out sensitive repository settings. This commit uses this change to filter out access_key and filter_key in S3 repository settings.

Closes elastic/elasticsearch-cloud-aws#184
2015-08-19 16:21:42 -04:00
Christoph Büscher 8454d49552 Plugins: Add 'name' property to plugin descriptor file to determine plugin name
At the moment, when installing from an url, a user provides the plugin name on
the command line like:

* bin/plugin install [plugin-name] --url [url]

This can lead to problems when picking an already existing name from another
plugin, and can potentially overwrite plugins already installed with that name.

This, this PR introduces a mandatory `name` property to the plugin descriptor
file which replaces the name formerly provided by the user.

With the addition of the `name` property to the plugin descriptor file, the user
does not need to specify the plugin name any longer when installing from a file
or url. Because of this, all arguments to `plugin install` command are now
either treated as a symbolic name, a URL or a file without the need to specify
this with an explicit option.

The new syntax for `plugin install` is now:

bin/plugin install [name or url]

* downloads official plugin
bin/plugin install analysis-kuromoji

* downloads github plugin
bin/plugin install lmenezes/elasticsearch-kopf

* install from URL or file
bin/plugin install http://link.to/foo.zip
bin/plugin install file:/path/to/foo.zip

If the argument does not parse to a valid URL, it is assumed to be a name and the
download location is resolved like before. Regardless of the source location of
the plugin, it is extracted to a temporary directory and the `name` property from
the descriptor file is used to determine the final install location.

Relates to #12715
2015-08-19 12:13:01 +02:00
Ryan Ernst 54ae55662c Remove leftover class and fix compile issues from master merge 2015-08-18 14:28:03 -07:00
Ryan Ernst 7393068417 Merge branch 'master' into construct_it_yourself 2015-08-18 14:15:08 -07:00
Simon Willnauer 0ffd99cca3 Drop commons-lang dependency
commons-lang really is only used by some core classes to join strings or modiy arrays.
It's not worth carrying the dependency. This commit removes the dependency on commons-lang
entirely.
2015-08-18 22:59:31 +02:00
Ryan Ernst c0444ccbc6 Merge branch 'master' into module_culling3 2015-08-18 10:16:40 -07:00
Ryan Ernst c3a22e6f0e Merge branch 'master' into construct_it_yourself 2015-08-18 09:50:47 -07:00
David Pilato 807d35e96f [maven] change murmur3 plugin groupId and name 2015-08-18 13:45:59 +02:00
David Pilato 692cc80523 [maven] also rename parent project artifactId
Also fixed bad scm links
2015-08-18 13:41:12 +02:00
David Pilato d21afc8090 [maven] rename artifactIds from `elasticsearch-something` to `something`
In plugins, we are using non consistent naming. We use `elasticsearch-cloud-aws` as the artifactId, which generates a jar file called `elasticsearch-cloud-aws-VERSION.jar`.

But when you want to install the plugin, you will end up with a shorter name for the plugin `cloud-aws`.

```
bin/plugin install cloud-aws
```

This commit changes that and use consistent names for `artifactId`, so `finalName`.

Also changed maven names.
2015-08-18 13:38:48 +02:00
Adrien Grand a72adbf0b4 Fix mapper-murmur3 compatibility version. 2015-08-18 12:31:36 +02:00
Adrien Grand c169386dd4 Merge pull request #12931 from jpountz/fix/murmur3_defaults
Move the `murmur3` field to a plugin and fix defaults.
2015-08-18 12:09:32 +02:00
Ryan Ernst dc1fa6736a Merged AbstractPlugin and Plugin. Also added Settings back to
indexModules and shardModules
2015-08-18 02:46:32 -07:00
Adrien Grand a91b3fcbb9 Move the `murmur3` field to a plugin and fix defaults.
This move the `murmur3` field to the `mapper-murmur3` plugin and fixes its
defaults so that values will not be indexed by default, as the only purpose
of this field is to speed up `cardinality` aggregations on high-cardinality
string fields, which only requires doc values.

I also removed the `rehash` option from the `cardinality` aggregation as it
doesn't bring much value (rehashing is cheap) and allowed to remove the
coupling between the `cardinality` aggregation and the `murmur3` field.

Close #12874
2015-08-18 11:41:52 +02:00
Ryan Ernst 2bf84593e0 Plugins: Simplify Plugin API for constructing modules
The Plugin interface currently contains 6 different methods for
adding modules. Elasticsearch has 3 different levels of injectors,
and for each of those, there are two methods. The first takes no
arguments and returns a collection of class objects to construct. The
second takes a Settings object and returns a collection of module
objects already constructed. The settings argument is unecessary because
the plugin can already get the settings from its constructor. Removing
that, the only difference between the two versions is returning an
already constructed Module, or a module Class, and there is no reason
the plugin can't construct all their modules themselves.

This change reduces the plugin api down to just 3 methods for adding
modules. Each returns a Collection<Module>. It also removes the
processModule method, which was unnecessary since onModule
implementations fullfill the same requirement. And finally, it renames
the modules() method to nodeModules() so it is clear these are created
once for each node.
2015-08-17 20:41:45 -07:00
Ryan Ernst 6f124e6eec Internal: Simplify custom repository type setup
Custom repository types are registered through the RepositoriesModule.
Later, when a specific repository type is used, the RespositoryModule
is installed, which in turn would spawn the module that was
registered for that repository type. However, a module is not needed
here. Each repository type has two associated classes, a Repository and
an IndexShardRepository.

This change makes the registration method for custom repository
types take both of these classes, instead of a module.

See #12783.
2015-08-17 15:08:08 -07:00
Robert Muir 68307aa9f3 Fix network binding for ipv4/ipv6
When elasticsearch is configured by interface (or default: loopback interfaces),
bind to all addresses on the interface rather than an arbitrary one.

If the publish address is not specified, default it from the bound addresses
based on the following sort ordering:

* ipv4/ipv6 (java.net.preferIPv4Stack, defaults to true)
* ordinary addresses
* site-local addresses
* link local addresses
* loopback addresses

One one address is published, and multicast is still always over ipv4: these
need to be future improvements.

Closes #12906
Closes #12915

Squashed commit of the following:

commit 7e60833312f329a5749f9a256b9c1331a956d98f
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 14:45:33 2015 -0400

    fix java 7 compilation oops

commit c7b9f3a42058beb061b05c6dd67fd91477fd258a
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 14:24:16 2015 -0400

    Cleanup/fix logic around custom resolvers

commit bd7065f1936e14a29c9eb8fe4ecab0ce512ac08e
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 13:29:42 2015 -0400

    Add some unit tests for utility methods

commit 0faf71cb0ee9a45462d58af3d1bf214e8a79347c
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 12:11:48 2015 -0400

    localhost all the way down

commit e198bb2bc0d1673288b96e07e6e6ad842179978c
Merge: b55d092 b93a75f
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 12:05:02 2015 -0400

    Merge branch 'master' into network_cleanup

commit b55d092811d7832bae579c5586e171e9cc1ebe9d
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 12:03:03 2015 -0400

    fix docs, fix another bug in multicast (publish host = bad here!)

commit 88c462eb302b30a82585f95413927a5cbb7d54c4
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 11:50:49 2015 -0400

    remove nocommit

commit 89547d7b10d68b23d7f24362e1f4782f5e1ca03c
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 11:49:35 2015 -0400

    fix http too

commit 9b9413aca8a3f6397b5031831f910791b685e5be
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Aug 17 11:06:02 2015 -0400

    Fix transport / interface code

    Next up: multicast and then http
2015-08-17 15:43:07 -04:00
Ryan Ernst 2450e3ccc8 Internal: Flatten IndicesModule and add tests
The IndicesModule was made up of two submodules, one which
handled registering queries, and the other for registering
hunspell dictionaries. This change moves those into
IndicesModule. It also adds a new extension point type,
InstanceMap. This is simply a Map<K,V>, where K and V are
actual objects, not classes like most other extension points.
I also added a test method to help testing instance map extensions.
This was particularly painful because of how guice binds the key
and value as separate bindings, and then reconstitutes them
into a Map at injection time. In order to gain access to the
object which links the key and value, I had to tweak our
guice copy to not use an anonymous inner class for the Provider.

Note that I also renamed the existing extension point types, since
they were very redundant. For example, ExtensionPoint.MapExtensionPoint
is now ExtensionPoint.ClassMap.

See #12783.
2015-08-16 17:56:35 -07:00
Ryan Ernst 754c1b44e7 Fix ZenDiscovery ctor change to remove dynamic cluster settings 2015-08-16 10:40:13 -07:00
Nik Everett 581fd16775 Merge pull request #12897 from nik9000/jvm-example-gets-bin
Add bin to jvm-example
2015-08-16 10:11:23 -07:00
Clinton Gormley e143c6e460 Docs: Prepare plugin and integration docs for 2.0
* Centralised plugin docs in docs/plugins/
* Moved integrations into same docs
* Moved community clients into the clients section of the docs
* Removed docs/community

Closes #11734
Closes #11724
Closes #11636
Closes #11635
Closes #11632
Closes #11630
Closes #12046
Closes #12438
Closes #12579
2015-08-15 18:02:43 +02:00
Simon Willnauer b447e2ae99 Move master to [2.1.0-SNAPSHOT] 2015-08-14 23:44:06 +02:00
Nik Everett fc954216e3 Testing: Add bin to jvm-example
This will be useful in testing the plugin installer.

Relates to #12651
2015-08-14 13:05:26 -07:00
Ryan Ernst f3d63095db Merge branch 'master' into resolve_your_own_config
Conflicts:
	core/src/main/java/org/elasticsearch/env/Environment.java
	core/src/test/java/org/elasticsearch/index/analysis/commongrams/CommonGramsTokenFilterFactoryTests.java
	core/src/test/java/org/elasticsearch/index/analysis/synonyms/SynonymsAnalysisTest.java
	plugins/analysis-kuromoji/src/test/java/org/elasticsearch/index/analysis/KuromojiAnalysisTests.java
2015-08-14 03:34:26 -07:00
Ryan Ernst be638fb6ef Internal: Remove Environment.resolveConfig
This method has multiple modes of resolving config files by
first looking in the config directory, then on the classpath,
and finally by prefixing with "config/" on the classpath.

Most of the places taking advantage of this were tests, so they
did not have to setup a real home dir with config. The only place
that was really relying on it was the code which loads names.txt
to randomly choose a node name.

This change fixes test to setup fake home dirs with their config
files. It also makes the logic for finding names.txt explicit:
look in config dir, and if it doesn't exist, load /config/names.txt
from the classpath.
2015-08-14 03:03:47 -07:00
Ryan Ernst 6dcfda99e8 Internal: Remove ClassLoader from Settings
Settings currently has a classloader member, which any user (plugin
or core ES code) can access to load classes/resources. This is extremely
error prone as setting the classloder on the Settings instance is a
public method. Furthermore, it is not really necessary. Classes that
need resources should load resources using normal means
(getClass().getResourceAsStream). Those that need classes
should use Class.forName, which will load the class with the
same classloader as the calling class. This means, in the few
places where classes are loaded by string name, they will use
the appropriate loader: either the default classloader which loads
core ES code, or a child classloader for each plugin.

This change removes the classloader member from Settings, as
well as other classloader related uses (except for a handful
of cases which must use a classloader, at least for now).
2015-08-13 23:55:27 -07:00
Ryan Ernst c16772b0fc Undo accidental commit of crap
This reverts commit 589eecf55d.
2015-08-13 19:39:55 -07:00
Ryan Ernst 589eecf55d Fourth time's a charm, one more mock class to add to test jar 2015-08-13 19:36:53 -07:00
David Pilato 48ef9e3cfb Merge branch 'aws/base_path' 2015-08-13 15:29:00 +02:00
Adrien Grand 28708f8013 Tests: Fix SimpleIcuAnalysisTests to not load a non-existent configuration file. 2015-08-13 14:39:10 +02:00
Simon Willnauer 605253a39f Cut over master to 2.0.0-SNAPSHOT 2015-08-12 21:16:08 +02:00
David Pilato beb7e16a43 [cloud-gce] replace integration tests by unit tests
We use google transport mock as we can simulate whatever JSON answer GCE platform will send us and really test Gce implementation.

We also remove GceSimpleITest as the goal of this class was only to check that when we start elasticsearch with this plugin, elasticsearch works fine.
We don't need that anymore as we now have RestIT which do that right (and better)!

Closes #12622
2015-08-12 17:47:53 +02:00
David Pilato 99fccddcf2 [cloud-gce] update to GCE API v1-rev71-1.20.0
Closes #12835
2015-08-12 17:38:04 +02:00
Nik Everett e4e60774c1 Merge pull request #12787 from nik9000/configured_plugin
QA: Add configuration to the jvm-example plugin
2015-08-12 07:36:34 -07:00
Clinton Gormley 29d1924d78 Add licenses/no_deps.txt to plugins/jvm-example to quiet the license checker 2015-08-11 19:55:41 +02:00
Nik Everett e807499c29 Merge configured-plugin into jvm-example plugin 2015-08-11 07:36:32 -07:00
Ryan Ernst 40f119d85a This method on settings loaded a class, based on a setting value, using
the default classloader. It had all kinds of leniency in how the
classname was found, and simply cannot work with plugins having isolated
classloaders.

This change removes that method. Some of the uses of it were for custom
extension points, like custom repository or discovery types. A lot were
just there to plugin mock implementations for tests. For the settings
that were legitimate, all now support plugins adding the given setting
via onModule. For those that were specific to tests for mocks, they now
use Classes.loadClass (a helper around Class.forName). This is a
temporary measure until (in a future PR) tests can change the
implementation via package private statics.

I also removed a number of unnecessary intermediate modules, added a
"jvm-example" plugin that can be filled in in the future as a smoke test
for breaking plugins, and gave some documentation to "spawn" modules
interface.

closes #12643
closes #12656
2015-08-10 14:04:45 -07:00
Colin Goodheart-Smithe 6abc69c488 Packaging: Makes sure all POMs contain a description
Adds an explicit description the RPM package so it doesn't inherit the description from the POM.

Closes #12550

Also, modified descriptions for deb and rpm packages to be the same and to reference the documentation rather than listing features that are out of date.
2015-08-10 15:21:24 +01:00
David Pilato f843a1375f [cloud-aws] add support for base_path in elasticsearch.yml
Related to https://github.com/elastic/elasticsearch-cloud-aws/issues/230

We now can support setting a global `base_path` in `elasticsearch.yml` using `repositories.s3.base_path` key.
2015-08-10 10:40:44 +02:00
Clinton Gormley a0774d0f48 no_deps.txt should be in the licenses/ directory in plugins 2015-08-09 20:16:11 +02:00
Clinton Gormley bc7dbce4fe Enable the license checker for plugins without third party deps too 2015-08-09 19:07:45 +02:00
Clinton Gormley 205262c737 Enable the license checker over distribution/* and plugins/*
Moved the license checker config into the parent pom, and overrede
the license dir/target-to-check in distributions/pom.

Disabled the license checker explicitly for projects which run integration
tests but have no licenses dir:

* core
* distribution
* qa
* plugins/delete-by-query
* plugins/mapper-size
* plugins/site-example

Closes #12752
Closes #12754
2015-08-09 18:18:59 +02:00
Robert Muir d1a5068b77 Merge branch 'master' into integ_randomization 2015-08-06 14:51:00 -04:00
Robert Muir 7414d19d28 Support jenkins randomization in integration tests 2015-08-06 14:48:27 -04:00
Adrien Grand 5dd5936fad Move the `_size` mapper to a plugin.
This is one of our esoteric metadata mappers so I think we should distribute
it in a plugin rather than in elasticsearch core.

This introduces one limitation: the value of the `_size` parameter is not
retrievable for documents that are only in the transaction log.
2015-08-06 20:35:22 +02:00
Adrien Grand b16e2d95af Allow scripts to expose whether they use the `_score`.
This commit adds a new API to allow scripts to say whether they need scores.
In practice, only the `expression` script engine makes use of it correctly,
other engines just return `true` since they can't predict whether they'll
need scores. This should make scripted aggregations and `function_query`
faster as we'll now be able to pass needsScores=false to Query.createWeight.
2015-08-06 17:25:51 +02:00
Robert Muir 5fe49e4420 cleanup skipping everywhere 2015-08-05 21:45:09 -04:00
xuzha 333ca689d3 Add retry when checking s3 bucket 2015-08-04 15:58:15 -07:00
Robert Muir 6f9a067197 Change master branch back to 2.0-beta1 2015-08-04 15:38:21 -04:00
Robert Muir 29d73accdd Merge pull request #12639 from rmuir/improve_integ
Don't use port 9200/9300 for integration tests
2015-08-04 13:09:47 -04:00
Robert Muir f1019ace3b Fix plugins integ dependencies and apply cleanups 2015-08-04 09:59:19 -04:00
Robert Muir cf6fe6cdcd Don't use port 9200/9300 for integration tests 2015-08-04 09:25:44 -04:00
Simon Willnauer 6753f7f03e Cut over master to 2.0.0-SNAPSHOT 2015-08-04 10:54:12 +02:00
Robert Muir a0a177222b fix typo in assume() when running from IDE 2015-08-03 23:57:27 -04:00
Robert Muir 64850c0539 Improve site-example integ test to test served contents.
Closes #12578
2015-08-03 23:47:36 -04:00
Ryan Ernst 1e12d03252 Tests: Rename base tests cases to use "TestCase" suffix
Most of the abstract base test classes we have were previously @Ignored.
However, there were also some other tests ignored. Having two ways to
quiet tests is confusing, and clearly it has caused some tests
to get lost in the fold.

This change moves all base test classes to use the "TestCase" suffix,
which is not picked up by the test class name pattern. It also removes
@Ignore from (almost) all tests, and adds it to forbidden apis.
And since we were renaming, I shorted base test class names to use
"ES" instead of "Elasticsearch". I type this a lot of types a day,
and I have heard others express a similar desire for a shorter name.

closes #10659
2015-08-03 17:43:00 -07:00
Ryan Ernst 32a5c20b33 Tests: Forbid @Slow
Now that all uses of @Slow are gone, we can forbid this annotations use
so tests are not confusing. Follow up to #12617.
2015-08-03 16:43:09 -07:00
Ryan Ernst be941f0669 Tests: Remove uses of @Slow and @Integration
Now that integ tests are moved into `mvn verify`, we don't really have
a need for @Slow, and especially not @Integration. This removes
uses of the first, and completely removes uses of the latter.
2015-08-03 16:26:07 -07:00
David Pilato ac89f43d11 Merge pull request #12604 from dadoonet/maven/rename-site-example
[build] rename site example plugin to site-example
2015-08-03 18:45:13 +02:00
David Pilato 1b0492f800 [build] rename site example plugin to site-example
Just to be consistent in plugin naming convention we have today: `plugin_type-plugin_name`

So it's a `site` plugin.
2015-08-03 18:41:04 +02:00
Robert Muir 379ffdb8ee Add integration test phase for elasticsearch core/ 2015-08-03 09:43:11 -04:00
Robert Muir c5b91b9058 Merge pull request #12589 from rmuir/shading_cleanup
Fix shaded jar packaging
2015-08-03 08:10:50 -04:00
Tanguy Leroux 3537610760 Add comment about AWS S3 "doesBucketExist()" method 2015-08-03 11:02:16 +02:00
Robert Muir fef55348f9 clean up .zip handling so its not uselessly downloaded in the build 2015-08-01 09:10:03 -04:00
Robert Muir f804ec3cd9 cleanup 2015-07-31 10:05:11 -04:00
Robert Muir fd8e92a18a don't represent site plugins with 'null' anymore 2015-07-31 09:41:39 -04:00
Jason Tedor b0564cb75c Add integration test for Azure snapshot repository bug
This commit adds an integration test that replicates the Azure snapshot repository bug in elastic/elasticsearch-cloud-azure#51.

Closes elastic/elasticsearch-cloud-azure#100
2015-07-27 13:52:51 -04:00
Jason Tedor 45fadd4955 Use correct settings keys for setting container, base path and chunk size in Azure integration test 2015-07-27 13:37:58 -04:00
Robert Muir 4040f194f5 Refactor pluginservice
Closes #12367

Squashed commit of the following:

commit 9453c411798121aa5439c52e95301f60a022ba5f
Merge: 3511a9c 828d8c7
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Jul 22 08:22:41 2015 -0400

    Merge branch 'master' into refactor_pluginservice

commit 3511a9c616503c447de9f0df9b4e9db3e22abd58
Author: Ryan Ernst <ryan@iernst.net>
Date:   Tue Jul 21 21:50:15 2015 -0700

    Remove duplicated constant

commit 4a9b5b4621b0ef2e74c1e017d9c8cf624dd27713
Author: Ryan Ernst <ryan@iernst.net>
Date:   Tue Jul 21 21:01:57 2015 -0700

    Add check that plugin must specify at least site or jvm

commit 19aef2f0596153a549ef4b7f4483694de41e101b
Author: Ryan Ernst <ryan@iernst.net>
Date:   Tue Jul 21 20:52:58 2015 -0700

    Change plugin "plugin" property to "classname"

commit 07ae396f30ed592b7499a086adca72d3f327fe4c
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 23:36:05 2015 -0400

    remove test with no methods

commit 550e73bf3d0f94562f4dde95239409dc5a24ce25
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 23:31:58 2015 -0400

    fix loading to use classname

commit 04463aed12046da0da5cac2a24c3ace51a79f799
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 23:24:19 2015 -0400

    rename to classname

commit 9f3afadd1caf89448c2eb913757036da48758b2d
Author: Ryan Ernst <ryan@iernst.net>
Date:   Tue Jul 21 20:18:46 2015 -0700

    moved PluginInfo and refactored parsing from properties file

commit df63ccc1b8b7cc64d3e59d23f6c8e827825eba87
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 23:08:26 2015 -0400

    fix test

commit c7febd844be358707823186a8c7a2d21e37540c9
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 23:03:44 2015 -0400

    remove test

commit 017b3410cf9d2b7fca1b8653e6f1ebe2f2519257
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 22:58:31 2015 -0400

    fix test

commit c9922938df48041ad43bbb3ed6746f71bc846629
Merge: ad59af4 01ea89a
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 22:37:28 2015 -0400

    Merge branch 'master' into refactor_pluginservice

commit ad59af465e1f1ac58897e63e0c25fcce641148a7
Author: Areek Zillur <areek.zillur@elasticsearch.com>
Date:   Tue Jul 21 19:30:26 2015 -0400

    [TEST] Verify expected number of nodes in cluster before issuing shardStores request

commit f0f5a1e087255215b93656550fbc6bd89b8b3205
Author: Lee Hinman <lee@writequit.org>
Date:   Tue Jul 21 11:27:28 2015 -0600

    Ignore EngineClosedException during translog fysnc

    When performing an operation on a primary, the state is captured and the
    operation is performed on the primary shard. The original request is
    then modified to increment the version of the operation as preparation
    for it to be sent to the replicas.

    If the request first fails on the primary during the translog sync
    (because the Engine is already closed due to shadow primaries closing
    the engine on relocation), then the operation is retried on the new primary
    after being modified for the replica shards. It will then fail due to the
    version being incorrect (the document does not yet exist but the request
    expects a version of "1").

    Order of operations:

    - Request is executed against primary
    - Request is modified (version incremented) so it can be sent to replicas
    - Engine's translog is fsync'd if necessary (failing, and throwing an exception)
    - Modified request is retried against new primary

    This change ignores the exception where the engine is already closed
    when syncing the translog (similar to how we ignore exceptions when
    refreshing the shard if the ?refresh=true flag is used).

commit 4ac68bb1658688550ced0c4f479dee6d8b617777
Author: Shay Banon <kimchy@gmail.com>
Date:   Tue Jul 21 22:37:29 2015 +0200

    Replica allocator unit tests
    First batch of unit tests to verify the behavior of replica allocator

commit 94609fc5943c8d85adc751b553847ab4cebe58a3
Author: Jason Tedor <jason@tedor.me>
Date:   Tue Jul 21 14:04:46 2015 -0400

    Correctly list blobs in Azure storage to prevent snapshot corruption and do not unnecessarily duplicate Lucene segments in Azure Storage

    This commit addresses an issue that was leading to snapshot corruption for snapshots stored as blobs in Azure Storage.

    The underlying issue is that in cases when multiple snapshots of an index were taken and persisted into Azure Storage, snapshots subsequent
    to the first would repeatedly overwrite the snapshot files. This issue does render useless all snapshots except the final snapshot.

    The root cause of this is due to String concatenation involving null. In particular, to list all of the blobs in a snapshot directory in
    Azure the code would use the method listBlobsByPrefix where the prefix is null. In the listBlobsByPrefix method, the path keyPath + prefix
    is constructed. However, per 5.1.11, 5.4 and 15.18.1 of the Java Language Specification, the reference null is first converted to the string
    "null" before performing the concatenation. This leads to no blobs being returned and therefore the snapshot mechanism would operate as if
    it were writing the first snapshot of the index. The fix is simply to check if prefix is null and handle the concatenation accordingly.

    Upon fixing this issue so that subsequent snapshots would no longer overwrite earlier snapshots, it was discovered that the snapshot metadata
    returned by the listBlobsByPrefix method was not sufficient for the snapshot layer to detect whether or not the Lucene segments had already
    been copied to the Azure storage layer in an earlier snapshot. This led the snapshot layer to unnecessarily duplicate these Lucene segments
    in Azure Storage.

    The root cause of this is due to known behavior in the CloudBlobContainer.getBlockBlobReference method in the Azure API. Namely, this method
    does not fetch blob attributes from Azure. As such, the lengths of all the blobs appeared to the snapshot layer to be of length zero and
    therefore they would compare as not equal to any new blobs that the snapshot layer is going to persist. To remediate this, the method
    CloudBlockBlob.downloadAttributes must be invoked. This will fetch the attributes from Azure Storage so that a proper comparison of the
    blobs can be performed.

    Closes elastic/elasticsearch-cloud-azure#51, closes elastic/elasticsearch-cloud-azure#99

commit cf1d481ce5dda0a45805e42f3b2e0e1e5d028b9e
Author: Lee Hinman <lee@writequit.org>
Date:   Mon Jul 20 08:41:55 2015 -0600

    Unit tests for `nodesAndVersions` on shared filesystems

    With the `recover_on_any_node` setting, these unit tests check that the
    correct node list and versions are returned.

commit 3c27cc32395c3624f7c794904d9ea4faf2eccbfb
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 14:15:59 2015 -0400

    don't fail junit4 integration tests if there are no tests.

    instead fail the failsafe plugin, which means the external cluster will still get shut down

commit 95d2756c5a8c21a157fa844273fc83dfa3c00aea
Author: Alexander Reelsen <alexander@reelsen.net>
Date:   Tue Jul 21 17:16:53 2015 +0200

    Testing: Fix help displaying tests under windows

    The help files are using a unix based file separator, where as
    the test relies on the help being based on the file system separator.

    This commit fixes the test to remove all `\r` characters before
    comparing strings.

    The test has also been moved into its own CliToolTestCase, as it does
    not need to be an integration test.

commit 944f06ea36bd836f007f8eaade8f571d6140aad9
Author: Clinton Gormley <clint@traveljury.com>
Date:   Tue Jul 21 18:04:52 2015 +0200

    Refactored check_license_and_sha.pl to accept a license dir and package path

    In preparation for the move to building the core zip, tar.gz, rpm, and deb as separate modules, refactored check_license_and_sha.pl to:

    * accept a license dir and path to the package to check on the command line
    * to be able to extract zip, tar.gz, deb, and rpm
    * all packages except rpm will work on Windows

commit 2585431e8dfa5c82a2cc5b304cd03eee9bed7a4c
Author: Chris Earle <pickypg@users.noreply.github.com>
Date:   Tue Jul 21 08:35:28 2015 -0700

    Updating breaking changes

    - field names cannot be mapped with `.` in them
    - fixed asciidoc issue where the list was not recognized as a list

commit de299b9d3f4615b12e2226a1e2eff5a38ecaf15f
Author: Shay Banon <kimchy@gmail.com>
Date:   Tue Jul 21 13:27:52 2015 +0200

    Replace primaryPostAllocated flag and use UnassignedInfo
    There is no need to maintain additional state as to if a primary was allocated post api creation on the index routing table, we hold all this information already in the UnassignedInfo class.
    closes #12374

commit 43080bff40f60bedce5bdbc92df302f73aeb9cae
Author: Alexander Reelsen <alexander@reelsen.net>
Date:   Tue Jul 21 15:45:05 2015 +0200

    PluginManager: Fix bin/plugin calls in scripts/bats test

    The release and smoke test python scripts used to install
    plugins in the old fashion.

    Also the BATS testing suite installed/removed plugins in that
    way. Here the marvel tests have been removed, as marvel currently
    does not work with the master branch.

    In addition documentation has been updated as well, where it was
    still missing.

commit b81ccba48993bc13c7678e6d979fd96998499233
Author: Boaz Leskes <b.leskes@gmail.com>
Date:   Tue Jul 21 11:37:50 2015 +0200

    Discovery: make sure NodeJoinController.ElectionCallback is always called from the update cluster state thread

    This is important for correct handling of the joining thread. This causes assertions to trip in our test runs. See http://build-us-00.elastic.co/job/es_g1gc_master_metal/11653/ as an example

    Closes #12372

commit 331853790bf29e34fb248ebc4c1ba585b44f5cab
Author: Boaz Leskes <b.leskes@gmail.com>
Date:   Tue Jul 21 15:54:36 2015 +0200

    Remove left over no commit from TransportReplicationAction

    It asks to double check thread pool rejection. I did and don't see problems with it.

commit e5724931bbc1603e37faa977af4235507f4811f5
Author: Alexander Reelsen <alexander@reelsen.net>
Date:   Tue Jul 21 15:31:57 2015 +0200

    CliTool: Various PluginManager fixes

    The new plugin manager parser was not called correctly in the scripts.
    In addition the plugin manager now creates a plugins/ directory in case
    it does not exist.

    Also the integration tests called the plugin manager in the deprecated way.

commit 7a815a370f83ff12ffb12717ac2fe62571311279
Author: Alexander Reelsen <alexander@reelsen.net>
Date:   Tue Jul 21 13:54:18 2015 +0200

    CLITool: Port PluginManager to use CLITool

    In order to unify the handling and reuse the CLITool infrastructure
    the plugin manager should make use of this as well.

    This obsolets the -i and --install options but requires the user
    to use `install` as the first argument of the CLI.

    This is basically just a port of the existing functionality, which
    is also the reason why this is not a refactoring of the plugin manager,
    which will come in a separate commit.

commit 7f171eba7b71ac5682a355684b6da703ffbfccc7
Author: Martijn van Groningen <martijn.v.groningen@gmail.com>
Date:   Tue Jul 21 10:44:21 2015 +0200

    Remove custom execute local logic in TransportSingleShardAction and TransportInstanceSingleOperationAction and rely on transport service to execute locally. (forking thread etc.)

    Change TransportInstanceSingleOperationAction to have shardActionHandler to, so we can execute locally without endless spinning.

commit 0f38e3eca6b570f74b552e70b4673f47934442e1
Author: Ryan Ernst <ryan@iernst.net>
Date:   Tue Jul 21 17:36:12 2015 -0700

    More readMetadata tests and pickiness

commit 880b47281bd69bd37807e8252934321b089c9f8e
Author: Ryan Ernst <ryan@iernst.net>
Date:   Tue Jul 21 14:42:09 2015 -0700

    Started unit tests for plugin service

commit cd7c8ddd7b8c4f3457824b493bffb19c156c7899
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 07:21:07 2015 -0400

    fix tests

commit 673454f0b14f072f66ed70e32110fae4f7aad642
Author: Robert Muir <rmuir@apache.org>
Date:   Tue Jul 21 06:58:25 2015 -0400

    refactor pluginservice
2015-07-22 10:45:45 -04:00
Jason Tedor 96c245c190 Merge pull request #12380 from jasontedor/fix/elasticsearch-cloud-azure-51
Correctly list blobs in Azure Storage to prevent snapshot corruption and do not unnecessarily duplicate Lucene segments in Azure Storage
2015-07-21 16:47:04 -04:00
Jason Tedor 0b8b147a74 Correctly list blobs in Azure storage to prevent snapshot corruption and do not unnecessarily duplicate Lucene segments in Azure Storage
This commit addresses an issue that was leading to snapshot corruption for snapshots stored as blobs in Azure Storage.

The underlying issue is that in cases when multiple snapshots of an index were taken and persisted into Azure Storage, snapshots subsequent
to the first would repeatedly overwrite the snapshot files. This issue does render useless all snapshots except the final snapshot.

The root cause of this is due to String concatenation involving null. In particular, to list all of the blobs in a snapshot directory in
Azure the code would use the method listBlobsByPrefix where the prefix is null. In the listBlobsByPrefix method, the path keyPath + prefix
is constructed. However, per 5.1.11, 5.4 and 15.18.1 of the Java Language Specification, the reference null is first converted to the string
"null" before performing the concatenation. This leads to no blobs being returned and therefore the snapshot mechanism would operate as if
it were writing the first snapshot of the index. The fix is simply to check if prefix is null and handle the concatenation accordingly.

Upon fixing this issue so that subsequent snapshots would no longer overwrite earlier snapshots, it was discovered that the snapshot metadata
returned by the listBlobsByPrefix method was not sufficient for the snapshot layer to detect whether or not the Lucene segments had already
been copied to the Azure storage layer in an earlier snapshot. This led the snapshot layer to unnecessarily duplicate these Lucene segments
in Azure Storage.

The root cause of this is due to known behavior in the CloudBlobContainer.getBlockBlobReference method in the Azure API. Namely, this method
does not fetch blob attributes from Azure. As such, the lengths of all the blobs appeared to the snapshot layer to be of length zero and
therefore they would compare as not equal to any new blobs that the snapshot layer is going to persist. To remediate this, the method
CloudBlockBlob.downloadAttributes must be invoked. This will fetch the attributes from Azure Storage so that a proper comparison of the
blobs can be performed.

Closes elastic/elasticsearch-cloud-azure#51, closes elastic/elasticsearch-cloud-azure#99
2015-07-21 16:33:33 -04:00
Alexander Reelsen 2f54b89a23 CLITool: Port PluginManager to use CLITool
In order to unify the handling and reuse the CLITool infrastructure
the plugin manager should make use of this as well.

This obsolets the -i and --install options but requires the user
to use `install` as the first argument of the CLI.

This is basically just a port of the existing functionality, which
is also the reason why this is not a refactoring of the plugin manager,
which will come in a separate commit.
2015-07-21 14:15:39 +02:00
Robert Muir a0a6299840 Update randomizedtesting to 2.1.16 2015-07-20 11:02:50 -04:00
Robert Muir 9e7882e604 protect logic under skipTests for stupid pom packaging 2015-07-16 20:49:16 -04:00
Robert Muir 6b7561ac9a Use junit4 for running integration tests, too
failsafe uses surefire, which sucks. It also mean integ tests act alien right now.
I would rather have the consistency, e.g. things formatted the same way, running integ tests under security manager, etc.
2015-07-16 19:43:33 -04:00
Robert Muir 68143dedf2 Refactor integration tests
1. tests don't have a bogus test dependency on zips anymore,
   instead we handle this in pre-integration-test. This reduces
   lots of confusion for e.g. mvn clean test.
2. refactor integ logic so that core/ and plugin/ share it.
   previously they were duplicates but the above change simplifies life.
   it also makes it easier for doing more interesting stuff
2015-07-14 23:37:04 -04:00
Simon Willnauer 7db293c616 Generify Index and Shard exceptions
Today we have a intermediate hierarchy for shard and index exceptions
which makes it hard to introduce generic exceptions like ResourceNotFoundException
intoduced in this commit. This commit breaks up the hierarchy by adding index and shard
as a special internal header that gets rendered for every exception that fills that header.
This commit removes dedicated exceptions like `IndexMissingException` or
`IndexShardMissingException` in favour of `ResourceNotFoundException`
2015-07-14 16:31:49 +02:00
uboness b40186652c updated the elasticsearch versioning format
Moving to from `X.Y.Z.beta1`/`X.Y.Z.RC1` to `X.Y.Z-beta1`/`X.Y.Z-rc1`
2015-07-13 20:26:37 +02:00
Jack Conradson c1137b3b78 Add script type and script name to error messages
Modified ScriptEngineService to pass in a CompiledScript object
with newly added name and type member variables.
This can in turn be used to give better scripting error messages
with the type of script used and the name of the script.

Required slight modifications to the caching mechanism.

Note that this does not enforce good behavior in that plugins will
have to write exceptions that also output the name of the script
in order to be effective. There was no way to wrap the script
methods in a try/catch block properly further up the chain because
many have script-like objects passed back that can be run at a
later time.

closes #6653
closes #11449
2015-07-10 18:27:48 -07:00
Martijn van Groningen 52859e3a52 Internal: refactored MetaData to split the concrete index name resolution to IndexNameExpressionResolver.
Changes in a nutshell:
* All expression logic is now encapsulated by ExpressionResolver interface.
* MetaData#convertFromWildcards() gets replaced by WildcardExpressionResolver.
* All of the indices expansion methods are being moved from MetaData class to the new IndexNameExpressionResolver class.
* All single index expansion optimisations are removed.

The logic for resolving a concrete index name from an expression has been moved from MetaData to IndexExpressionResolver. The logic has been cleaned up and simplified were was possible without breaking bwc.

Also the notion of aliasOrIndex has been changed to index expression.

The IndexNameExpressionResolver translates index name expressions into concrete indices. The list of index name expressions are first delegated to the known ExpressionResolverS. An ExpressionResolver is responsible for translating if possible an expression into another expression (possibly but not required this can be concrete indices or aliases) otherwise the expressions are left untouched. Concretely this means converting wildcard expressions into concrete indices or aliases, but in the future other implementations could convert expressions based on different rules.

To prevent many overloading of methods, DocumentRequest extends now from IndicesRequest. All implementation of DocumentRequest already did implement IndicesRequest indirectly.
2015-07-10 15:18:28 +02:00
Simon Willnauer e0708813a9 Make 2.0.0.beta1-SNAPSHOT the current version.
Today everything is tight to having the next version as the latest.
In order to work towards 2.0.0.beta1 we need to fix all the usage of
2.0.0-SNAPSHOT to reflect the version we will release soon.
Usually we do this on the release branch but to simplify things I wanna
keep this on master for now and move to 2.1.0-SNAPSHOT on master once
we created a 2.0 branch.

Closes #12148
2015-07-09 21:24:32 +02:00
Robert Muir a4dba6256d Make integ tests antfile a property, so it can be changed 2015-07-09 00:39:28 -04:00
David Pilato 02874ea411 [build] explicitly add http-client dependency
As discussed in the PR https://github.com/elastic/elasticsearch/pull/12036#issuecomment-119584570
2015-07-08 16:17:27 +02:00
David Pilato 88cf29b903 [build] cloud-aws doesn't register s3 repos anymore
Reported in https://github.com/elastic/elasticsearch/issues/11647#issuecomment-118523861

> btw, I think you broke some plugins on Master, cloud-aws doesn't register s3 repos anymore.

```
org.elasticsearch.common.inject.CreationException: Guice creation errors:

1) Error injecting constructor, java.lang.NoClassDefFoundError: org/apache/http/protocol/HttpContext
  at org.elasticsearch.repositories.s3.S3Repository.<init>(Unknown Source)
  while locating org.elasticsearch.repositories.s3.S3Repository
  while locating org.elasticsearch.repositories.Repository

1 error
	at org.elasticsearch.common.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:344)
	at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:178)
	at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110)
	at org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:140)
	at org.elasticsearch.common.inject.ModulesBuilder.createChildInjector(ModulesBuilder.java:69)
	at org.elasticsearch.repositories.RepositoriesService.createRepositoryHolder(RepositoriesService.java:404)
	at org.elasticsearch.repositories.RepositoriesService.registerRepository(RepositoriesService.java:368)
	at org.elasticsearch.repositories.RepositoriesService.access$100(RepositoriesService.java:55)
	at org.elasticsearch.repositories.RepositoriesService$1.execute(RepositoriesService.java:110)
	at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:378)
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:209)
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:179)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoClassDefFoundError: org/apache/http/protocol/HttpContext
	at com.amazonaws.AmazonWebServiceClient.<init>(AmazonWebServiceClient.java:129)
	at com.amazonaws.services.s3.AmazonS3Client.<init>(AmazonS3Client.java:432)
	at com.amazonaws.services.s3.AmazonS3Client.<init>(AmazonS3Client.java:414)
	at org.elasticsearch.cloud.aws.InternalAwsS3Service.getClient(InternalAwsS3Service.java:153)
	at org.elasticsearch.cloud.aws.InternalAwsS3Service.client(InternalAwsS3Service.java:82)
	at org.elasticsearch.repositories.s3.S3Repository.<init>(S3Repository.java:125)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
	at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:56)
	at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86)
	at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:104)
	at org.elasticsearch.common.inject.FactoryProxy.get(FactoryProxy.java:54)
	at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:47)
	at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:865)
	at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:43)
	at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:59)
	at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:46)
	at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:201)
	at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:193)
	at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:858)
	at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:193)
	at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:175)
	... 13 more
Caused by: java.lang.ClassNotFoundException: org.apache.http.protocol.HttpContext
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 37 more
```

Closes #12034.
2015-07-08 15:37:24 +02:00
David Pilato 9519100eca [test] fix missing REST specs
Something went wrong with my last commit.

Fixing it here.
2015-07-08 15:22:43 +02:00
David Pilato 97019ce3e9 [python] move integration tests to REST tests
We can keep only unit tests in plugins instead of starting each time a local node and running tests against it.

Also follow up of #12091
2015-07-08 15:19:26 +02:00
David Pilato c99bc81bdd [Javascript] move integration tests to REST tests
We can keep only unit tests in plugins instead of starting each time a local node and running tests against it.

Also follow up of #12091
2015-07-08 15:19:26 +02:00
David Pilato 19b2e77bc2 [Stempel] move integration tests to REST tests
We can keep only unit tests in plugins instead of starting each time a local node and running tests against it.

Also follow up of #12091
2015-07-08 15:19:26 +02:00
David Pilato 7a1255d520 [Smartcn] move integration tests to REST tests
We can keep only unit tests in plugins instead of starting each time a local node and running tests against it.

Also follow up of #12091
2015-07-08 15:19:26 +02:00
David Pilato 4738fa04b1 [ICU] move integration tests to REST tests
We can keep only unit tests in plugins instead of starting each time a local node and running tests against it.

Also follow up of #12091

Tests can use more than one JVM

We don't need anymore to set the number of jvm to run tests as we moved IT to Rest Tests
For all plugins but cloud plugins which will require another way for running integration tests.
2015-07-08 15:19:26 +02:00
David Pilato 786d034d75 [phonetic] move integration tests to REST tests
Fix for #12095
2015-07-08 10:17:43 +02:00
David Pilato b970fbacee [phonetic] move integration tests to REST tests
We can keep only unit tests in plugins instead of starting each time a local node and running tests against it.

Also follow up of #12091
2015-07-08 10:04:18 +02:00
Robert Muir 99ab4e70c0 wait for yellow in integration tests. closes #12099 2015-07-07 17:11:29 -04:00
Clinton Gormley 956123157b Added rest tests for language plugins 2015-07-07 19:26:31 +02:00
Clinton Gormley 7aac4d5417 Added rest tests for cloud plugins 2015-07-07 19:26:31 +02:00
Clinton Gormley eefb3cf41c Added rest tests for analysis-phonetic plugin 2015-07-07 19:26:31 +02:00
David Pilato 7843c63aa3 [kuromoji] move integration tests to REST tests
We can keep only unit tests in plugins instead of starting each time a local node and running tests against it.
2015-07-07 17:40:54 +02:00
Robert Muir 0736f97cfb add stempel integration tests 2015-07-07 01:27:25 -04:00
Robert Muir 5bcc247eda Add integration tests for smart chinese 2015-07-07 01:14:39 -04:00
Robert Muir f412c5a6ed add kuromoji integration tests 2015-07-07 00:51:52 -04:00
Robert Muir c88c12c6c8 Add rest tests for analysis-icu 2015-07-07 00:15:49 -04:00
Robert Muir 9a146b9e75 Get delete by query rest tests running 2015-07-06 15:13:50 -04:00
Robert Muir 1c114110c5 integration tests need the zip 2015-07-06 12:29:30 -04:00
David Pilato e7a6b51bab [maven] change groupId / artifactId
When we generate our project, we can get something like:

```
├── dev-tools
├── elasticsearch
├── elasticsearch-parent
├── elasticsearch-plugin
├── plugin
│   ├── elasticsearch-analysis-icu
│   ├── elasticsearch-analysis-kuromoji
│   ├── elasticsearch-analysis-phonetic
│   ├── elasticsearch-analysis-smartcn
│   ├── elasticsearch-analysis-stempel
│   ├── elasticsearch-cloud-aws
│   ├── elasticsearch-cloud-azure
│   ├── elasticsearch-cloud-gce
│   ├── elasticsearch-delete-by-query
│   ├── elasticsearch-lang-javascript
│   └── elasticsearch-lang-python
├── rest-api-spec
└── securemock
```

I propose here to use a common naming for artifacts: start always with `elasticsearch-`.
Also, move `elasticsearch-plugin` to `org.elasticsearch.plugin` groupId.

So we could have:

```
├── elasticsearch
├── elasticsearch-dev-tools
├── elasticsearch-parent
├── elasticsearch-rest-api-spec
├── elasticsearch-securemock
├── plugin
│   ├── elasticsearch-analysis-icu
│   ├── elasticsearch-analysis-kuromoji
│   ├── elasticsearch-analysis-phonetic
│   ├── elasticsearch-analysis-smartcn
│   ├── elasticsearch-analysis-stempel
│   ├── elasticsearch-cloud-aws
│   ├── elasticsearch-cloud-azure
│   ├── elasticsearch-cloud-gce
│   ├── elasticsearch-delete-by-query
│   ├── elasticsearch-lang-javascript
│   ├── elasticsearch-lang-python
│   └── elasticsearch-plugin
```
2015-07-06 17:17:07 +02:00
Tanguy Leroux 9495816cb7 Remove sigar completely 2015-07-03 15:49:17 +02:00
David Pilato e429b8d190 [build] include in plugins only needed jars
Follow up for https://github.com/elastic/elasticsearch-analysis-kuromoji/issues/61

We don't shade anymore elasticsearch dependencies, so plugins might include jars in the distribution ZIP file which might not be needed anymore.

For example, `elasticsearch-cloud-aws` comes with:

```
Archive:  cloud-aws/target/releases/elasticsearch-cloud-aws-2.0.0-SNAPSHOT.zip
  Length     Date   Time    Name
 --------    ----   ----    ----
  1920788  05-18-15 09:42   aws-java-sdk-ec2-1.9.34.jar
   503963  05-18-15 09:42   aws-java-sdk-core-1.9.34.jar
   232771  01-19-15 09:24   commons-codec-1.6.jar
   915096  01-19-15 09:24   jackson-databind-2.3.2.jar
   252288  05-18-15 09:42   aws-java-sdk-kms-1.9.34.jar
    62050  01-19-15 09:24   commons-logging-1.1.3.jar
   282269  10-31-14 13:19   httpcore-4.3.2.jar
    35058  01-19-15 09:24   jackson-annotations-2.3.0.jar
   229998  05-29-15 12:28   jackson-core-2.5.3.jar
   589289  01-19-15 09:24   joda-time-2.7.jar
   562858  05-18-15 09:42   aws-java-sdk-s3-1.9.34.jar
   590533  10-31-14 13:19   httpclient-4.3.5.jar
    44854  06-12-15 19:22   elasticsearch-cloud-aws-2.0.0-SNAPSHOT.jar
 --------                   -------
  6221815                   13 files
```

A lot of those files are already distributed with elasticsearch itself so classes are available within the classloader.

We mark all es core dependencies as provided in plugins.
We also remove `groupId` as already defined in parent pom.
And we remove non needed licenses files as some jars are not included anymore in plugins.

Closes #11647.
2015-07-01 21:37:27 +02:00
Robert Muir 46256e25f6 fix licensing 2015-06-30 00:55:11 -04:00
Robert Muir b58eb35dfa detect jar hell in tests better and fix more stuff 2015-06-30 00:12:40 -04:00
Robert Muir f8fbf1fa0c Fail startup (and tests) on jar hell 2015-06-29 23:13:45 -04:00
David Pursehouse b49e66c3a1 Replace references to ImmutableSettings with Settings
ImmutableSettings was merged into Settings in commit 4873070.

Change-Id: I06bd0150381d131593920c2328c46beacf49661f
2015-06-24 14:54:53 +09:00
Clinton Gormley d8a186e121 Added LICENSE and NOTICE files for all plugins 2015-06-23 12:50:31 +02:00
Simon Willnauer 8b60083dda Add high-level javadocs to delete-by-query
This commit adds high-level javadocs to the delete-by-query explaining
it's semantics and why it's moved to a plugin.

Closes #11723
2015-06-22 14:34:16 +02:00
Colin Goodheart-Smithe 772d0cc6e7 Build: Make rest-spec-api a project so eclipse build works
The change makes rest-spec-api a project in the same way as we build dev-tools. it packages the tests and api in a bundle using the maven-remote-resources-plugin and uses the same plugin in the plugins and core pom to unpack the rest-api-spec into the target directory and references the rest tests there in the test resources.

The main stimulus for this change is that for those using Eclipse the current build does not work. After running `mvn eclipse:eclipse` the Eclipse IDE errors because the rest-api-spec is outside of the project scope, meaning that every time the command is run (required whenever any dependencies change), the class path of all the projects has to be manually fixed.
2015-06-22 11:41:44 +01:00
Simon Willnauer 895a7dd03c [TEST] assume assertions are enabled 2015-06-22 11:25:02 +02:00
David Pilato 42f852d399 [test] ByteSizeUnit.BYTES should be explicitly set
With a recent change in core, we don't support anymore non explicit byte size units when setting values.
This commit fix that in azure repository. Otherwise, tests can't be executed.
2015-06-18 17:29:10 +02:00
Simon Willnauer 5808102d19 Upgrade AWS dependency to 1.10.0
This also apgrades JodaTime to 2.8.0.
2015-06-18 14:04:38 +02:00
Tanguy Leroux 8bd3d7e4a4 Fix number of deleted/missing documents in Delete-By-Query
The deleted counter is incremented even if the document is missing. Also, this commit ensures that the scroll id is cleared even if no documents are found by the scan request.
2015-06-18 13:09:42 +02:00
Simon Willnauer 90f9b5f60d Allow to opt-out of loading packaged REST tests
this is really just a workaround for plugins to run their own
REST tests instead of the core ones. It opts out of the rest test
loading from the core jar file and tries to load from the classpath instead.
Eventually we need to fix this infrastrucutre to move away from parameterized
tests such that subclasses can override behavior.

Closes #11721
2015-06-17 20:37:59 +02:00
Tanguy Leroux ba3540675a Add delete-by-query plugin
The delete by query plugin adds support for deleting all of the documents (from one or more indices) which match the specified query. It is a replacement for the problematic delete-by-query functionality which has been removed from Elasticsearch core in 2.0. Internally, it uses the Scan/Scroll and Bulk APIs to delete documents in an efficient and safe manner. It is slower than the old delete-by-query functionality, but fixes the problems with the previous implementation.

Closes #7052
2015-06-17 14:52:25 +02:00
David Pilato fbf83e3568 Merge pull request #11637 from dadoonet/plugins/elasticsearch-provided
[build] mark elasticsearch as provided in plugins

When we build a plugin, we suppose it will be executed within elasticsearch server.
So we should mark it as `provided`.

If a java developer needs to embed the plugin and elasticsearch, it will make sense to declare both in its `pom.xml` file.
2015-06-16 09:31:24 +02:00
David Pilato bd5c7d0ea2 [maven] clean pom.xml
In Maven parent project, in dependency management, we should only declare which versions of 3rd party jars we want to use but not force any scope.
It makes then more obvious in modules what is exactly the scope of any dependency.

For example, one could imagine importing `jimfs` as a `compile` dependency in another module/plugin with:

```xml
<dependency>
   <groupId>com.google.jimfs</groupId>
   <artifactId>jimfs</artifactId>
</dependency>
```

But it won't work as expected as the default maven `scope` should be `compile` but here it's `test` as defined in the parent project.

So, if you want to use this lib for tests, you should simply define:

```xml
<dependency>
   <groupId>com.google.jimfs</groupId>
   <artifactId>jimfs</artifactId>
   <scope>test</scope>
</dependency>
```

We also remove `maven-s3-wagon` from gce plugin as it's not used.
2015-06-15 17:08:15 +02:00
Simon Willnauer 364cbbd282 [TEST] Fix JavaScript tests - case matters here 2015-06-13 10:34:36 +02:00
Simon Willnauer 09a30e7eee Remove deprecated script APIs
The script APIs have been deprecated long ago we can now remove them.
This commit still keeps the parsing code since it might be used in a
query that is still stuck in transaction log. This issue should be discussed
elsewhere.

Closes #11619
2015-06-13 10:11:43 +02:00
David Pilato 7c365daf67 [build] mark elasticsearch as provided in plugins
When we build a plugin, we suppose it will be executed within elasticsearch server.
So we should mark it as `provided`.

If a java developer needs to embed the plugin and elasticsearch, it will make sense to declare both in its `pom.xml` file.
2015-06-12 18:53:35 +02:00
Boaz Leskes 134d898be9 Test: added awaitFix to AzureTwoStartedNodesTest.two_nodes_should_run_using_private_ip 2015-06-11 08:59:44 +02:00
Simon Willnauer 54815f259e add lang-javascript module 2015-06-09 15:17:27 +02:00
Simon Willnauer 64fdb0eeb0 migrate branch for lang-javascript 2015-06-09 15:17:27 +02:00
Simon Willnauer a24c10c703 add lang-javascript module 2015-06-09 15:17:25 +02:00
Simon Willnauer d9fe7cc252 add lang-python module 2015-06-09 15:17:23 +02:00
Simon Willnauer a04a9777d4 migrate branch for lang-python 2015-06-09 15:17:23 +02:00
Simon Willnauer a1b410652d add lang-python module 2015-06-09 15:17:22 +02:00
Boaz Leskes 8f2dc10832 AwaitFix to two_nodes_should_run_using_public_ip
https://github.com/elastic/elasticsearch/issues/11533
2015-06-08 12:45:48 +02:00
Simon Willnauer 5beed150ed add cloud-aws module 2015-06-05 13:12:38 +02:00
Simon Willnauer b3088d50a1 migrate branch for cloud-aws 2015-06-05 13:12:38 +02:00
Simon Willnauer a704a9ff83 add cloud-aws module 2015-06-05 13:12:36 +02:00
Simon Willnauer 7c23416b4a add cloud-azure module 2015-06-05 13:12:34 +02:00
Simon Willnauer 68ba0d9bab migrate branch for cloud-azure 2015-06-05 13:12:34 +02:00
Simon Willnauer 553c913aa3 add cloud-azure module 2015-06-05 13:12:32 +02:00
Simon Willnauer f10a971e8f add cloud-gce module 2015-06-05 13:12:30 +02:00
Simon Willnauer 3ab627b72f migrate branch for cloud-gce 2015-06-05 13:12:30 +02:00
Simon Willnauer 4e9e74da97 add cloud-gce module 2015-06-05 13:12:28 +02:00
Simon Willnauer 682ad1e5f9 add analysis-icu module 2015-06-05 13:12:26 +02:00
Simon Willnauer f3228e394d migrate branch for analysis-icu 2015-06-05 13:12:26 +02:00
Simon Willnauer ed3cc8d034 add analysis-icu module 2015-06-05 13:12:23 +02:00
Simon Willnauer d0be9e5f7a add analysis-phonetic module 2015-06-05 13:12:22 +02:00
Simon Willnauer 0d328b07bd migrate branch for analysis-phonetic 2015-06-05 13:12:21 +02:00
Simon Willnauer 8d9ff57bc3 add analysis-phonetic module 2015-06-05 13:12:20 +02:00
Simon Willnauer 1cb86507d4 add analysis-stempel module 2015-06-05 13:12:18 +02:00
Simon Willnauer 263bc6bff8 migrate branch for analysis-stempel 2015-06-05 13:12:17 +02:00