Commit Graph

1740 Commits

Author SHA1 Message Date
Jason Tedor 3d4fac6256 Remove unused import in o.e.t.UpdateThreadPoolSettingsTests 2015-09-15 10:48:01 -04:00
Jason Tedor 2357e65fcc Remove unused import in o.e.a.t.TermVectorsFilter 2015-09-15 10:46:59 -04:00
Michael McCandless 028acdd06f Merge pull request #13548 from mikemccand/indexing_memory_controller
Improve IndexingMemoryController a bit:
      - promptly push indexing buffer changes to IndexWriter, instead of waiting for next refresh/flush
      - don't wait for merges to finish before dropping a shards's indexing buffer to 512 KB once it's inactive
      - fix NPE if indices.memory.index_buffer_size is in node's settings with a bytes (not %) unit
      - add some more logger.debug
2015-09-15 10:39:10 -04:00
Jason Tedor 2833ad4d6f Remove unused import in o.e.t.ESIntegTestCase 2015-09-15 10:37:57 -04:00
Jason Tedor b3c6327caf Remove and forbid use of com.google.common.base.Joiner
This commit removes and now forbids all uses of
com.google.common.base.Joiner across the codebase. This is one of many
steps in the eventual removal of Guava as a dependency.

Relates #13224
2015-09-15 10:30:42 -04:00
Jason Tedor 7a29febd72 Remove and forbid use of com.google.common.math.LongMath
This commit removes and now forbids all uses of
com.google.common.math.LongMath across the codebase. This is one step
of many in the eventual removal of Guava as a dependency.
2015-09-15 10:28:02 -04:00
Simon Willnauer ff4a11aa32 Replace and ban next batch of Guava classes
This commit replaces and bans:
 * com.google.common.util.concurrent.UncheckedExecutionException
 * com.google.common.util.concurrent.AtomicLongMap
 * com.google.common.primitives.Longs
 * com.google.common.io.ByteStreams
 * com.google.common.collect.UnmodifiableIterator
 * com.google.common.collect.ObjectArrays
 * com.google.common.collect.Multimap
 * com.google.common.collect.MultimapBuilder

Relates to #13224
2015-09-15 14:44:41 +02:00
Jason Tedor 527ab95c39 Remove and forbid use of com.google.common.collect.Iterables
This commit removes and now forbids all uses of
com.google.common.collect.Iterables across the codebase. This is one of
many steps in the eventual removal of Guava as a dependency.

Relates #13224
2015-09-15 07:47:36 -04:00
Jason Tedor acbede48d7 Completely remove Preconditions.checkArgument 2015-09-15 07:23:35 -04:00
Jason Tedor 78bff93ab1 Completely remove Preconditions.checkState 2015-09-15 07:23:34 -04:00
Jason Tedor f5c408535d Remove and forbid use of com.google.common.base.Preconditions
This commit removes and now forbids all uses of
com.google.common.base.Preconditions across the codebase. This is one
of many steps in the eventual removal of Guava as a dependency.

Relates #13224
2015-09-15 07:23:34 -04:00
Christoph Büscher dbaa90a3f8 Merge branch 'master' into feature/query-refactoring 2015-09-15 11:47:42 +02:00
Lee Hinman de2fcd4be4 [TEST] Use greaterThanOrEqualTo for testClusterInfoServiceCollectsInformation 2015-09-14 21:10:13 -06:00
Michael McCandless ceab9a898b remove index.buffer_size setting 2015-09-14 16:52:32 -04:00
Christoph Büscher 01ea42619a Tests: re-enable SearchQueryIT#testIndicesQuerySkipParsing and fixing exception message 2015-09-14 22:41:37 +02:00
Nik Everett 713009df37 Merge branch 'Output plugin info only in verbose mode'
https://github.com/elastic/elasticsearch/pull/12908

Moved output to verbose level and additionally changed the plugin info output format

Before:

```shell
PluginInfo{name='cloud-aws', description='The Amazon Web Service (AWS) Cloud plugin allows to use AWS API for the unicast discovery mechanism and add S3 repositories.', site=false, jvm=true, classname=org.elasticsearch.plugin.cloud.aws.CloudAwsPlugin, isolated=true, version='2.1.0-SNAPSHOT'}
```

After:

```shell
- Plugin information:
Name: cloud-aws
Description: The Amazon Web Service (AWS) Cloud plugin allows to use AWS API for the unicast discovery mechanism and add S3 repositories.
Site: false
Version: 2.1.0-SNAPSHOT
JVM: true
* Classname: org.elasticsearch.plugin.cloud.aws.CloudAwsPlugin
* Isolated: true
```

Fixes #12907
2015-09-14 15:36:20 -04:00
Britta Weber 6ea6876839 [test] wait until master actually has applied cluster state that contains second node
If we don't wait we might retrieve the cluster state before second node
was added when we try to add the delegate in which the discovery node for
node_2 is null.
2015-09-14 18:56:40 +02:00
Nik Everett 3fa3b1597f [test] PluginManager plugin info printing
Also removes @Test in keeping with Elasticsearch's slow migration away from
the annotation in favor of test methods named testSomeThing.
2015-09-14 11:32:33 -04:00
Britta Weber ba4712fb38 [test] make sure ESSingleNodeTestCase waits after starting node until all blocks removed
When a single node starts up it will first elect itself as master and then tries to
recover the cluster state or, if there is none,  initialize an empty one and publish it.
Until it has done that, the cluster state will contain a global block and
requests might fail with
SERVICE_UNAVAILABLE/1/state not recovered / initialized
We need to wait for green.
2015-09-14 17:06:34 +02:00
Lee Hinman 004f23500e [TEST] Add AwaitsFix for https://github.com/elastic/elasticsearch/issues/13558 2015-09-14 08:40:36 -06:00
Britta Weber 997616a42b [test] test that shards are deleted if the first attempt after relocation failed and index deleted next 2015-09-14 16:09:15 +02:00
Michael McCandless 2edf9cf82d logger.trace not logger.debug; fix test to verify bytes-sized index buffer size in node settings on node init took 2015-09-14 08:41:31 -04:00
Christoph Büscher 90f24c1a79 Merge branch 'master' into feature/query-refactoring
Conflicts:
	core/src/main/java/org/elasticsearch/index/query/GeoDistanceRangeQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/GeoPolygonQueryBuilder.java
	core/src/main/java/org/elasticsearch/index/query/GeohashCellQuery.java
	core/src/main/java/org/elasticsearch/index/query/WrapperQueryBuilder.java
	core/src/test/java/org/elasticsearch/search/child/ChildQuerySearchIT.java
2015-09-14 14:31:01 +02:00
Simon Willnauer bd025080c4 Don't use forbidden API in FailableCache 2015-09-14 14:00:50 +02:00
Jason Tedor 57c5efacf6 Fix test bug in o.e.a.s.b.n.TransportBroadcastByNodeActionTests
This commit fixes a test bug in
o.e.a.s.b.n.TransportBroadcastByNodeActionTests. Namely, the randomized
test allowed for the creation of cluster states that allocated indices
having zero shards. This ultimately surfaced in a
NoSuchElementException when attempting to iterate over the nonexistent
shards. The fix is merely to draw the random number of shards from 1 to
10 instead of 0 to 10.
2015-09-14 07:54:05 -04:00
Simon Willnauer 30bd46ab25 Replace LoadingCache usage with a simple ConcurrentHashMap
This commit replaces the usage of LoadedCache with a simple CHM and calls
to computeIfAbsent and adds LoadingCache and CacheLoader to forbidden APIs

Relates to #13224
2015-09-14 13:25:58 +02:00
Alexander Reelsen a96350d785 Release: Add 1.7.2 bwc indices/versions
* Added BWC indices
* Added snapshot version to Version.java
* Fixed create_bwc_index to use localhost instead of localhost and 127.0.0.1 (problem with ipv4/6 setup)
2015-09-14 12:58:46 +02:00
Michael McCandless 698da29573 pull out string constants for settings, and add javadocs 2015-09-14 05:58:21 -04:00
Michael McCandless 8dc08f70b4 remove forced refresh on settings update 2015-09-14 05:32:59 -04:00
Michael McCandless 274044e7dd push settings change before refresh, when shard becomes inactive 2015-09-14 05:18:07 -04:00
Christoph Büscher adaa998225 Query Refactoring: adding illegal argument checks to MatchQueryBuilder 2015-09-14 11:12:56 +02:00
Michael McCandless 9148382be0 even if engineUnsafe() is null we still must update the EngineConfig 2015-09-14 05:12:06 -04:00
Simon Willnauer 40959068d5 Remove and forbid use of guava Function, Charsets, Collections2
This commit removes and now forbids all uses of
Function, Charsets, Collections2  across the codebase. This
is one of many steps in the eventual removal of Guava as a dependency.

Relates #13224
2015-09-14 10:27:12 +02:00
Robert Muir d0deb28336 Merge pull request #13550 from rmuir/this_test_is_fucked_up
NewPathForShardTests is broken every which way from Sunday.
2015-09-14 04:13:55 -04:00
Boaz Leskes 15f4863ef6 Discovery: Add two phased commit to Cluster State publishing
When publishing a new cluster state, the master will send it to all the node of the cluster, noting down how many *master* nodes responded successfully. The nodes do not yet process the new cluster state, but rather park it in memory. As soon as at least minimum master nodes have ack-ed the cluster state change, it is committed and a commit request is sent to all the node that responded so far (and will respond in the future). Once receiving the commit requests the nodes continue to process the cluster state change as they did before this change.

A few notable comments:
1. For this change to have effect, min master nodes must be configured.
2. All basic cluster state validation is done in the first phase of publish and is thus now part of `ShardOperationResult`
3. A new `COMMIT_TIMEOUT` settings is introduced, dictating how long a master should wait for nodes to ack the first phase. Unlike `PUBLISH_TIMEOUT`, if waiting for a commit times out, the cluster state change will be rejected.
4. Failing to achieve a min master node of acks, will cause the master to step down as it clearly doesn't have enough active followers.
5. Previously there was a short window between the moment a master lost it's followers and it stepping down because of node fault detection failures. In this short window, the master could process any change (but fail to publish it). This PR closes this gap to 0.
6. A dedicated pending cluster states queue was added to keep pending non-comitted cluster states and manage the logic around processing committed cluster states. See #13303 for details.

Closes #13062 , Closes #13303
2015-09-14 09:55:41 +02:00
Simon Willnauer 562126a332 Clenup InternalClusterInfoService
This commit allows to refresh the info service in a blocking fashion
which allows tests to prevent installing listeners alltogether and
makes the class easier to test. Installing a listnener is always subject
to concurrent modifications where the listener might be called mulitple times
or with stale information which causes tests to fail.
2015-09-14 09:37:19 +02:00
Simon Willnauer a0c10850d9 Use Supplier instead of Reflection
Java 8 allows for method references which in-turn will cause
compile errors if a method is not visible while reflection fails late
and maybe too late. We can now register Request instances via FooRequest::new
instead of passing FooRequest.class and call it's ctor via reflection.
2015-09-14 09:11:32 +02:00
Boaz Leskes 564e1eca42 merge from master 2015-09-14 09:03:45 +02:00
Tanguy Leroux 561f5b590f Fix AwarenessAllocationIT.testAwarenessZones test
This test sometimes fails because the first node is elected as master and waits 30s for incoming joins but in the meanwhile the 3 other nodes form a cluster on their side. The index will be created and its shards allocated on these 3 nodes, then the test checks for the number of shards on each node (it should be 2 or 3) but because the first node has not fully join the cluster yet one node will have 5 shards.

 closes #13305
2015-09-14 08:57:31 +02:00
Robert Muir 4a9b8a94c9 NewPathForShardTests is broken every which way from Sunday.
1. FileSystem wrapping code is broken, thats why you get providermismatch exception!
   Instead of fixing this, it SuppressesForbidden!!!!
2. Because it uses SuppressForbidden on the test, the whole thing is lenient, it uses java.io.File for example!
3. Of course it fails consistently on windows because it can't remove files, because it leaks file handles (locks)
   like a sieve since it does not close node environment. With correct wrapping this is always detected by e.g.
   our leak detection FS. Instead of fixing the leak, it assumesFalse(WINDOWS) !!!!!

I do not know how this snuck past me, but I need this fixed to remove setAccessible.
2015-09-14 01:30:09 -04:00
Jason Tedor b4adad7c0a Code cleanup in TransportBroadcastByNodeAction 2015-09-13 21:35:24 -04:00
Michael McCandless c43ede5416 Improve IndexingMemoryController a bit:
- promptly push indexing buffer changes to IndexWriter, instead of waiting for next refresh/flush
  - don't wait for merges to finish before dropping a shards's indexing buffer to 512 KB
  - fix NPE if indices.memory.index_buffer_size is in node's settings with a bytes (not %) unit
  - add some more logger.debug
2015-09-13 18:19:41 -04:00
Boaz Leskes a758eec78a Recovery: failed to properly ack translog ops during wait on mapping changes
During the second phase of recovery, replayed transaction log entries may need to wait on mapping changes that have not yet propagated to the target node. Currently we correctly replay the operation at a later stage, but we acknowledge the replay request before actually performing the work.

Example failure: http://build-us-00.elastic.co/job/es_feature_two_phase_pub/859/

Closes #13535
2015-09-13 21:14:40 +02:00
Simon Willnauer 5098dcaf96 [TEST] Don't rely on currentTimeMillis in RoutingServiceTests
If the machine is very slow this test fails if the delta of the unallocaiton
timestamp and the last scheduled delay is greater than the scheduled delay time.
2015-09-13 16:58:56 +02:00
Britta Weber 5a35a8582b [test] mute and add link to issue
see #13542
2015-09-13 15:06:28 +02:00
Robert Muir 7703aab95a support mocking these components in tests without setAccessible 2015-09-12 20:49:20 -04:00
Robert Muir b16e1569fe Remove all setAccessible in tests and forbid 2015-09-12 19:46:39 -04:00
Robert Muir 3a0dd59a15 Merge pull request #13537 from rmuir/die_setaccessible_in_tests_die
Remove easy uses of setAccessible in tests.
2015-09-12 17:19:38 -04:00
Robert Muir 6a15518e8c don't list this class twice (vi typo) 2015-09-12 17:07:18 -04:00
Robert Muir 174ca77ebf Hack around aws security hole of accessing sun.security.ssl, s3 repository works on java 9 again
Today this is really horrible, and we have a PR sent to fix it, but nobody
does anything: https://github.com/aws/aws-sdk-java/pull/432

With java 9, we cannot even grant the permission, this kind of sheistiness is not allowed,
and s3 repository is completely broken.

The problem is their code is still broken, and won't handle neither SecurityException (our PR)
nor the new InaccessibleObjectException they will get from java 9.

We use a really hacky hack to deliver an exception that their code catches (IllegalAccessException) instead.

This means s3 repository is working on java 9, and we close off access to sun.security.ssl completely
2015-09-12 14:16:16 -04:00
Robert Muir a7418cd9bb Remove easy uses of setAccessible in tests.
Don't worry, I will fix the rest. But some of those remaining will need a lucene upgrade,
we need to add a getter or two for tests to do things cleanly.
2015-09-12 11:57:30 -04:00
Simon Willnauer f2b4ba9a7c [TEST] BooleanQuery iteration order is not consistent 2015-09-12 11:18:44 +02:00
Robert Muir d6f56030d8 ban setAccessible from core code.
In addition to being a big security problem, setAccessible is a risk
for java 9 migration. We need to clean up our code so we can ban it
and eventually enforce this with security manager for third-party code, too,
or we may have problems.

Instead of using setAccessible, use the correct modifier (e.g. public).

TODO: ban in tests
TODO: ban in security manager at runtime
2015-09-12 02:11:06 -04:00
Robert Muir 3ec5cf6263 get our stats back by reflecting mxbeans correctly 2015-09-11 22:28:37 -04:00
Robert Muir 8be7fde181 Revert "Update monitor probe tests for java 9: this stuff is no longer accessible"
This reverts commit af2df9aef6.
2015-09-11 22:21:02 -04:00
Jason Tedor d6d8d30d47 Remove and forbid use of com.google.common.collect.ImmutableSortedMap
This commit removes and now forbids all uses of
com.google.common.collect.ImmutableSortedMap across the codebase. This
is one of many steps in the eventual removal of Guava as a dependency.

Relates #13224
2015-09-11 17:42:19 -04:00
Igor Motov 39ca45050a Simplify the BlobContainer blob writing interface
Instead of asking blob store to create output for posting blob content, this change provides that content of the blob to the blob store for writing. This will significantly simplify the  interface for S3 and Azure plugins.
2015-09-11 16:39:58 -04:00
Simon Willnauer 7b0f086946 Remove and forbid use of several com.google.common.util. classes
This commit replaces:
 * com.google.common.util.concurrent.ListenableFuture
 * com.google.common.util.concurrent.SettableFuture
 * com.google.common.util.concurrent.Futures
 * com.google.common.util.concurrent.MoreExecutors

And forbits its usage via forbidden APIs. This is one of
many steps in the eventual removal of Guava as a dependency.

Relates to #13224
2015-09-11 21:15:12 +02:00
Igor Motov 3fd7d95f52 Snapshot restore request should accept indices options
Fixes an issue introduced in #10744 as a result of which the restore request stopped accepting indices options such as ignore_unavailable.

Fixes #13335
2015-09-11 13:43:00 -04:00
Simon Willnauer e35293b055 Merge pull request #13518 from s1monw/cleanup_search_request
Cleanup SearchRequest & SearchRequestBuilder
2015-09-11 19:41:52 +02:00
Jason Tedor 67dea6b3f6 Merge pull request #13517 from jasontedor/settings-should-not-leak-guava-dependency
Stop o.e.c.s.Settings from leaking Guava dependency
2015-09-11 11:44:38 -04:00
Simon Willnauer 01a2ba1b95 Cleanup SearchRequest & SearchRequestBuilder
We have a gazillion ways to specify the source of the search request.
There is no need to so many we can reduce them dramatically and also remove
some deprecated API.
2015-09-11 17:38:35 +02:00
Lee Hinman 016ba3531a Merge remote-tracking branch 'dakrone/remove-disable-decider' 2015-09-11 09:28:13 -06:00
Boaz Leskes e370b83685 fix old style plugin injection 2015-09-11 17:10:41 +02:00
Jason Tedor f1ff6706aa Stop o.e.c.s.Settings from leaking Guava dependency
The public org.elasticsearch.common.settings.Settings#getAsMap method
leaks the dependency on Guava by returning a
com.google.common.collect.ImmutableMap. The leaking of this dependency
should be removed in preparation for the eventual complete removal of
Guava as a dependency.

Relates #13224
2015-09-11 11:09:08 -04:00
Nicholas Knize bc75fe4a45 add a weighted centroid to the geohash_grid aggregator and cut over to lucene GeoHashUtils 2015-09-11 09:57:08 -05:00
Britta Weber 73b4391a46 [test] fix urls for windows 2015-09-11 16:25:19 +02:00
Britta Weber 53bb17c83c [test] use tmp dir otherwise the security manager will complain 2015-09-11 14:48:19 +02:00
javanna 79c58a122e adapt to upstream lucene changes 2015-09-11 14:37:27 +02:00
javanna e522458735 remove guava usage from IdsQueryBuilder 2015-09-11 14:20:21 +02:00
javanna 73f7df510e Merge branch 'master' into feature/query-refactoring
Conflicts:
	core/src/main/java/org/elasticsearch/index/query/HasChildQueryBuilder.java
	core/src/main/java/org/elasticsearch/index/query/HasChildQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/HasParentQueryBuilder.java
	core/src/main/java/org/elasticsearch/index/query/HasParentQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/functionscore/FunctionScoreQueryBuilder.java
	core/src/main/java/org/elasticsearch/index/query/functionscore/FunctionScoreQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/functionscore/factor/FactorParser.java
	core/src/test/java/org/elasticsearch/ExceptionSerializationTests.java
	core/src/test/java/org/elasticsearch/benchmark/search/child/ChildSearchBenchmark.java
	core/src/test/java/org/elasticsearch/benchmark/search/child/ChildSearchShortCircuitBenchmark.java
	core/src/test/java/org/elasticsearch/index/query/SimpleIndexQueryParserTests.java
	core/src/test/java/org/elasticsearch/percolator/PercolatorIT.java
	core/src/test/java/org/elasticsearch/search/child/ChildQuerySearchIT.java
	docs/reference/query-dsl/has-parent-query.asciidoc
2015-09-11 14:15:12 +02:00
Masaru Hasegawa 6a9ef99d0f Take relocating shard into consideration during awareness allocation
Previous fix #12551 counted twice for relocating shard (source and target).
Fix it to consider only target node.
2015-09-11 21:09:31 +09:00
Simon Willnauer 6b4699fbf7 Only add unicast provider if all settings are present
Closes #13492
2015-09-11 13:21:03 +02:00
Britta Weber 2c618a11de plugins: fix print of url when it contains spaces 2015-09-11 13:13:48 +02:00
javanna 380fe52b30 Java api: remove filterBuilder element from FunctionScoreQueryBuilder
The filter element has been deprecated in the function_score query parser. Whenever a filter is found it gets wrapped into a query automatically. The filter in the java api builder is always null, there is no way to set its value, just a leftover.
2015-09-11 13:08:33 +02:00
javanna a641c7ae52 Function score query: remove deprecated support for boost_factor
boost_factor was deprecated in 1.4.0.beta1. We can remove the support for it in 3.0, its replacement is `weight`.

Closes #13510
2015-09-11 12:58:25 +02:00
Colin Goodheart-Smithe d49a744b7e [TEST] Removed @Repeat Annotation 2015-09-11 09:36:25 +01:00
Colin Goodheart-Smithe f8c47345b0 Fixed Typo in Test Name 2015-09-11 09:30:55 +01:00
Colin Goodheart-Smithe 2118936deb Query Refactoring: Refactor of GeoShapeQuery
Moving the query building functionality from the parser to the builders
new toQuery() method analogous to other recent query refactorings.

Relates to #10217

PR goes against the query-refactoring branch
2015-09-11 09:26:09 +01:00
Simon Willnauer 853b7fdb7c Fold MoreLikeThisFetchService into MoreLikeThisQueryParser
now that we have a Client on the Shard context we can fold
the doc fetching into the parser / builder.

Relates to #13488
2015-09-11 10:21:40 +02:00
Boaz Leskes 1b8047e51c merge master 2015-09-11 09:56:13 +02:00
Adrien Grand e988a9edc6 Merge pull request #13439 from rmuir/1702090
upgrade lucene to r1702265
2015-09-11 09:48:43 +02:00
Boaz Leskes 80b59e0d66 Discovery: Add a dedicate queue for incoming ClusterStates
The initial implementation of two phase commit based cluster state publishing (#13062) relied on a single in memory "pending" cluster state that is only processed by ZenDiscovery once committed by the master. While this is fine on it's own, it resulted in an issue with acknowledged APIs, such as the open index API, in the extreme case where a node falls behind and receives a commit message after a new cluster state has been published. Specifically:

1) Master receives and acked-API call and publishes cluster state CS1
2) Master waits for a min-master nodes to receives CS1 and commits it.
3) All nodes that have responded to CS1 are sent a commit message, however, node N didn't respond yet
4) Master waits for publish timeout (defaults to 30s) for all nodes to process the commit. Node N fails to do so.
5) Master publishes a cluster state CS2. Node N responds to cluster state CS1's publishing but receives cluster state CS2 before the commit for CS1 arrives.
6) The commit message for cluster CS1 is processed on node N, but fails because CS2 is pending. This caused the acked API in step 1 to return (but CS2 , is not yet processed).

In this case, the action indicated by CS1 is not yet executed on node N and therefore the acked API calls return pre-maturely. Note that once CS2 is processed but the change in CS1 takes effect (cluster state operations are safe to batch and we do so all the time).

An example failure can be found on: http://build-us-00.elastic.co/job/es_feature_two_phase_pub/314/

This commit extracts the already existing pending cluster state queue (processNewClusterStates) from ZenDiscovery into it's own class, which serves as a temporary container for in-flight cluster states. Once committed the cluster states are transferred to ZenDiscovery as they used to before. This allows "lagging" cluster states to still be successfully committed and processed (and likely to be ignored as a newer cluster state has already been processed).

As a side effect, all batching logic is now extracted from ZenDiscovery and is unit tested.
2015-09-11 09:23:41 +02:00
Simon Willnauer 94a37d486f Remove ShapeFetchService and TermsLookupFetchService and use a Client instead
This commit removes all the optional injects etc. for the FetchServices and
provides a Client via IndexQueryParserService / Context. This allows direct
injection instead of optional injection. It also allows to remove all the
unnecessary services and use the fetch code where it belongs.
This commit also adds testing infrastructure for intercepting client calls
to AbstractQueryTestCase to support GET calls in query tests.
2015-09-11 09:07:18 +02:00
Colin Goodheart-Smithe cac72c9207 Query Refactoring: Refactor of GeoPolygonQuery
Moving the query building functionality from the parser to the builders
new toQuery() method analogous to other recent query refactorings.

Relates to #10217

PR goes against the query-refactoring branch
2015-09-11 07:54:00 +01:00
Masaru Hasegawa 5ae00a6129 Take initializing shards into consideration during awareness allocation
It makes decision consistent.
Fixes #12522
2015-09-11 13:13:36 +09:00
Ryan Ernst f3142e0c93 Merge pull request #13497 from rjernst/test_jar_cleanup
Reorganize sharing of constants for mock cluster info service
2015-09-10 21:10:29 -07:00
Jason Tedor b5b22d4b01 Remove and forbid use of com.google.common.collect.Queues
This commit removes and now forbids all uses of
com.google.common.collect.Queues across the codebase. This is one of
many steps in the eventual removal of Guava as a dependency.

Relates #13224
2015-09-10 23:59:59 -04:00
Jason Tedor 65353b8a32 Remove and forbid use of com.google.common.collect.ImmutableSortedSet
This commit removes and now forbids all uses of
com.google.common.collect.ImmutableSortedSet across the codebase. This
is one of many steps in the eventual removal of Guava as a dependency.

Relates #13224
2015-09-10 23:33:55 -04:00
Robert Muir af2df9aef6 Update monitor probe tests for java 9: this stuff is no longer accessible 2015-09-10 23:18:58 -04:00
Ryan Ernst 4a9f5bcf1d Tests: Reorganize sharing of constants for mock cluster info service
The MockInternalClusterInfoService depends on a constant and helper
method from actual test cases. This moves the constant and helper method
into the mock itself. Without this change, the classes put into the test
jar are not completely useable on their own (since this mock is in the
test jar, but the test cases are not).
2015-09-10 20:08:04 -07:00
Robert Muir 6ba4d132df Disable some exception serialization asserts on java 9, likely StackTraceElement has new stuff in it and we have to think about what to do 2015-09-10 22:45:53 -04:00
Robert Muir 552d5b0f65 Add java 9 support for bootclasspath to JvmInfo (it throws UOE otherwise which is not properly handled) 2015-09-10 22:15:27 -04:00
Robert Muir c663d9b6a5 Merge branch 'jarhell2.0' into 1702090 2015-09-10 22:07:37 -04:00
Robert Muir 7aeff9f526 fix merge 2015-09-10 22:07:20 -04:00
Robert Muir dd208002c9 Merge branch 'master' into 1702090 2015-09-10 21:41:07 -04:00
Robert Muir 6f59e3fca1 cutover jarhell to parsing classpath all the time 2015-09-10 21:31:25 -04:00
Jason Tedor 5d4d34ab16 Remove and forbid use of c.g.c.b.Preconditions#checkNotNull
This commit removes and now forbids all uses of
com.google.common.base.Preconditions#checkNotNull across the codebase.
This is one of many steps in the eventual removal of Guava as a
dependency.

Relates #13224
2015-09-10 17:57:00 -04:00
Robert Muir c1f2fc76c2 Upgrade lucene to r1702090
The semantics of the `boost` parameter for `function_score` changed. This is
due to the fact that Lucene now requires that query boosts and top-level boosts
are applied the same way.
2015-09-10 23:36:43 +02:00
Nik Everett e4981968ad [search] Limit the size of the result window
Requesting a million hits, or page 100,000 is always a bad idea, but users
may not be aware of this. This adds a per-index limit on the maximum size +
from that can be requested which defaults to 10,000.

This should not interfere with deep-scrolling.

Closes #9311
2015-09-10 15:38:29 -04:00
Robert Muir 6cdcc805cc hack jython tests to work on windows
windows needs access to the "root" holding the jar file (see https://github.com/int3/jython/blob/master/src/org/python/core/PySystemState.java#L571-L616)

Its different on windows, because when canonicalizing the file (case sensitivity), it needs access to the file.

Closes #13476
2015-09-10 14:59:23 -04:00
Ryan Ernst ca8867a334 Test: Fix tests looking for old indexes to resolve against the correct dir 2015-09-10 11:07:02 -07:00
Lee Hinman b21d3d2fb5 [TEST] Additional logging for testDelayedUnassignedScheduleReroute
Relates to #13485
2015-09-10 11:44:09 -06:00
Ryan Ernst 47de1bd923 Merge pull request #13443 from rjernst/simplify_bwc_path
Move static bwc indexes to a shared location
2015-09-10 10:31:50 -07:00
Jason Tedor 3bd1d38176 Remove and forbid use of com.google.common.collect.Sets
This commit removes and now forbids all uses of
com.google.common.collect.Sets across the codebase. This is one of many
steps in the eventual removal of Guava as a dependency.

Relates #13224
2015-09-10 11:22:09 -04:00
Martijn van Groningen ab0847e0df parent/child: several cleanups
* Dropped ScoreType in favour of Lucene's ScoreMode
* Removed `score_type` option from `has_child` and `has_parent` queries in favour for the already existing `score_mode` option.
* Removed the score mode `sum` in favour for the already existing `total` score mode. (`sum` doesn't exist in Lucene's ScoreMode class)
* If `max_children` is set to `0` it now really means that zero children are allowed to match.
2015-09-10 17:15:41 +02:00
Nik Everett 60a2dd7020 Merge pull request #13189 from nik9000/fix_nodeattrs_test
Fix test for _cat/nodeattrs
2015-09-10 11:12:01 -04:00
Nik Everett 56c3471851 Fix test for _cat/nodeattrs
Adds a node attribute to all test runs and uses the attribute to test
`_cat/nodeattrs`.

Note that its quite possible create an impressively slow regex while doing
this and you have to be careful. See comment in commit for more if curious.

Closes #12558
2015-09-10 10:50:51 -04:00
Colin Goodheart-Smithe 53d29e51e0 Aggregations: Pipeline Aggregations at the root of the agg tree are now validated
Previously PipelineAggregatorFactory's at the root to the agg tree (top-level aggs) were not validated. This commit adds a call to PipelineAggregatoFactory.validate() for that case.

Closes #13179
2015-09-10 15:44:52 +01:00
Nik Everett 3839d15ea0 Merge branch 'pr/13130' 2015-09-10 10:13:01 -04:00
javanna 2e68738429 adapt to upstream changes 2015-09-10 16:04:55 +02:00
javanna 56b3db6ba3 Merge branch 'master' into feature/query-refactoring
Conflicts:
	core/src/main/java/org/elasticsearch/index/query/AndQueryBuilder.java
	core/src/main/java/org/elasticsearch/index/query/AndQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/FQueryFilterParser.java
	core/src/main/java/org/elasticsearch/index/query/FilteredQueryBuilder.java
	core/src/main/java/org/elasticsearch/index/query/FilteredQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/HasChildQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/IndicesQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/LimitQueryBuilder.java
	core/src/main/java/org/elasticsearch/index/query/LimitQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/MultiMatchQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/OrQueryBuilder.java
	core/src/main/java/org/elasticsearch/index/query/OrQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/QueryFilterBuilder.java
	core/src/main/java/org/elasticsearch/index/query/QueryFilterParser.java
	core/src/main/java/org/elasticsearch/index/query/QueryParseContext.java
	core/src/main/java/org/elasticsearch/index/query/ScriptQueryParser.java
	core/src/main/java/org/elasticsearch/indices/query/IndicesQueriesRegistry.java
	core/src/main/java/org/elasticsearch/search/sort/GeoDistanceSortParser.java
	core/src/main/java/org/elasticsearch/search/sort/SortParseElement.java
	core/src/test/java/org/elasticsearch/benchmark/search/child/ChildSearchShortCircuitBenchmark.java
	core/src/test/java/org/elasticsearch/index/query/SimpleIndexQueryParserTests.java
2015-09-10 15:52:28 +02:00
Martijn van Groningen a0fea6d365 set query cache to null 2015-09-10 15:19:12 +02:00
André Carvalho 9802c38fa7 [Stats] Adds counter for failed indexing requests
Adds the stats to the _cat/indices, _cat/shards, and the _stats apis.

Closes #8938
2015-09-10 09:03:24 -04:00
Britta Weber c10f116a84 [test] don't catch AssertionError from indexRandom(). just index and catch MapperParsingException 2015-09-10 14:51:23 +02:00
Simon Willnauer 8d2a2f8d66 Split NestedQueryParser into toQuery and formXContent
This commit splits NestedQueryParser into toQuery and fromXContent.

Relates to #10217
2015-09-10 14:40:17 +02:00
Simon Willnauer de79faec70 Don't be lenient if JarHell is found
Closes #13404
2015-09-10 14:21:33 +02:00
javanna 1dc985a2c4 Remove ordinal member from Operator enum
Use the already available enum ordinal instead.

Closes #13456
2015-09-10 14:19:10 +02:00
Simon Willnauer e2bbfbe1d5 Use AbstractQueryTestCase in MatchQueryBuilderTests 2015-09-10 13:21:59 +02:00
Simon Willnauer feb5bdb23e Merge pull request #13441 from s1monw/fix_query_base_test
Cleanup QueryTest framework and be more strict
2015-09-10 13:12:26 +02:00
Simon Willnauer de3c9a23e9 Cleanup code and always use strict parsing on query tests 2015-09-10 12:58:25 +02:00
Martijn van Groningen 314e1c8a3e parent/child: has_child has_parent queries shouldn't require search context 2015-09-10 12:47:06 +02:00
Martijn van Groningen 2eadc6d595 nested sorting: If sorting by nested field then the `nested_path` should always be specified.
Closes #13420
2015-09-10 12:21:12 +02:00
Martijn van Groningen 89159f073c Merge pull request #13430 from martijnvg/pc/parent_field_always_store_doc_values
Parent field mapper should always store doc values join field.
2015-09-10 12:15:39 +02:00
Christoph Büscher 90fac17a2d Query refactoring: MatchQueryBuilder
This add equals, hashcode, read/write methods, separates toQuery and JSON parsing and adds tests.
Also moving MatchQueryBuilder.Type to MatchQuery to MatchQuery, adding serialization and hashcode,
equals there.

Relates to #10217
2015-09-10 12:14:55 +02:00
javanna 5cc423a229 delete unused XContentStructure class 2015-09-10 11:02:10 +02:00
Simon Willnauer 9f722a6674 Cleanup QueryTest framework and be more strict
This commit contains:
 * renaming BaseQueryTestCase to AbstractQueryTestCase
 * uses always STRICT parsing when parsing from builders XContent
 * ensures that SearchContext is only but always available during toQuery but never during parse phase
 * adds a way to override the default ParseFieldMatcher to allow queries to set deprecated API by default
2015-09-10 10:57:13 +02:00
Simon Willnauer 7fff399834 Reenable XTestSecurityManager 2015-09-10 10:49:13 +02:00
Adrien Grand e89ec468c3 Remove support for deprecated queries.
This removes support for the `and`, `or`, `fquery`, `limit` and `filtered`
queries. `query` is still supported until #13326 is fixed.
2015-09-10 10:38:11 +02:00
javanna 484fcd49e5 Query refactoring: query_string
Relates to #10217
2015-09-10 10:09:24 +02:00
Simon Willnauer 7ab0e2c532 Swap out XTestSecurityManager until we figured out why it crashes the JVM 2015-09-10 09:57:02 +02:00
Ryan Ernst c1dd9b8a98 add temp test security manager to test jar 2015-09-09 16:08:50 -07:00
Ryan Ernst 5c810eff59 Add helper method so indices path is not copied all over 2015-09-09 15:51:34 -07:00
Ryan Ernst 8f75c2b3a8 Tests: Move static bwc indexes to a shared location
There are a few tests that currently use the statically generated
backcompat indexes. This change moves them to a shared location, so they
no longer have to build a path based on the package name of the old
index tests.
2015-09-09 15:42:26 -07:00
Robert Muir e67140b954 Merge pull request #13442 from rmuir/ide_noise
Remove noise when IDE test runners try to System.exit
2015-09-09 18:22:05 -04:00
Ryan Ernst 7854ccf797 Tweak intellij junit package name 2015-09-09 15:17:48 -07:00
Robert Muir e7b98ac513 lie to the stupid license checker 2015-09-09 18:03:32 -04:00
Robert Muir c273895d82 Remove noise when IDE test runners try to System.exit 2015-09-09 18:00:46 -04:00
Jason Tedor 6b19aebab0 Merge pull request #13438 from jasontedor/maps-be-gone
Remove and forbid use of com.google.common.collect.Maps
2015-09-09 17:58:44 -04:00
Jason Tedor 2a5412ebf0 Remove and forbid use of com.google.common.collect.Maps
This commit removes and now forbids all uses of
com.google.common.collect.Maps across the codebase. This is one of many
steps in the eventual removal of Guava as a dependency.

Relates #13224
2015-09-09 17:41:41 -04:00
Jason Tedor b18ee60f80 Merge pull request #13440 from jasontedor/remove-thread-dumps-on-test-failures
Do not dump stack traces of threads on test failure
2015-09-09 17:23:41 -04:00
Jason Tedor 3dcb1a81c8 Do not dump stack traces of threads on test failure
Reverts 878301c795, relates #12425
2015-09-09 17:07:13 -04:00
Ryan Ernst 9d71f0726e Merge pull request #13410 from rjernst/you_were_warned
Fix compiler warnings
2015-09-09 14:04:54 -07:00
Ryan Ernst 9e8a90a657 Add xlint ignores for warning classes, where appropriate. 2015-09-09 12:47:07 -07:00
Alex Ksikes 274611c7c0 [TEST] fix template query with unmapped fields 2015-09-09 21:13:14 +02:00
gmarz 9e6115b066 Packaging: Fix Windows service start/stop issues
This commit addresses several bugs that prevented the Windows
service from being started or stopped:

- Extra white space in the concatenation of java options in
  elasticsearch.in.bat which tripped up Apache Commons Daemon
  and caused ES to startup without any params, eventually leading
  to the "path.home is not configured" exception.

- service.bat was not passing the start argument to ES

- The service could not be stopped gracefully via the stop command
  because there wasn't a method for procrun to call.

Closes #13247
Closes #13401
2015-09-09 13:31:46 -04:00
Martijn van Groningen c54135c1e1 parent/child: Always store doc values join field.
Now that the pre 2.0 parent child implementation has been removed there is no need to have logic that decides not to store the join doc values field.
2015-09-09 18:46:40 +02:00
javanna 7a2dd7d55c adapted to changes upstream 2015-09-09 17:15:44 +02:00
javanna db705ab460 Merge branch 'master' into feature/query-refactoring
Conflicts:
	core/src/main/java/org/elasticsearch/index/mapper/internal/ParentFieldMapper.java
	core/src/main/java/org/elasticsearch/index/query/HasChildQueryBuilder.java
	core/src/main/java/org/elasticsearch/index/query/HasChildQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/HasParentQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/MoreLikeThisQueryBuilder.java
	core/src/main/java/org/elasticsearch/index/query/MoreLikeThisQueryParser.java
	core/src/main/java/org/elasticsearch/index/search/child/ChildrenQuery.java
	core/src/test/java/org/elasticsearch/index/query/SimpleIndexQueryParserTests.java
	core/src/test/java/org/elasticsearch/index/search/child/AbstractChildTestCase.java
	core/src/test/java/org/elasticsearch/index/search/child/ChildrenConstantScoreQueryTests.java
	core/src/test/java/org/elasticsearch/index/search/child/ChildrenQueryTests.java
	core/src/test/java/org/elasticsearch/search/child/ChildQuerySearchBwcIT.java
	core/src/test/java/org/elasticsearch/search/child/ChildQuerySearchIT.java
	core/src/test/resources/org/elasticsearch/index/query/simple-query-string.json
2015-09-09 17:08:22 +02:00
Colin Goodheart-Smithe 678e1be5cf [TEST] fix geo query failures when no types are defined 2015-09-09 14:24:53 +01:00
Martijn van Groningen 73d84e4797 Merge pull request #13376 from martijnvg/remove_pre2x_slow_parent_child
Removed pre 2.x parent child implementation
2015-09-09 15:20:10 +02:00
David Pilato 5b9d183bf4 Merge remote-tracking branch 'origin/master' 2015-09-09 15:17:48 +02:00
David Pilato 35049a05c3 Allocation: add support for filtering by transport IP address
Allocation filtering by IP only works today using the node host address. But in some cases, you might want to filter using the publish address which could be different.
2015-09-09 15:15:53 +02:00
Robert Muir f5270849fa Merge pull request #13407 from rmuir/ide_security_manager
Enable security manager by default in tests (e.g. IDEs)
2015-09-09 09:12:35 -04:00
Colin Goodheart-Smithe 19792516ba Query Refactoring: Refactor of GeoDistanceRangeQuery
Moving the query building functionality from the parser to the builders
new toQuery() method analogous to other recent query refactorings.

Also this PR removes the check that the index is created before 2.0 for the normalize parameter. The parameter is now always parsed but as a deprecated parameter. We cannot and should not access the index version during parsing.

Relates to #10217

PR goes against the query-refactoring branch
2015-09-09 13:40:24 +01:00
Colin Goodheart-Smithe 4f0ae05da5 Query Refactoring: Refactor of GeohashCellQuery
Moving the query building functionality from the parser to the builders
new toQuery() method analogous to other recent query refactorings.

Relates to #10217

PR goes against the query-refactoring branch
2015-09-09 13:33:33 +01:00
Alex Ksikes c2ccb2157c Refactors TemplateQueryBuilder and Parser
Relates to #10217

This PR is against the query-refactoring branch.

Closes #13253
2015-09-09 14:16:49 +02:00
Boaz Leskes 30ca6d3970 Internal: extract gateway required allocation calculations
and add a basic test

Closes #13391
2015-09-09 13:23:14 +02:00
Martijn van Groningen 2fb2a12c52 Removed pre 2.x parent child implementation 2015-09-09 13:21:40 +02:00
Boaz Leskes 37d4727c0a Gateway: allow overriding the Gateway implementation
Allows mocking the gateway and overriding it from plugins.

Closes #13412
2015-09-09 13:19:25 +02:00
Martijn van Groningen 65e7aba780 inner hits: Protected against specifying a size larger than the total amount of documents in an index.
Closes #13394
2015-09-09 13:13:41 +02:00
Britta Weber 281eac757d [test] make sure result is never null and that flush and get start at same time 2015-09-09 12:41:32 +02:00
Simon Willnauer d2e53e0e0c Split HasParentQueryParser into toQuery and formXContent
This commit splits HasParentQueryParser into toQuery and fromXContent.
This change also deprecates several keys in favor of simplified settings
and adds basic unittests for HasParentQueryParser.

Relates to #10217
2015-09-09 12:06:44 +02:00
Britta Weber fa9696fb8c Merge pull request #13414 from brwe/commit-refresh-order
Engine: refresh before translog commit
2015-09-09 12:06:26 +02:00
Alex Ksikes a45ee273e3 MLT: builder takes a new Item object like its parser
Previously the parser could take any Term Vectors request, but this would be
not the case of the builder which would still use MultiGetRequest.Item. This
introduces a new Item class which is used by both the builder and parser.

Beyond that the rest is mostly cleanups such as:

1) Deprecating the ignoreLike methods, in favor to using unlike.

2) Deprecating and renaming MoreLikeThisBuilder#addItem to addLikeItem.

3) Ordering the methods of MoreLikeThisBuilder more logically.

This change is needed for the upcoming query refactoring of MLT.

Closes #13372
2015-09-09 11:40:27 +02:00
Martijn van Groningen e23d116bc5 parent/child: Split the _parent field mapping's field type into three field types:
1) A shared immutable fieldtype for the _parent field (used for direct access to that field in the dsl). This field type is stored and indexed.
2) A per type field type for the child join field. The field type has doc values enabled if index is created on or post 2.0 and field data type is allowed to be changed.
3) A per type field type for the parent join field. The field type has doc values enabled if index is created on or post 2.0.

This resolves the issue that a mapping is not compatible if parent and child types have different field data loading settings.

Closes #13169
2015-09-09 11:10:43 +02:00
Britta Weber 0ce66b4d70 Engine: refresh before translog commit
When we commit the translog, documents that were in it before cannot be retrieved from
it anymore via get and have to be retrieved from the index instead. But they will only
be visible if between index and get a refresh is called. Therfore we have to call
first refresh and then translog.commit() because otherwise there is a small gap
in which we cannot read from the translog anymore but also not from the index.

closes #13379
2015-09-09 10:37:13 +02:00
javanna 6395349bc0 [TEST] re-enable boost assertions for queries that don't support it
This will be useful to find out whether we forgot to override `setFinalBoost`, otherwise any bost check is simply skipped.
2015-09-09 09:59:42 +02:00
Ryan Ernst b04160d9f5 Add Xlint options and fix some warnings 2015-09-08 23:49:20 -07:00
Ryan Ernst 2d15cab68c Build: Fix compiler warnings
We have a handful of compiler warnings, mostly because of passing an
array to varargs methods. This change fixes these warnings and adds
-Werror so we don't get anymore of these warnings.

Note this does *not* enable deprecation or unchecked type warnings, so
these remain "hidden". We should work towards removing those as well,
but this is a first step.
2015-09-08 22:15:34 -07:00
Jason Tedor 1806c1e0c7 Remove and forbid use of com.google.common.base.Throwables
This commit removes and now forbids all uses of
com.google.common.base.Throwables across the codebase.

For uses of com.google.common.base.Throwables#getStackTraceAsString,
use org.elasticsearch.ExceptionsHelper#stackTrace.

Relates #13224
2015-09-08 21:56:23 -04:00
Ryan Ernst ea4c27a089 Add back path.logs setup in internal settings preparer 2015-09-08 15:29:06 -07:00
Nicholas Knize e4e71d8a9a add points_only option to GeoShapeFieldMapper for optimizing indexing performance on geo_shape indexes designed to store only points. Includes updated documentation and exception handling for ensuring index integrity on points only data. 2015-09-08 16:17:50 -05:00
Ryan Ernst 55795f8ec2 Merge pull request #13383 from rjernst/go_away_transport_paths
Remove environment from transport client
2015-09-08 13:28:21 -07:00
Ryan Ernst 4914b0c465 Merge pull request #13385 from rjernst/wrong_resources
Move resources to their proper location
2015-09-08 13:25:31 -07:00
Ryan Ernst f3fb702e88 Move logging specific settings setup to log configurator. 2015-09-08 13:13:10 -07:00
Ryan Ernst e218b31322 Address PR comments 2015-09-08 12:57:28 -07:00
Robert Muir 29698f6945 Enable security manager by default in tests (e.g. IDEs)
Otherwise people will be confused when they use maven.
2015-09-08 15:55:33 -04:00
Robert Muir 164cf2775f Give intellij its jar hell back. This is not an invitation for more leniency. 2015-09-08 15:32:07 -04:00
javanna 50c43c9328 [TEST] minor adjustments to boost assertions for queries that don't support it 2015-09-08 19:40:31 +02:00
Ryan Ernst 2e63290548 Finish removing loadConfigSettings, and fix node startup to use correct
settings object
2015-09-08 10:16:25 -07:00
Ryan Ernst 92b62c0c6b Merge branch 'master' into go_away_transport_paths 2015-09-08 08:55:17 -07:00
Ryan Ernst d19e04fbb6 Change prepareSettingsAndEnvironment to prepareEnvironment 2015-09-08 08:54:18 -07:00
Robert Muir c354165bc9 remove intellij leniency. fix your IDE instead. it gives people the wrong idea. 2015-09-08 11:32:12 -04:00
Simon Willnauer 18bec264f9 Split HasChildQueryParser into toQuery and formXContent
This is an intial commit that splits HasChildQueryParser / Builder into
the two seperate steps. This one is particularly nasty since it transports
a pretty wild InnerHits object that needs heavy refactoring. Yet, this commit
has still some nocommits and needs more tests and maybe another cleanup but
it's a start to get the code out there.
2015-09-08 17:11:57 +02:00
Britta Weber 3de8cbb54c Merge pull request #13380 from brwe/exception-on-force-merge
Engine: Let AlreadyClosedException and EngineClosedExceptionBubble up
2015-09-08 17:01:10 +02:00
javanna f2605b34d6 Merge branch 'master' into feature/query-refactoring
Conflicts:
	core/src/main/java/org/elasticsearch/index/query/ScriptQueryParser.java
2015-09-08 16:11:22 +02:00
Britta Weber 2288d44eeb Engine: Let AlreadyClosedException and EngineClosedException bubble up
Whe we call optimize we ignore Exceptions that indicate a closed shard.
However, when a shard is closed while an optimize request is in flight it
might also trigger an AlreadyClosedException from the IndexWriter when we
get the config or ForceMergeFailedEngineException with the EngineClosedException
wrapped inside. Because these are not identified as exceptions that indicate
a closed shard (TransportActions.isShardNotAvailableException(..)) optimize
would sometimes report failures when shards were relocating while optimize was called
and sometimes not. This caused weird test failures, see #13266 .
Instead, we should let EngineClosedException bubble up and also recognize
AlreadyClosedException as an indicator for a closed shard.
2015-09-08 16:05:40 +02:00
Simon Willnauer f208aaa47c Also use PriorityComparator in shard balancer
Today we try to allocate primaries first and then replicas
but don't take the index creation date and priority into account
as we do in the GatewayAlloactor.

Closes #13249
2015-09-08 15:22:22 +02:00
Jason Tedor cf539a2158 Add trace logging before sending single shard requests 2015-09-08 08:41:39 -04:00
Simon Willnauer ed5f6e5a0c Remove cyclic dependencies between IndexService and FieldData / BitSet caches
Adds a listeners to each of the caches that allows us to remove the dependency on IndexService which is cyclic since
the IndexService depends on both of these caches. This cyclic dependency makes
testing the indiviual parts very hard and is only added for the sake of
incrementing some stats.
2015-09-08 13:06:32 +02:00
Boaz Leskes a2f94f2e2f move ZenDiscovery.rejoin to protected
So we can override it from tests and plugins.
2015-09-08 11:02:29 +02:00
Ryan Ernst 6b88093f78 Add back filtering for build properties, and remove unused resource
file.
2015-09-08 01:15:46 -07:00
Ryan Ernst 1b99b1d8b4 Remove unnecessary maven resources config 2015-09-07 22:50:01 -07:00
Ryan Ernst bcc4175b53 Build: Move resources to their proper location
We have a a number of resources in the java source tree, mostly for
tests. This change moves them to their proper location.
2015-09-07 19:45:08 -07:00
Ryan Ernst 1ff49eb8de Settings: Remove environment from transport client
Transport clients run embedded within external applications, so
elasticsearch should not be doing anything with the filesystem, as there
is not elasticsearch home.

This change makes a number of cleanups to the internal API for loading
settings and creating an environment. The loadFromConfig option was
removed, since it was always true except for tests. We now always
attempt to load settings from config a file when an environment is
created. The prepare methods were also simplified so there is now
prepareSettingsAndEnvironment which nodes use, and prepareSettings which
the transport client uses. I also attempted to improve the tests, but
there is a still a lot of follow up work to do there.

closes #13155
2015-09-07 17:26:41 -07:00
Adrien Grand 4173b00241 Merge pull request #13310 from jpountz/remove/count_scan
Remove the scan and count search types.
2015-09-07 16:21:14 +02:00
Jason Tedor a6ffe8f6d5 Remove and forbid use of com.google.common.base.Strings
This commit removes and now forbids all uses of
com.google.common.base.Strings across the codebase.

For uses of com.google.common.base.Strings.isNullOrEmpty, use
org.elasticsearch.common.Strings.isNullOrEmpty.

For uses of com.google.common.base.Strings.padStart use
org.elasticsearch.common.Strings.padStart.

For uses of com.google.common.base.Strings.nullToEmpty use
org.elasticsearch.common.Strings.coalesceToEmpty.

Relates #13224
2015-09-07 09:40:14 -04:00
Adrien Grand 0c26e7cd83 Remove the scan and count search types.
These search types have been deprecated in 2.1 and 2.0 respectively, and will
be removed in 3.0.
2015-09-07 15:18:45 +02:00
Britta Weber c5cb559014 Merge pull request #13341 from brwe/shard-not-available-broadcast-replication
fix exception handling for unavailable shards in broadcast replicatio…
2015-09-07 14:47:55 +02:00
Britta Weber 41a2375c9e fix exception handling for unavailable shards in broadcast replication action
Before #13068 refresh and flush ignored all exceptions that matched
TransportActions.isShardNotAvailableException(e) and this should not change.
In addition, refresh and flush which are based on broadcast replication
might now get UnavailableShardsException from TransportReplicationAction if a shard
is unavailable and this is not caught by TransportActions.isShardNotAvailableException(e).
This must be ignored as well.
2015-09-07 14:47:28 +02:00
Britta Weber 972c1cc398 [test] remove timeout test 0s for flush and refresh - we decided we use 1m timeout
see #13238
2015-09-07 14:39:59 +02:00
Jason Tedor 8a3411e5e4 Remove and forbid the use of com.google.common.base.Predicate(s)?
This commit removes and now forbids all uses of
com.google.common.base.Predicate and com.google.common.base.Predicates
across the codebase. This is one of the many steps in the eventual
removal of Guava as a dependency. This was enabled by #13314.

Relates #13224
2015-09-06 07:20:24 -04:00
Jason Tedor 14e4882425 Remove and forbid use of com.google.common.base.Objects
This commit removes and now forbids all uses of
com.google.common.base.Objects across the codebase. This is a small
step in the eventual removal of Guava as a dependency.

Relates #13224
2015-09-05 08:52:30 -04:00
Lee Hinman 701dc340a8 Allow deleting closed indices with shadow replicas
Previously we skip deleting the index store for indices on a shared
filesystem, because we don't want to delete the data when the shard is
relocating around the cluster. This adds a flag to the
`deleteIndexStore` method signifying that the index is closed and that
we should allow deleting the contents even if it is on a shared
filesystem.

Includes a unit test for the IndicesService.canDeleteIndexContents and
integration tests ensure a closed shadow replica index deletes files
correctly.

Resolves #13297
2015-09-04 15:28:18 -06:00
Martijn van Groningen ac951e86d1 fix npe 2015-09-04 22:03:25 +02:00
Zachary Tong 5aa9639034 Add missing License header 2015-09-04 15:50:02 -04:00
Zachary Tong 397d5beae1 Aggregations: Add stats_bucket / extended_stats_bucket pipeline aggregations
These are the complements to the stats/extended_stats metric aggregations, and can be used
to calculate a variety of statistics over buckets
2015-09-04 15:23:48 -04:00
Jason Tedor f4213614c3 Remove use of underscore as an identifier
As a refinement to Project Coin (JEP-213, JDK-8042880), Java 9 is going
to disallow the use of ‘_’ as a one-character identifier. This will be
done by adding ‘_’ as a keyword to the Java language (JDK-8065599).
Currently, uses of ‘_’ as a one-character identifier are warnings in
the Java 8 compiler. This commit removes all uses of ‘_’ as a
one-character identifier from the codebase.
2015-09-04 14:18:30 -04:00
javanna 083c774b5e adapt to upstream changes 2015-09-04 19:56:32 +02:00
javanna be3409f1db Merge branch 'master' into feature/query-refactoring
Conflicts:
	core/src/main/java/org/elasticsearch/index/query/AndQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/BoolQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/ExistsQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/GeoShapeQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/MissingQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/OrQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/QueryParseContext.java
	core/src/main/java/org/elasticsearch/index/query/SimpleQueryStringParser.java
	core/src/main/java/org/elasticsearch/index/query/SpanContainingQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/SpanWithinQueryParser.java
	core/src/main/java/org/elasticsearch/index/query/TermsQueryParser.java
	core/src/main/java/org/elasticsearch/search/sort/GeoDistanceSortParser.java
	core/src/test/java/org/elasticsearch/index/query/SimpleIndexQueryParserTests.java
	core/src/test/java/org/elasticsearch/index/search/child/AbstractChildTestCase.java
2015-09-04 19:21:38 +02:00
Jason Tedor b98cd5f611 Should be asserting JVM plugins do not have a URL 2015-09-04 12:13:54 -04:00
javanna 873d69f157 Query DSL: span_containing and span_within override default boost coming from lucene
SpanContainingQueryParser and SpanWithinQueryParser always set the boost to the parsed lucene query, even if it is the default one. The default boost of the main query though is the boost coming from the inner little query, value that we end up overriding all the time. We should instead set the boost to the main query only if it differs from the default, to mimic lucene's behaviour.

Relates to #13272
Closes #13339
2015-09-04 17:31:49 +02:00
Adrien Grand 37c90b1047 Tests: Add back BitSetFilterCacheTests which was lost on #13308 2015-09-04 17:13:50 +02:00
javanna bd5613708e Query DSL: simple_query_string overrides boost coming from lucene
SimpleQueryStringParser applies whatever boost the query holds, even if the default 1, to the query obtained from parsing of the query string. that might contain its boost, for instance if it resolved to a simple query like term (single term query against a single field). We should rather multiply the existing boost with the boost set to the query, same as we do in query_string

Relates to #13272
Closes #13331
2015-09-04 16:07:38 +02:00
Jason Tedor cb12107123 Fix Javadoc for o.e.c.r.RoutingNodes.UnassignedIterator#remove 2015-09-04 09:08:12 -04:00
Zachary Tong ab2f295c16 Collect and pass unparsed params to buildFactory(), replacing doParse()
doParse() was supposed to allow aggs to perform extra parsing.  Unfortunately, this forced the
parser to carry instance-level state, which would carry-over and "corrupt" any other aggs of the
same type in the same query.

Instead, we are now collecting all unknown params and pasing them as a Map<String, Object>
to buildFactory().  The agg may then parse them and instantiate a factory.  Each param the
agg uses, it should unset from the unusedParams object.

After building the factory, the parser verifies that unusedParams is empty.  If it is not empty,
an exception is raised so the user knows they provided unknown params.

Fixes #13337
2015-09-04 08:35:13 -04:00
Tanguy Leroux 41aa1a7a71 Manually synchronize listeners when iterating on them in InternalClusterInfoService 2015-09-04 13:50:57 +02:00
Adrien Grand 216335abcf Tests: @AwaitsFix on PercentilesBucketIT.testNested. 2015-09-04 10:47:30 +02:00
Adrien Grand 4f5591be8d Fix deprecations introduced by the upgrade to Lucene 5.3
This changes construction of Phrase and Boolean queries to use the builder,
and replaces BitDocIdSetFilter with BitSetProducer for nested and parent/child
queries. I had to remove the ParentIdsFilter for the case when there was a
single parent as it was using the source of BitSets for parents as a regular
Filter, which is not possible anymore now. I don't think this is an issue since
this case rarely occurs, and the alternative logic for when there are several
matching parent ids should not be much worse.
2015-09-04 10:16:01 +02:00
Zachary Tong 1016734b4c Aggregations: Add percentiles_bucket pipeline aggregations
This pipeline will calculate percentiles over a set of sibling buckets.  This is an exact
implementation, meaning it needs to cache a copy of the series in memory and sort it to determine
the percentiles.

This comes with a few limitations: to prevent serializing data around, only the requested percentiles
are calculated (unlike the TDigest version, which allows the java API to ask for any percentile).
It also needs to store the data in-memory, resulting in some overhead if the requested series is
very large.
2015-09-03 22:24:14 -04:00
Nicholas Knize 17460ae92d Refactor ignore_malformed and coerce from geo_point field type to mapper
This commit moves ignore_malformed and coerce options from the GeoPointFieldType to the Builder in GeoPointFieldMapper. This makes these options consistent with other types in 2.0.
2015-09-03 17:02:51 -05:00
David Pilato b2c584b21d Remove jmeter files
Follow up for #13317.
2015-09-03 22:36:50 +02:00
David Pilato 6319f383c4 Remove assembly files in core
Closes #13317.
2015-09-03 22:36:50 +02:00
Robert Muir 529ad7fe79 Remove broadcast address check.
This was supposed to just help the user, in case they misconfigured something.
Broadcast is an ipv4 only thing, the only way you can really detect its a broadcast
address, is to look and see if an interface has that address as its broadcast address.

But we cannot trust that container interfaces won't have a crazy setup...

Closes #13327
2015-09-03 16:01:14 -04:00
Robert Muir 3e9daa1040 Merge pull request #13324 from rmuir/lucene_1701068
Upgrade master to lucene 5.4-snapshot r1701068
2015-09-03 15:28:52 -04:00
javanna ea83007c7e [TEST] adapt FilteredQueryBuilderTests to upstream changes 2015-09-03 21:18:24 +02:00
Robert Muir f216d92d19 Upgrade to lucene 5.4-snapshot r1701068 2015-09-03 15:13:33 -04:00
javanna 8bd7cdbdad Merge branch 'master' into feature/query-refactoring
Conflicts:
	core/src/main/java/org/elasticsearch/index/query/FilteredQueryParser.java
2015-09-03 20:56:06 +02:00
javanna da554fc30c Query DSL: match_phrase_prefix to take boost into account
The match_phrase_prefix query properly parses the boost etc. but it loses it in its rewrite method. Fixed that by setting the orginal boost to the rewritten query before returning it. Also cleaned up some warning in MultiPhrasePrefixQuery.

Closes #13129
Closes #13142
2015-09-03 19:46:42 +02:00
Jason Tedor 5c4b864a42 Workaround pitfall in Java 8 target-type inference
Target-type inference has been improved in Java 8. This leads to these
lines now being interpreted as invoking String#valueOf(char[]) whereas
they previously were interpreted as invoking String#valueOf(Object).
This change leads to ClassCastExceptions during test execution. Simply
casting the parameter to Object restores the old invocation.

Closes #13315
2015-09-03 13:10:01 -04:00
Robert Muir 11314336b6 bump master (3.0-snapshot) to java 8
Closes #13314

Squashed commit of the following:

commit 0e2d6c8e55e07957d0bea08f3c80f51f481ba6d4
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Sep 3 12:49:51 2015 -0400

    re-enable the rest of this test, as its one method that hangs

commit 04dc6b62d4800f574babcdbbf0e44f5c3400e0f0
Merge: 2b12805 52945b2
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Sep 3 12:48:55 2015 -0400

    Merge branch 'master' into java8

commit 2b128056ded741cddaf1d92c55920d7baf80dc95
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Sep 3 12:28:07 2015 -0400

    more jsr166e removal

commit 5826feba484389a87dbcad9a53ee5bf9a306eb61
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Sep 3 12:15:13 2015 -0400

    disable broken test

commit aabd88b85f2dd7b9b2b9308ffd52bc71aef8ab82
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Sep 3 12:11:12 2015 -0400

    Disable broken test

commit 863463303a12415dfecd6a46a6767eb07a2c7415
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Sep 3 11:38:32 2015 -0400

    bump master (3.0-snapshot) to java 8
2015-09-03 12:55:21 -04:00
Britta Weber 52945b2b46 [test] mute testReadOnPostRecoveryShards 2015-09-03 18:27:09 +02:00
javanna 6e00a613ee Internal: simplify filtered query conversion to lucene query
We have some optimization in FilteredQueryParser that tries to mimic what the rewrite method in lucene does, based on what gets parsed we return the simplest query possible. That might cause issues with boost values though, if specified in both the main query and the inner query that we shortcut to. We should rather rely on lucene's rewrite method to simplify the lucene representation of the query, and always build a filtered query instead.

relates to #13272
Closes #13312
2015-09-03 17:13:00 +02:00
Lee Hinman ffb2f75305 Remove DisableAllocationDecider
The `EnableAllocatiorDecider` has replaced the
`DisableAllocationDecider`, which has been deprecated. It can now be
removed.
2015-09-03 09:10:48 -06:00
Adrien Grand 745b977ce7 Optimize scrolls for constant-score queries.
We currently optimize scroll when sort=_doc because docs are returned in order.
But documents are also returned in order when sorting by score and the query
gives constant scores. This optimization has the nice side-effect of also
optimizing scrolls with the default `match_all` query.
2015-09-03 16:12:04 +02:00
Robert Muir 4b0182932a Merge pull request #13299 from rmuir/network_config_improvements
Improve situation when network.host is set to wildcard (e.g. 0.0.0.0)
2015-09-03 08:18:46 -04:00
javanna 107a4b14e0 Merge branch 'master' into feature/query-refactoring 2015-09-03 14:09:39 +02:00
javanna 4559db9cd9 [TEST] renamed tests according to updated naming conventions (Tests suffix enforced) 2015-09-03 13:30:07 +02:00
javanna fa93cc7b89 Merge branch 'master' into feature/query-refactoring
Conflicts:
	core/src/test/java/org/elasticsearch/ExceptionSerializationTests.java
2015-09-03 13:02:49 +02:00
David Pilato ac9682dd00 [test] cloud-aws has been replaced by repository-s3 and discovery-ec2 2015-09-03 12:52:02 +02:00
David Pilato a117fc85a6 Update help install for discovery-ec2 and repository-s3 2015-09-03 11:29:45 +02:00
David Pilato 2931687b46 Merge remote-tracking branch 'origin/master' 2015-09-03 11:13:19 +02:00
David Pilato 30aa231f8e [plugin] split cloud-aws in repository-s3 and discovery-ec2
Until now we had a cloud-aws plugin which is providing 2 disctinct features:

* discovery on EC2
* snapshot/restore on S3

This commit splits the plugin by feature so people can use either one or the other or both features.

Doc is updated accordingly.
2015-09-03 11:12:20 +02:00
Colin Goodheart-Smithe ddd064ce1f [TESTS] Changed message in NamingConventionTests to reflect code checks
Before this change the check would check that all test classes end in Tests but the message would say they need to end in Test or Tests which was confusing.
2015-09-03 10:08:25 +01:00
Simon Willnauer 796701d52e Move version to 3.0.0-SNAPSHOT 2015-09-03 10:43:28 +02:00
Adrien Grand 5ff9ca9965 Optimize counts on simple queries.
Today we always collect in order to compute counts, but some of them can be
easily optimized by using pre-computed index statistics. This is especially
true in the case that there are no deletions, which should be common for the
time-based data use-case.

Counts on match_all queries can always be optimized, so requests like

```
GET index/_search?size=0

GET index/_search
{
  "size": 0,
  "query" : {
    "match_all": {}
  }
}
```

should now return almost instantly. Additionally, when there are no deletions,
term queries are also optimized, so the below queries which all boil down to a
single term query would also return almost immediately:

```
GET index/type/_search?size=0

GET index/_search
{
  "size": 0,
  "query" : {
    "match": {
      "foo": "bar"
    }
  }
}

GET index/_search
{
  "size": 0,
  "query" : {
    "constant_score": {
      "filter": {
        "exists": {
          "field": "foo"
        }
      }
    }
  }
}
```
2015-09-03 09:36:51 +02:00
Adrien Grand d85cbd9f37 Merge pull request #13295 from sksamuel/patch-1
Updated param name to match type
2015-09-03 09:23:20 +02:00
Robert Muir 6bdf89c47e Improve situation when network.host is set to wildcard (e.g. 0.0.0.0)
Users might specify something like -Des.network.host=0.0.0.0, as that
was the old default with previous versions of elasticsearch. This means
to bind to all interfaces, but it makes no sense as a publish address.

Pick a good one in this case, just like we do in other cases where
publish isn't explicitly specified and we are bound to multiple (e.g.
when configured by interface, or dns hostname with multiple addresses).
However, in this case warn the user about it: since its arbitrarily
picking the first non-loopback address like the old versions
did, thats a little too heuristical, but lets make the cutover easy.

Separately, fail hard if things like multicast or broadcast addresses are
configured as bind or publish addresses, as that is simply invalid.

Closes #13274
2015-09-02 23:03:31 -04:00
Igor Motov e288c9b0ce Tests: set the upper bound on the number of simulated failures in a restore test
The number and distribution of errors in some restore test may cause restore process to continue to fail for a prolong time. This test caps the total number of simulated failures to make sure that restore is guaranteed to eventually succeed after a limited number of retries.
2015-09-02 19:13:45 -04:00