Commit Graph

5281 Commits

Author SHA1 Message Date
Lee Hinman c86fdecd25 [TESTS] Be less strict about breaker child limit
Failing a parent breaker check is eventually consistent, so the test
could fail the parent limit, throw an exception, and before being
adjusted back down, increment more and throw a circuit breaking
exception on the child. This increases the child's limit, to ensure
we're only testing the parent limit.

It adds an additional assert to ensure that the breaker total is
correctly re-adjusted when the parent breaker has been tripped.
2014-09-30 13:01:27 +02:00
Michael McCandless 4e3f3e7ef8 1.3.4 release: add 1.3.5 Version constant 2014-09-30 06:44:19 -04:00
Michael McCandless 0be4c6a73d Core: go back to unbounded (scaling) thread pool for management threads (revert #7318) 2014-09-30 04:54:28 -04:00
Britta Weber e99be5cb0b [TEST] Mute MoreLikeThisActionTests#*ArtificialDocs 2014-09-30 09:29:32 +02:00
Ryan Ernst 37b294aaec Fix optimize behavior with 'force' and 'flush' flags.
This does the following:
* Make 'force' flag only build a merge if the delegate MP returned no merges
* Add async handling for 'flush' when 'waitForMerges' is false
* Remove flush at the beginning of optimize.  This is something the user can
  do if they wish, before calling optimize.

closes #7886
closes #7904
closes #7920
2014-09-29 15:20:19 -07:00
Suyog Rao 25bce1db5d Nest original exception while creating NoShardAvailableActionException
Closes #7756
2014-09-29 14:10:16 -07:00
Simon Willnauer 20a0c68964 [BUILD] Release version should match latest version
This commit ensures that the latest version in our code is identical
to the project.version specified in the pom.xml file.
2014-09-29 17:45:10 +02:00
Simon Willnauer cfd9ac2f63 [TEST] Use Shutdown API only if nodes are on 1.3.3 or newer to prevent shutdown problems 2014-09-29 17:18:26 +02:00
Michael McCandless aa89c481b0 1.3.3 release: add 1.3.4 version constant 2014-09-29 10:29:18 -04:00
javanna c06b772df0 [TEST] make sure that IndicesRequestTests is repeateable using the same seed
Remove the creation of a node client if not there before each test through setup method. `numClientNodes` makes sure that the client node gets created during suite cluster initialization.
2014-09-29 15:57:14 +02:00
Alex Ksikes b118558962 MLT Query: Support for artificial documents
Previously, the only way to specify a document not present in the index was to
use `like_text`. This would usually lead to complex queries made of multiple
MLT queries per document field. This commit adds the ability to the MLT query
to directly specify documents not present in the index (artificial documents).
The syntax is similar to the Percolator API or to the Multi Term Vector API.

Closes #7725
2014-09-29 15:49:13 +02:00
javanna 43a1e1c353 [TEST] create client nodes using node.client: true instead node.data: false and node.master: false
Create client nodes using `node.client: true` instead of `node.data: false` and `node.master: false`.

We should create client nodes in our test infra using the `node.client:true` settings as that is the one that users use, and the one that we use as well in `ClientNodePredicate` thus we end up not finding client nodes otherwise as they weren't created with the proper setting.

Updated also the `DataNodePredicate` so that `client: true` is enough, no need for `data: false` as well.

Closes #7911
2014-09-29 15:24:17 +02:00
Lee Hinman ab9cc336e5 [TESTS] Additional logging for `testThreadedUpdatesToChildBreakerWithParentLimit` 2014-09-29 15:06:36 +02:00
Boaz Leskes 9b4bf4379a Test: testNodeNotReachableFromMaster had a typo when choosing a non master node 2014-09-29 11:38:39 +02:00
Alex Ksikes 5014158d6b MLT Query: use minimum should match more extensive syntax
The minimum number of optional should clauses of the generated query to match
can now be set using the more extensive minimum should match syntax. This
makes the `percent_terms_to_match` parameter deprecated, and replaced in favor
to a new `minimum_should_match` parameter.

Closes #7898
2014-09-29 11:14:56 +02:00
Boaz Leskes 03d880de38 Discovery: master fault detection fall back to cluster state thread upon error
With #7834, we simplified ZenDiscovery by making it use the current cluster state for all it's decision. This had the side effect a node may start it's Master FD before the master  has fully processed that cluster state update that adds that node (or elects the master master). This is due to the fact that master FD is started when a node receives a cluster state from the master but the master it self may still be publishing to other node.

This commit makes sure that a master FD ping is only failed once we know that there is no current cluster state update in progress.

Closes #7908
2014-09-29 11:12:11 +02:00
Lee Hinman 168b3752ef Refactor the Translog.read(Location) method
It was only used by `readSource`, it has been changed to return a
Translog.Operation, which can have .getSource() called on it to return
the source. `readSource` has been removed.

This also removes the checked IOException, any exception thrown is
unexpected and should throw a runtime exception.

Moves the ReleasableBytesStreamOutput allocation into the body of the
try-catch block so the lock can be released in the event of an exception
during allocation.
2014-09-29 10:13:45 +02:00
mikemccand 6bf635039c Core: upgrade to Lucene 4.10.1 2014-09-28 13:42:12 -04:00
mikemccand 9e8c51b70d fix concurrency bug in index throttling 2014-09-28 12:30:48 -04:00
Boaz Leskes b70f0d5eef Internal: MulticastChannel should wait on receiver thread to stop during shutdown
This was signaled by our tests which shutdown class and check for thread leakage.

Closes #7835
2014-09-27 14:23:07 +02:00
Martijn van Groningen 71adb3ada2 If a node is being shutdown some in flight ping request may be executed. Make sure to keep track of those ping requests and close the unicast connect executor service.
Closes #7903
2014-09-27 00:05:15 +02:00
javanna e85e07941d Internal: split internal fetch request used within scroll and search
Similar to #7856 but relates to the fetch shard level requests. We currently use the same internal request when we need to fetch within search and scroll. The two original requests though diverged after #6933 as SearchRequest implements IndicesRequest while SearchScrollRequest doesn't. That said, with #7319 we made `FetchSearchRequest` implement IndicesRequest by making it hold the original indices taken from the original request, which are null if the fetch was originated by a search scroll, and that is why original indices are optional there.

This commit introduces a separate fetch request and transport action for scroll, which doesn't hold original indices. The new action is only used against nodes that expose it, the previous action name will be used for nodes older than 1.4.0.Beta1.

As a result, in 1.4 we have a new `indices:data/read/search[phase/fetch/id/scroll]` action that is equivalent to the previous `indices:data/read/search[phase/fetch/id]` whose request implements now IndicesRequest and holds the original indices coming from the original request. The original indices in the latter request can only be null during a rolling upgrade (already existing version checks make sure that serialization is bw compatible), when some nodes are still < 1.4.

Closes #7870
2014-09-26 18:24:53 +02:00
Britta Weber bac1da25f6 node shutdown: make close() syncronized
An example scenario where this will help:

When the node is shutdown via api call
(https://github.com/elasticsearch/elasticsearch/blob/master/src/test/java/org/elasticsearch/test/ExternalNode.java#L219 )
then the call returns immediately even if the node is not actually shutdown yet
(https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/action/admin/cluster/node/shutdown/TransportNodesShutdownAction.java#L226).
If at the same time the proces is killed, then the hook that would usually prevent
uncontrolled shutdown
(https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/bootstrap/Bootstrap.java#L75)
has no effect: It again calls close() which might then just return
for example because one of the lifecycles was moved to closed already.

The bwc test FunctionScoreBackwardCompatibilityTests.testSimpleFunctionScoreParsingWorks
failed because of this. The translog was not properly
written because if the shutdown was called via api, the following process.destroy()
(https://github.com/elasticsearch/elasticsearch/blob/master/src/test/java/org/elasticsearch/test/ExternalNode.java#L225)
killed the node before the translog was written to disk.

closes #7885
2014-09-26 12:46:18 +02:00
Boaz Leskes 36c3e896de NodesFD: simplify concurrency control to fully rely on a single map
The node fault detection class is used by the master node to ping the nodes in the cluster and verify they are alive. This PR simplifies the concurrency controls in the class + adds a test for a scenario that surfaced the problem.

Closes #7889
2014-09-26 11:21:55 +02:00
Boaz Leskes db54e9c2d5 Discovery: remove any local state and use clusterService.state instead
At the moment, ZenDiscovery contains a local copy of the disco nodes plus a flag that indicates whether the local node is master or not. This is redundant as the same information is stored in the cluster state. Have duplicate copy can lead to unneeded concurrency issues. This PR removes the duplication, including moving the ownership of the localNode creation to ClusterState

The PR introduces a tighter control of the background joining thread to make sure it is started and stopped together with any cluster state changes. This solves potentially concurrency bugs where a joining thread may fail to start.

Last we add a couple of safety checks to make sure that if a nodes receives a cluster state from a new master while actively trying to join another one (or electing itself) we go back to pinging to actively join it.

Closes #7834
2014-09-26 11:21:55 +02:00
Britta Weber eb9d39f611 [TEST] wait for yellow else assertSearchResponse will trip 2014-09-26 11:13:12 +02:00
Britta Weber 75d2a84772 [TEST] wait for yellow else assertSearchResponse will trip 2014-09-26 10:52:44 +02:00
Michael McCandless e207189037 Tests: turn off CheckIndex for now (it's buggy: there is a race w/ deletion of all files in the data dirs) 2014-09-26 04:44:11 -04:00
Michael McCandless 87e9aba2ac disable CheckIndex for these no-ack tests 2014-09-26 04:08:03 -04:00
Britta Weber 526b464025 field name lookup: return List instead of Set for names matching a pattern
The returned sets are only used for iterating. Therefore we might
as well return a list since this guaratees order.

This is the same effect as in
https://github.com/elasticsearch/elasticsearch/pull/7698
The test SimpleIndexQueryParserTests#testQueryStringFieldsMatch
failed on openjdk 1.7.0_65 with
<jdk.map.althashing.threshold>0</jdk.map.althashing.threshold>

closes #7709
2014-09-26 09:59:12 +02:00
Britta Weber 7feb742a9b script with _score: remove dependency of DocLookup and scorer
As pointed out in #7487 DocLookup is a variable that is accessible by all scripts
for one doc while the query is executed. But the _score and therfore the scorer
depends on the current context, that is, which part of query is currently executed.
Instead of setting the scorer for DocLookup
and have Script access the DocLookup for getting the score, the Scorer should just
be explicitely set for each script.
DocLookup should not have any reference to a scorer.
This was similarly discussed in #7043.

This dependency caused a stackoverflow when running script score in combination with an
aggregation on _score. Also the wrong scorer was called when nesting several script scores.

closes #7487
closes #7819
2014-09-26 09:59:12 +02:00
Igor Motov 9c9cd01854 Fix NumberFormatException in Simple Query String Query
Incorrect usage of XContentParser.hasTextCharacters() can result in NumberFormatException as well as other possible issues in template query parser and phrase suggest parsers.

Fixes #7875
2014-09-26 10:49:05 +04:00
Michael McCandless 3db50b2ebf don't CheckIndex for this test case 2014-09-25 18:21:12 -04:00
Michael McCandless 637c6d1606 Tests: always run Lucene's CheckIndex when shards are closed in tests and fail the test if corruption is detected
Today we only run 10% of the time, and the test doesn't fail when
corruption is detected.

I think it's better to always run and fail the test, so we can catch
any possible resiliency bugs in Lucene/Elasticsearch causing corruption.

For known tests that create corrupted indices, it's easy to set
MockFSDirectoryService.CHECK_INDEX_ON_CLOSE to false...

Closes #7730
2014-09-25 16:50:48 -04:00
markharwood e97b8fd217 Aggs - support for arrays of numeric values in include/exclude clauses
Closes #7714
2014-09-25 11:02:29 +01:00
Simon Willnauer a90d7b1670 [TRANSPORT] never send requests after transport service is stopped
With local transport or any transport that doesn't necessarily send
notification if connections are closed we might miss a node
disconnection and the request handler hangs forever / until the timeout
kicks in. This window only exists during shutdown and is likely
unproblematic in practice but tests might run into this problem when
local transport is used.
2014-09-25 11:51:06 +02:00
Shay Banon a82d486bda Add a listener thread pool
Today, when executing an action (mainly when using the Java API), a listener threaded flag can be set to true in order to execute the listener on a different thread pool. Today, this thread pool is the generic thread pool, which is cached. This can create problems for Java clients (mainly) around potential thread explosion.
Introduce a new thread pool called listener, that is fixed sized and defaults to the half the cores maxed at 10, and use it where listeners are executed.
relates to #5152
closes #7837
2014-09-25 11:25:13 +02:00
Simon Willnauer 4bd37d7ee6 [TEST] Reenable threadleak filters with 5 sec. lingering 2014-09-25 10:48:13 +02:00
Simon Willnauer a236b80392 [CORE] Add ThreadPool.terminate to streamline shutdown
Shutting down threadpools and executor services is done in very similar
fashion across the codebase. This commit streamlines the process by
adding a terminate method to ThreadPool.
2014-09-25 10:48:12 +02:00
Alex Ksikes 51bf3e6730 MLT Query: fix percent_terms_to_match
The parameter `percent_terms_to_match` (percentage of terms that must match in
the generated query) was wrongly set to the top level boolean query. This
would lead to zero or all results type of situations. This commit ensures that
the parameter is indeed applied to the query of generated terms.

Closes #7754
2014-09-25 09:56:53 +02:00
Michael McCandless 5e9e2cf50c Core: try again to upgrade to Lucene 4.10.1-snapshot 2014-09-24 13:48:49 -04:00
Michael McCandless ab3be76644 Revert Lucene upgrade 2014-09-24 13:25:55 -04:00
Michael McCandless 15c75b1967 Core: upgrade to Lucene 4.10.1 snapshot
Lucene will soon release official 4.10.1, but by upgrading sooner we can 1) sidestep the false failures due to the 1.8.0_20 JVM hotspot bug (has caused a number of false failures in recent Jenkins tests), 2) make sure none of the Lucene changes in 4.10.1 are problematic.

Closes #7844
2014-09-24 13:13:07 -04:00
Boaz Leskes 0f121ff351 Test: ClusterServiceTests.testLocalNodeMasterListenerCallbacks - increase ping timeout
was 200ms, now 400ms
2014-09-24 17:26:40 +02:00
javanna 17b1fd1a6a Internal: split internal free context request used after scroll and search
We currently use the same internal request when we need to free the search context after a search and a scroll. The two original requests though diverged after #6933 as `SearchRequest` implements `IndicesRequest` while `SearchScrollRequest` and `ClearScrollRequest` don't. That said, with #7319 we made `SearchFreeContextRequest` implement `IndicesRequest` by making it hold the original indices taken from the original request, which are null if the free context was originated by a scroll or by a clear scroll call, and that is why original indices are optional there.

This commit introduces a separate free context request and transport action for scroll, which doesn't hold original indices. The new action is only used against nodes that expose it, the previous action name will be used for nodes older than 1.4.0.Beta1.

As a result, in 1.4 we have a new `indices:data/read/search[free_context/scroll]` action that is equivalent to the previous `indices:data/read/search[free_context]` whose request implements now `IndicesRequest` and holds the original indices coming from the original request. The original indices in the latter requests can only be null during a rolling upgrade (already existing version checks make sure that serialization is bw compatible), when some nodes are still < 1.4.

Closes #7856
2014-09-24 15:33:27 +02:00
Simon Willnauer ea49a3e269 Update version flags after backporting to 1.3.3
Relates to #7857
2014-09-24 15:23:57 +02:00
Colin Goodheart-Smithe f37815a53b Aggregations: Significant Terms Heuristics now registered correctly
Closes #7840
2014-09-24 11:48:26 +01:00
Simon Willnauer ca86e1c824 [TEST] Disable thread filter for now 2014-09-23 14:01:22 +02:00
Simon Willnauer 45319dc27f [TEST] use more terminate calls and wait for termination 2014-09-23 13:58:47 +02:00
Simon Willnauer 30acba624d [TEST] Add a more restrictive thread leaks filter
Today all threads are allowed to leak a suite. This is tricky since
it essentially allows resource leaks by default where for instance
test private TransportClients will never get closed and consume
resources influencing other tests. It also hides threads that
are not fully under elasticsearchs control like the Lucene
TimeLimitingCollector thread. This commit restricts the threads
that can leak a suite to the threads spawned from testclusters
and fixes sevearl places that leaked threads.

Closes #7833
2014-09-23 13:36:21 +02:00
Simon Willnauer 18212ba09c [TEST] Make sure test actually throttles 2014-09-23 12:53:47 +02:00
Simon Willnauer 68c5206e50 [TEST] disable translog based flushes when corrupting files
These tests rely on the fact that all files stay the same after
the corruption and if we run into a translog based flush we might
use a new / different delete file causing the test to fail.
2014-09-23 12:30:06 +02:00
Shay Banon d4d77cdb66 Chunk direct buffer usage by networking layer
Today, due to how netty works (both on http layer and transport layer), and even though the buffers sent over to netty are paged (CompositeChannelBuffer), it ends up re-copying the whole buffer into another heap buffer (bad), and then send it over directly to sun.nio which allocates a full thread local direct buffer to send it (which can be repeated if not all message is sent).
  This is problematic for very large messages, aside from the extra heap temporal usage, the large direct buffers will stay around and not released by the JVM.
  This change forces the use of gathering when building a CompositeChannelBuffer, which results in netty using the sun.nio write method that accepts an array of ByteBuffer (so no extra heap copying), and also reduces the amount of direct memory allocated for large messages.
  See the doc on NettyUtils#DEFAULT_GATHERING for more info.
closes #7811
2014-09-23 12:15:19 +02:00
Simon Willnauer 6c8aa5fa6c [RECOVERY] Mark last file chunk to fail fast if payload is truncated
Today we rely on the metadata length of the file we are recoverying
to indicate when the last chunk was received. Yet, this might hide bugs
on the compression layer if payloads are truncated. We should indicate
if the last chunk is send to make sure we validate checksums
accordingly if possible.

Closes #7830
2014-09-23 11:32:18 +02:00
Simon Willnauer 5533495171 [TEST] Ensure primaries are allocated before bulk indexing with dymamic mappings 2014-09-23 08:34:25 +02:00
Shay Banon f6a0fe5c2f Apply bulk change to 1.3.3
relates to #7729
2014-09-22 15:13:35 +02:00
Boaz Leskes b1851906d8 Tests: extend testRecoverFromPreviousVersion to sometimes index during relocation
Relates to #7729

Closes #7768
2014-09-22 11:19:38 +02:00
Boaz Leskes 4677d05048 Recovery: mapping check during phase2 should be done in cluster state update task
Before phase2 we check verify that the local mapping is in sync with the cluster state mapping (and send & wait on a master update mapping task if not). This check should be done under a cluster state update task to make sure an incoming cluster state update to do not change things while we check.

Closes #7744
2014-09-22 11:05:00 +02:00
Boaz Leskes d17fd26f23 Test: RecoveryWhileUnderLoadTests.recoverWhileRelocating should report cluster state when failing to reach green 2014-09-21 20:16:45 +02:00
Boaz Leskes 41fd5d02f4 Discovery: Give a unique id to each ping response
During discovery a node gossips with other nodes to discover the current state of the cluster - what nodes are out there, what version they use and most importantly whether there is an active master out there. During this ping process we may end up in a situation where old information is mixed with new. This is comment if a couple of master election happen in rapid succession.

This commit adds a monotonically increasing id to each ping response. This makes it easy to always select the last ping from every node.

Closes #7769
2014-09-20 12:58:15 +02:00
Martijn van Groningen afcbffbfc1 Core: Check if from + size don't cause overflow and fail with a better error.
Closes #7778
2014-09-20 12:34:48 +02:00
mikemccand dbe4e6e674 Internal: remove ForceSyncDirectory
Historical code, not used anymore.

Closes #7804
2014-09-19 14:15:06 -04:00
Brian Murphy 8e742c2096 Indexed Scripts/Templates : Cleanup
This contains several cleanups to the indexed scripts.
Remove the unused FetchSourceContext from the Get request..
Add lang,_version,_id to the REST GET API.
Removes the routing from GetIndexedScriptRequest since the script index is a single shard that is replicated across all nodes.
Fix backward compatible template file reference
Before 1.3.0 on disk scripts could be referenced by requesting
````
_search/template

{
  "template" : "ondiskscript"
}
````
This was broken in 1.3.0 by requiring
````
{
  "template" :
  {
    "file" : "ondiskscript"
  }
}
````
This commit restores the previous behavior.
Remove support for preference, realtime and refresh
These parameters don't make sense anymore for indexed scripts as we always force the preference to _local and
always refresh after a Put to the indexed scripts index.

Closes #7568
Closes #7559
Closes #7647
Closes #7567
2014-09-19 11:59:08 +01:00
Lee Hinman 4185566e93 Add option to take currently relocating shards' sizes into account
When using the DiskThresholdDecider, it's possible that shards could
already be marked as relocating to the node being evaluated. This commit
adds a new setting `cluster.routing.allocation.disk.include_relocations`
which adds the size of the shards currently being relocated to this node
to the node's used disk space.

This new option defaults to `true`, however it's possible to
over-estimate the usage for a node if the relocation is already
partially complete, for instance:

A node with a 10gb shard that's 45% of the way through a relocation
would add 10gb + (.45 * 10) = 14.5gb to the node's disk usage before
examining the watermarks to see if a new shard can be allocated.

Fixes #7753
Relates to #6168
2014-09-19 12:36:51 +02:00
Brian Murphy 61c21f9a0e Bulk API: Do not fail whole request on closed index
The bulk API request was marked as completely failed,
in case a request with a closed index was referred in
any of the requests inside of a bulk one.

Implementation Note: Currently the implementation is a bit more verbose in order to prevent an instanceof check and another cast - if that is fast enough, we could execute that logic only once at the
beginning of the loop (thinking this might be a bit overoptimization here).

Closes #6410
2014-09-19 10:55:49 +01:00
Brian Murphy 4f791b06db Revert "Bulk API: Do not fail whole request on closed index"
This reverts commit 405e5816b8.
2014-09-19 10:27:28 +01:00
Brian Murphy c7c61bfd91 Revert "Bulk Request : Add Document Request"
This reverts commit 86f575dcea.
2014-09-19 10:27:16 +01:00
Brian Murphy 86f575dcea Bulk Request : Add Document Request
This file was missing.
2014-09-19 10:05:45 +01:00
Brian Murphy 405e5816b8 Bulk API: Do not fail whole request on closed index
The bulk API request was marked as completely failed,
in case a request with a closed index was referred in
any of the requests inside of a bulk one.

Implementation Note: Currently the implementation is a bit more verbose in order to prevent an instanceof check and another cast - if that is fast enough, we could execute that logic only once at the beginning of the loop (thinking this might be a bit overoptimization here).

Closes #6410
2014-09-19 09:56:49 +01:00
javanna 4fa924494d [TEST] move REST tests to their own test group
Closes #7795
2014-09-19 10:34:30 +02:00
Simon Willnauer 9f6d6d540b [ENGINE] try increment store before searcher is acquired
InternalEngine#refreshNeeded must increment the ref count on the
store used before it's checking if the searcher is current since
internally a searcher ref is acquired and if that happens concurrently
to a engine close it might violate the assumption that all files
are closed when the store is closed.

This commit also converts some try / finally into try / with.
2014-09-19 00:37:30 +02:00
javanna 508ff29e0d [TEST] allow to fully disable REST tests included parsing via -Dtests.rest=false
We currently look for REST tests on file system although they are disabled. We should not do that and move the check earlier on. This way third parties using our test infra, which don't have REST tests on file system, can effectively disable the REST tests, otherwise they would get initialization error despite having disabled them.  The downside is that the number of tests visualized is going to be zero instead of the real number of parsed REST tests, but there is nothing we can do about this. Tests get ignored anyways.
2014-09-18 17:05:34 +02:00
Colin Goodheart-Smithe 66417a93a0 Aggregations: Removes isSingleUserCriteria check
This change removes the backwards compatibility workaround that checks that a compoundOrder originated from a single user defined criteria for the purposes of serialising to older versioned nodes.
2014-09-18 15:22:43 +01:00
javanna 5e1f95ca93 [TEST] close REST test execution context only if not null
The context can be null when REST tests are disabled via sysprop.
2014-09-18 16:14:59 +02:00
Simon Willnauer d3e348ef90 [CORE] Add AbstractRunnable support to ThreadPool to simplify async operation on bounded threadpools
today we have to catch rejected operation exceptions in various places
and notify an ActionListener. This pattern is error prone and adds a lot
of boilerplait code. It's also easy to miss catching this exception
which only is relevant if nodes are under high load. This commit adds
infrastructure that makes ActionListener first class citizen on async
actions.

Closes #7765
2014-09-18 15:25:10 +02:00
Simon Willnauer b2477a43c8 [TEST] Reimplement AckTests#testDeleteWarmerNoAcknowledgement
This test was not testing what it was supposed to test. This commit
implements the test as an actual delete warmer test without ack
returned.
2014-09-18 15:21:02 +02:00
javanna 5f97bccb54 Internal: add indices setter to IndicesRequest interface
We currently expose generic getters for indices and indicesOptions on the IndicesRequest interface. This commit adds a generic setter as well, which can be used to set the indices to a request. The setter impl throws `UnsupportedOperationException` if called on internal requests. Also throws exception if called on single index operations, since it accepts an array as argument.

Closes #7734
2014-09-18 14:26:11 +02:00
javanna 6717de9e46 Internal: make sure that update internal requests share the same original headers and request context
Update request internally executes index and delete operations. We need to make sure that those internal operations hold the same headers and context as the original update request. Achieved via copy constructors that accept the current request and the original request.

Closes #7766
2014-09-18 14:00:51 +02:00
javanna b9b5842acc Internal: make sure that all delete mapping internal requests share the same original headers and context
Delete mapping executes flush, delete by query and refresh operations internally. Those internal requests are now initialized by passing in the original delete mapping request so that its headers and request context are kept around.

Closes #7736
2014-09-18 13:58:03 +02:00
Martijn van Groningen f43a8e2961 Aggregations: Fix regression bug for the support of terms aggregation on the `_parent` field. 2014-09-18 12:27:55 +02:00
Simon Willnauer 66421c5a83 [TEST] Only reset test cluster if a test actually failed
Previously we resetted the test cluster for all subsequent tests
even though they didn't fail. This make suites like REST tests faster
and prevents crazy timeouts.

Closes #7775
2014-09-18 10:40:30 +02:00
Simon Willnauer dd97a95b04 [TEST] Wait until warmer is registered when testing timeout 2014-09-18 09:15:17 +02:00
Simon Willnauer 19c969a800 [SNAPSHOT] Minor code cleanup 2014-09-17 19:17:40 +02:00
Simon Willnauer 63eb49d202 [TEST] adjust chunk size to create less but bigger files to trigger throtteling more reliably 2014-09-17 19:17:40 +02:00
Martijn van Groningen 94ecf59e65 Test: increase zen logging 2014-09-17 19:06:24 +02:00
Simon Willnauer 2be018db84 [TEST] Only close GLOBAL_CLUSTER if it's non-null 2014-09-17 14:53:58 +02:00
Colin Goodheart-Smithe 8a70b115f2 Aggregations: More consistent response format for scripted metrics aggregation
Changes the name of the field in the scripted metrics aggregation from 'aggregation' to 'value' to be more in line with the other metrics aggregations like 'avg'
2014-09-17 11:46:26 +01:00
javanna dd2ef8e014 Internal: make sure that internally generated percolate request re-uses the original headers and request context
Closes #7767
2014-09-17 12:34:29 +02:00
Shay Banon b75d1d885a Add missing cluster blocks handling for master operations
Master node related operations were missing proper handling of cluster blocks, allowing for example to perform cluster level update settings even before the state was fully restored on initial cluster startup

Note, the change allows to change read only related settings without checking for blocks on update settings, as without it, it means one can't re-enable metadata/write. Also, it doesn't check for blocks on cluster state and health API, as those are allowed to be used even when blocked to figure out what causes the block.
closes #7763
closes #7740
2014-09-17 10:55:34 +02:00
Simon Willnauer a2d07058e8 [CORE] Notify listener when execution was rejected 2014-09-17 09:48:51 +02:00
Britta Weber 364de19251 [TEST] wait until all nodes have joined the cluster after upgrade
upgradeOneNode() only checked if the new node is in the nodes info.
However, this does not guarantee that all nodes have joined the cluster
already. For example the new node could be the master and might not yet
know about all nodes and the other nodes might not know about the new
master yet.
Depending on which client is picked later, the client might then try to
send request to the old node that was shut down instead of the new one.
Instead of just checking if the new node is in the nodes info we should
therefore also check if the all nodes are in the nodes info.
2014-09-16 23:17:30 +02:00
Simon Willnauer 76657251e0 [SNAPSHOT] Reset missing file hash instead of existing hash
Commit e8a1f2598b504183c1a3f2e60363ceaa0d4b298e introduced a regression
where the already existing hash was replaced instead of the missing.
2014-09-16 22:51:54 +02:00
Simon Willnauer 21f6bc84fa [TEST] Mute DedicatedClusterSnapshotRestoreTests#restorePersistentSettingsTest 2014-09-16 22:13:25 +02:00
Simon Willnauer d2e19ea665 [TEST] Wait for nodes before calling the API stats API 2014-09-16 19:35:35 +02:00
Boaz Leskes 2083ca0aa3 Discovery: UnicastZenPing don't rename configure host name
#7719 introduced temporary node ids for nodes that can't be resolved via their address. The change is overly aggressive and creates temporary nodes also for the configure target hosts.

Closes #7747
2014-09-16 18:05:35 +02:00
javanna e78737b19b [TEST] Update REST client before each test in our REST tests
In #7723 we removed the `updateAddresses` method from `RestClient` under the assumption that the addresses never change during the suite execution, as REST tests rely on the global cluster. Due to #6734 we restart the global cluster though before each test if there was a failure in the suite. If that happens we do need to make sure that the REST client points to the proper nodes. What was missing before was the http call to verify the es version every time the addresses change, which we do now since we effectively re-initialize the REST client when needed (if the http addresses have changed).

Closes #7737
2014-09-16 16:09:48 +02:00
Simon Willnauer cc99bfe802 [TEST] ensure HTTP is enabled in JsonP tests 2014-09-16 14:56:05 +02:00
Simon Willnauer f4d8f1673a [TEST] Improve HTTP support in TestClusters
This commit disalbes HTTP for all the suite and test scope tests
since it's an unused / unneeded module which takes time to startup.
This also uses a JVM private port range for HTTP ports to ensure
there are no cross JVM conflicts.
2014-09-16 14:31:43 +02:00
Simon Willnauer 3ed32e022e [SNAPSHOT] Trigger retry logic also if we hit a JsonException
We rely on retry logic when reading a snapshot since it's concurrently
serialized. We should move to a better logic here but the refactoring
of the blobstore change the semantics and this now throws Json
exceptions rather than returning an unexpected Token
2014-09-16 14:15:00 +02:00
Shay Banon 99f91f7616 Bulk operation can create duplicates on primary relocation
When executing a bulk request, with create index operation and auto generate id, if while the primary is relocating the bulk is executed, and the relocation is done while N items from the bulk have executed, the full shard bulk request will be retried on the new primary. This can create duplicates because the request is not makred as potentially holding conflicts.

This change carries over the response for each item on the request level, and if a conflict is detected on the primary shard, and the response is there (indicating that the request was executed once already), use the mentioned response as the actual response for that bulk shard item.

On top of that, when a primary fails and is retried, the change now marks the request as potentially causing duplicates, so the actual impl will do the extra lookup needed.

This change also fixes a bug in our exception handling on the replica, where if a specific item failed, and its not an exception we can ignore, we should actually cause the shard to fail.

closes #7729
2014-09-16 12:13:39 +02:00
Boaz Leskes 12cbb3223a Discovery: node join requests should be handled at lower priority than master election
When a node is elected as master or receives a join request, we submit a cluster state update task. We should give the node join update task a lower priority than the elect as master to increase the chance it will not be rejected. During master election there is a big chance that these will happen concurrently.

This commit lowers the priority of node joins from IMMEDIATE to URGENT

Closes #7733
2014-09-16 11:41:59 +02:00
Simon Willnauer ec28d7c465 [STORE] Fold two hashFile implemenation into one 2014-09-16 11:01:49 +02:00
Simon Willnauer 723a40ef34 [VERSION] s/V_1_4_0_Beta/V_1_4_0_Beta1/g 2014-09-16 10:54:41 +02:00
Simon Willnauer a7dde8dd80 [TEST] Make flush in #indexRandom optinal
Some tests like CorruptedTranslogTests rely on the fact that we
are recovering from translog. In those cases we need to prevent
flushes from happening during indexing. This change adds an optional
flag on the #indexRandom utility to disable flushes.
2014-09-16 10:43:28 +02:00
javanna 38f5aa2248 [TEST] Fixed ActionNamesTests to not use random action names that conflict with existing ones
ActionNamesTests#testIncomingAction rarely uses a random action name to make sure that actions registered via plugins work properly. In some cases the random action would conflict with existing one (e.g. tv) and make the test fail. Fixed also testOutgoingAction although the probability of conflict there is way lower due to longer action names used from 1.4 on.
2014-09-16 10:23:02 +02:00
Simon Willnauer fbf2c3f9f7 [TEST] Use node names for transport clients an close them in tests 2014-09-16 10:08:25 +02:00
Boaz Leskes 7253646664 Discovery: not all master election related cluster state update task use Priority.IMMEDIATE
Most notably the elected_as_master task should run as soon as possible. This is an issue as node join request do use `Priority.IMMEDIATE` and can be unjustly rejected.

Closes #7718
2014-09-15 21:26:01 +02:00
Boaz Leskes db13eead54 Internal: ClusterHealthAPI does not respect waitForEvents when local flag is set
It uses a cluster state update task and it gets rejected if not run on a master node. We should enable running on non-masters if the local flag is set.

Also, report any unexpected error that may happen during this cluster state update task

Closes #7731
2014-09-15 21:02:23 +02:00
Alexander Reelsen ec86808fa9 Netty: Make sure channel closing never happens on i/o thread
Similar to NettyTransport.doStop() all actions which disconnect
from a node (and thus call awaitUnterruptibly) should not be executed
on the I/O thread.

This patch ensures that all disconnects happen in the generic threadpool, trying to avoid unnecessary `disconnectFromNode` calls.

Also added a missing return statement in case the component was not yet
started when catching an exception on the netty layer.

Closes #7726
2014-09-15 17:54:15 +02:00
Boaz Leskes 2250f58757 Discovery: UnicastZenPing - use temporary node ids if can't resolve node by it's address
The Unicast Zen Ping mechanism is configured to ping certain host:port combinations in order to discover other node. Since this is only a ping, we do not setup a full connection but rather do a light connect with one channel. This light connection is closed at the end of the pinging.

During pinging, we may discover disco nodes which are not yet connected (via temporalResponses). UnicastZenPing will setup the same light connection for those node. However, during pinging a cluster state may arrive with those nodes in it. In that case , we will mistakenly believe those nodes are connected and at the end of pinging we will mistakenly disconnect those valid node.

This commit makes sure that all nodes UnicastZenPing connects to have a unique id and can be safely disconnected.

Closes #7719
2014-09-15 16:47:39 +02:00
mikemccand 7ca64237a8 Test: always run CheckIndex for these two tests 2014-09-15 09:58:14 -04:00
Simon Willnauer 509f71cd55 [TEST] Log if we use transport client in trace mode 2014-09-15 15:42:34 +02:00
Boaz Leskes d228606bab Recovery: remove unneeded waits on recovery cancellation
When cancelling recoveries, we wait for up to 10s for the source node to be notified before continuing. This is not needed in two cases:
1) The source node has been disconnected due to node shutdown (recovery is canceled as a response to cluster state processing)
2) The current thread is the one that will be notifying the source node (happens when one of the calls from the source nodes discoveres the local index is closed)

The first one is especially important as it may delay cluster state update processing with 10s.

Closes #7717
2014-09-15 15:30:54 +02:00
javanna ede39edbba [TEST] Minor REST tests infra cleanup
Make the http addresses within the REST client final. It makes no sense to update them before each test if we don't check the version of the nodes again, which would mean adding too much overhead (an additional http call before each test) for no reason. We just reuse the same nodes for the whole suite and check the version once while initializing the client. Would be nice to make the REST client within the execution context final but its initialization still needs to happen after the `ElasticsearchIntegrationTest#beforeInternal` that assigns `GLOBAL_CLUSTER` to `currentCluster`.

Closes #7723
2014-09-15 15:27:14 +02:00
uboness b619cd1112 Added scrollId/s setters to the different scroll requests/responses 2014-09-15 13:56:48 +03:00
Lee Hinman 964db64ed1 Only set `breaker` when stats are retrieved
When communicating with 1.3 and earlier nodes, it's possible that the
field data breaker info is not sent at all. When this happens, we should
leave the `breaker` variable as-is (unset) instead of creating an
AllCircuitBreakerStats object with a null fd breaker and fake request &
parent breakers.
2014-09-15 12:29:57 +02:00
Colin Goodheart-Smithe d4e83df3b8 Aggregations: Adds ability to sort on multiple criteria
The terms aggregation can now support sorting on multiple criteria by replacing the sort object with an array or sort object whose order signifies the priority of the sort. The existing syntax for sorting on a single criteria also still works.

Contributes to #6917
Replaces #7588
2014-09-15 11:08:29 +01:00
Britta Weber be7c75c745 function score: fix cast in Gaussian decay function
Also fix the test
FunctionScoreTests#simpleWeightedFunctionsTestWithRandomWeightsAndRandomCombineMode
which sometimes failed due to rounding issues. Make sure
only floats are returned as scores to assure ratio of
expected and returned score is 1.0f.
2014-09-15 11:46:20 +02:00
Boaz Leskes 3142fec206 Test: ZenUnicastDiscoveryTests.testNormalClusterForming should start unicast hosts first
The test starts a cluster with random nodes as unicast hosts but *doesn't* use min_master_nodes. If the unicast hosts are started last, nodes may elect themselves as master as they do not have mechanism yet to share information.
2014-09-15 11:23:12 +02:00
javanna 8cf922bf9e Internal: make sure that original headers are used when executing search as part of put warmer
Closes #7711
2014-09-15 09:32:17 +02:00
javanna ec5ceecb97 [TEST] Expose ability to provide http headers when sending requests in our REST tests
ElasticsearchRestTests has now a `restClientSettings` method that can be overriden to provide headers as settings (similarly to what we do with transport client). Those headers will be sent together with every REST requests within the tests.

Closes #7710
2014-09-15 09:30:45 +02:00
Boaz Leskes f96bfd3773 Tests: added trace action.search.type to GeoBoundsTests 2014-09-12 20:16:42 +02:00
Britta Weber 5a8ebab96e [TEST] Fix test explain now that explanation is fixed 2014-09-12 18:12:34 +02:00
Philipp Jardas 5e0f67b516 Fixed explanation for GaussDecayFunction
The explanation now gives the correct value instead of the negative.
2014-09-12 18:12:34 +02:00
Martijn van Groningen 91144fc92f Parent/child: If a p/c query is wrapped in a query filter then CustomQueryWrappingFilter must always be used and any filter wrapping the query filter must never be cached.
Closes #7685
2014-09-12 17:22:07 +02:00
markharwood 3c8f8cc090 Aggs enhancement - allow Include/Exclude clauses to use array of terms as alternative to a regex
Closes #6782
2014-09-12 15:28:03 +01:00
Lee Hinman 3e589cd25b [TEST] Additional logging info for node with primary 2014-09-12 15:42:41 +02:00
Colin Goodheart-Smithe 722ff1f56e [TEST] added trace logging for index recovery in GeoBoundsTests 2014-09-12 13:23:45 +01:00
Boaz Leskes 1002ff2f15 Discovery: restore preference to latest unicast pings describing the same node
Closes #7702
2014-09-12 14:02:43 +02:00
Colin Goodheart-Smithe f8d75faaad Geo: Fixes BoundingBox across complete longitudinal range
Adds a special case to the GeoBoundingBoxFilterParser so that the left of the box is not normalised in the case where left and right are 360 apart.  Before this change the left would be normalised to 180 in this case and the filter would only match points with a longitude of 180 (or -180).

Closes #5128
2014-09-12 10:09:02 +01:00
Colin Goodheart-Smithe 2837800500 [TEST] Adds tests for GeoUtils 2014-09-12 09:43:25 +01:00
Boaz Leskes 1bd2a491d1 Tests: add a comment to DiscoveryWithServiceDisruptions.testAckedIndexing reminding to port it to 1.x once the awaitFix is removed 2014-09-12 10:35:18 +02:00
Boaz Leskes 5b461454c2 Tests: add an awaitFix to IndicesLifecycleListenerTests 2014-09-12 10:24:55 +02:00
Simon Willnauer a3f2677b70 [CORE] Ensure GroupShardsIterator is consistent across requests
GroupShardsIterator is used in many places like the search execution
to determin which shards to query. This can hold shards of one index
as well as shards of multiple indices. The iteration order is used
to assigne a per-request shard ID for each shard that is used as a
tie-breaker when scores are the same. Today the iteration order is
soely depending on the HashMap iteration order which is undefined or
rather implementation dependent. This causes search requests to return
inconsistent results across requests if for instance different nodes
are coordinating the requests.

Simple queries like `match_all` may return results in arbitrary order
if pagination is used or may even return different results for the same
request even though there hasn't been a refresh call and preferences are
used.
2014-09-12 07:33:07 +02:00
Simon Willnauer 929a4a54f7 [VERSION] Added Version [1.5.0] 2014-09-11 22:38:03 +02:00
Simon Willnauer b9ee915763 [Version] Add Version 1.4.0-Beta 2014-09-11 22:10:19 +02:00
Simon Willnauer a0e9951e8a [STORE] Turn unexpected exception into CorruptedIndexException
Today if we run into exception like NumberFormatException or IAE
when we try to open a commit point to retrieve checksums and calculate
store metadata we just bubble them up. Yet, those are very likely index
corruptions. In such a case we should really mark the shard as
corrupted.
2014-09-11 21:15:02 +02:00
Britta Weber 9b5497f6ca [TEST] fix another rounding issue 2014-09-11 19:49:51 +02:00
Simon Willnauer b0a377bae8 [TEST] Use a sorted set since sets are compared and compare is order specific 2014-09-11 17:42:37 +02:00
javanna 7e0481d906 More Like This API: remove unused search_query_hint parameter
Closes #7691
2014-09-11 17:34:54 +02:00
Martijn van Groningen d0300b3f59 Aggregations top_hits: Fixed inconsistent sorting of the hits
In the reduce logic of the top_hits aggregation if the first shard result to process contained has no results then the merging of all the shard results can go wrong resulting in an incorrect sorted hits.
This bug can only manifest with a sort other than score.

Closes #7697
2014-09-11 17:26:22 +02:00
Simon Willnauer 3ef6860679 [STORE] Improve exception from Store.failIfCorrupted
If you have previously corrupted files, this method currently builds an
exception like:
```
    failed engine [corrupted preexisting index]
    failed to start shard
```

Followed by a CorruptIndexException. This commit writes the entire
stacktrace to provide additional information. It also changes the
failure message from `corrupted preexisting index` to `preexisting
corrupted index` to prevent confusion.

Closes #7596
2014-09-11 17:11:48 +02:00
Simon Willnauer 595472014e [TEST] Use a real unique clustername for InternalTestClusterTests 2014-09-11 16:20:51 +02:00
Martijn van Groningen f8e93fa2aa Test: Let both types have a non_analyzed id field. 2014-09-11 15:20:28 +02:00
javanna fd6798df69 [TEST] parse response body as json depending on the content-type in our REST tests 2014-09-11 14:32:59 +02:00
javanna 4ab268bab2 Internal: refactor copy headers mechanism to not require a client factory
With #7594 we replaced the static `BaseRestHandler#addUsefulHeaders` by introducing the `RestClientFactory` that can be injected and used to register the relevant headers. To simplify things, we can now register relevant headers through the `RestController` and remove the `RestClientFactory` that was just introduced.

Closes #7675
2014-09-11 13:18:08 +02:00
Colin Goodheart-Smithe 8720a4dcd2 [TEST] added exception for GET index API to bwc tests 2014-09-11 11:59:17 +01:00
Colin Goodheart-Smithe 5fe782b784 Indices API: Added GET Index API
Returns information about settings, aliases, warmers, and mappings. Basically returns the IndexMetadata. This new endpoint replaces the /{index}/_alias|_aliases|_mapping|_mappings|_settings|_warmer|_warmers and /_alias|_aliases|_mapping|_mappings|_settings|_warmer|_warmers endpoints whilst maintaining the same response formats.  The only exception to this is on the /_alias|_aliases|_warmer|_warmers endpoint which will now return a section for 'aliases' or 'warmers' even if no aliases or warmers exist. This backwards compatibility change is documented in the reference docs.

Closes #4069
2014-09-11 11:19:21 +01:00
Boaz Leskes a50934ea3e Resiliency: Master election should demotes nodes which try to join the cluster for the first time
With the change in #7493,  we introduced a pinging round when a master nodes goes down. That pinging round helps validating the current state of the cluster and takes, by default, 3 seconds. It may be that during that window, a new node tries to join the cluster and starts pinging (this is typical when you quickly restart the current master).  If this node gets elected as the new master it will force recovery from the gateway (it has no in memory cluster state), which in turn will cause a full cluster shard synchronisation. While this is not a problem on it's own, it's a shame. This commit demotes "new" nodes during master election so the will only be elected if really needed.

Closes #7558
2014-09-11 11:19:10 +02:00
Simon Willnauer 8618d3624a Add inline comment to prevent confusion 2014-09-11 11:07:23 +02:00