Commit Graph

9712 Commits

Author SHA1 Message Date
Michael McCandless 5e9e2cf50c Core: try again to upgrade to Lucene 4.10.1-snapshot 2014-09-24 13:48:49 -04:00
Michael McCandless ab3be76644 Revert Lucene upgrade 2014-09-24 13:25:55 -04:00
Michael McCandless 15c75b1967 Core: upgrade to Lucene 4.10.1 snapshot
Lucene will soon release official 4.10.1, but by upgrading sooner we can 1) sidestep the false failures due to the 1.8.0_20 JVM hotspot bug (has caused a number of false failures in recent Jenkins tests), 2) make sure none of the Lucene changes in 4.10.1 are problematic.

Closes #7844
2014-09-24 13:13:07 -04:00
Boaz Leskes 0f121ff351 Test: ClusterServiceTests.testLocalNodeMasterListenerCallbacks - increase ping timeout
was 200ms, now 400ms
2014-09-24 17:26:40 +02:00
javanna 17b1fd1a6a Internal: split internal free context request used after scroll and search
We currently use the same internal request when we need to free the search context after a search and a scroll. The two original requests though diverged after #6933 as `SearchRequest` implements `IndicesRequest` while `SearchScrollRequest` and `ClearScrollRequest` don't. That said, with #7319 we made `SearchFreeContextRequest` implement `IndicesRequest` by making it hold the original indices taken from the original request, which are null if the free context was originated by a scroll or by a clear scroll call, and that is why original indices are optional there.

This commit introduces a separate free context request and transport action for scroll, which doesn't hold original indices. The new action is only used against nodes that expose it, the previous action name will be used for nodes older than 1.4.0.Beta1.

As a result, in 1.4 we have a new `indices:data/read/search[free_context/scroll]` action that is equivalent to the previous `indices:data/read/search[free_context]` whose request implements now `IndicesRequest` and holds the original indices coming from the original request. The original indices in the latter requests can only be null during a rolling upgrade (already existing version checks make sure that serialization is bw compatible), when some nodes are still < 1.4.

Closes #7856
2014-09-24 15:33:27 +02:00
Simon Willnauer ea49a3e269 Update version flags after backporting to 1.3.3
Relates to #7857
2014-09-24 15:23:57 +02:00
Colin Goodheart-Smithe f37815a53b Aggregations: Significant Terms Heuristics now registered correctly
Closes #7840
2014-09-24 11:48:26 +01:00
David Pilato f2be8d83dc Docs: fix typo 2014-09-23 23:41:07 +02:00
Simon Willnauer ca86e1c824 [TEST] Disable thread filter for now 2014-09-23 14:01:22 +02:00
Simon Willnauer 45319dc27f [TEST] use more terminate calls and wait for termination 2014-09-23 13:58:47 +02:00
Simon Willnauer 30acba624d [TEST] Add a more restrictive thread leaks filter
Today all threads are allowed to leak a suite. This is tricky since
it essentially allows resource leaks by default where for instance
test private TransportClients will never get closed and consume
resources influencing other tests. It also hides threads that
are not fully under elasticsearchs control like the Lucene
TimeLimitingCollector thread. This commit restricts the threads
that can leak a suite to the threads spawned from testclusters
and fixes sevearl places that leaked threads.

Closes #7833
2014-09-23 13:36:21 +02:00
David Pilato 046a3a02f5 Docs: fix no callouts refer to list item 3 2014-09-23 13:30:36 +02:00
Simon Willnauer 18212ba09c [TEST] Make sure test actually throttles 2014-09-23 12:53:47 +02:00
David Pilato cdd4d00f46 Docs: java add static imports for query and filter builders
Also move and add inline comments to references

Closes #6920.
2014-09-23 12:39:57 +02:00
Simon Willnauer 68c5206e50 [TEST] disable translog based flushes when corrupting files
These tests rely on the fact that all files stay the same after
the corruption and if we run into a translog based flush we might
use a new / different delete file causing the test to fail.
2014-09-23 12:30:06 +02:00
Shay Banon d4d77cdb66 Chunk direct buffer usage by networking layer
Today, due to how netty works (both on http layer and transport layer), and even though the buffers sent over to netty are paged (CompositeChannelBuffer), it ends up re-copying the whole buffer into another heap buffer (bad), and then send it over directly to sun.nio which allocates a full thread local direct buffer to send it (which can be repeated if not all message is sent).
  This is problematic for very large messages, aside from the extra heap temporal usage, the large direct buffers will stay around and not released by the JVM.
  This change forces the use of gathering when building a CompositeChannelBuffer, which results in netty using the sun.nio write method that accepts an array of ByteBuffer (so no extra heap copying), and also reduces the amount of direct memory allocated for large messages.
  See the doc on NettyUtils#DEFAULT_GATHERING for more info.
closes #7811
2014-09-23 12:15:19 +02:00
Simon Willnauer 6c8aa5fa6c [RECOVERY] Mark last file chunk to fail fast if payload is truncated
Today we rely on the metadata length of the file we are recoverying
to indicate when the last chunk was received. Yet, this might hide bugs
on the compression layer if payloads are truncated. We should indicate
if the last chunk is send to make sure we validate checksums
accordingly if possible.

Closes #7830
2014-09-23 11:32:18 +02:00
Simon Willnauer 5533495171 [TEST] Ensure primaries are allocated before bulk indexing with dymamic mappings 2014-09-23 08:34:25 +02:00
David Pilato f0dc8a8ffb Document the Java BulkProcessor
Closes #7638.
2014-09-22 15:32:29 +02:00
David Pilato 53c0e92801 Docs: IndexResponse.matches() does not exist anymore
Since 1.0, percolator has been redesigned so percolator is not applied anymore at index time.

Closes #7548.
2014-09-22 15:20:31 +02:00
Shay Banon f6a0fe5c2f Apply bulk change to 1.3.3
relates to #7729
2014-09-22 15:13:35 +02:00
Elie A 40608ce266 Update java source example
From the version 1.0 FilterBuilders and QueryBuilders are not part from org.elasticsearch.index.query.xcontent package no more.

Closes #7701.

(cherry picked from commit 32d4200)
2014-09-22 14:37:27 +02:00
Boaz Leskes b1851906d8 Tests: extend testRecoverFromPreviousVersion to sometimes index during relocation
Relates to #7729

Closes #7768
2014-09-22 11:19:38 +02:00
Boaz Leskes 4677d05048 Recovery: mapping check during phase2 should be done in cluster state update task
Before phase2 we check verify that the local mapping is in sync with the cluster state mapping (and send & wait on a master update mapping task if not). This check should be done under a cluster state update task to make sure an incoming cluster state update to do not change things while we check.

Closes #7744
2014-09-22 11:05:00 +02:00
Boaz Leskes d17fd26f23 Test: RecoveryWhileUnderLoadTests.recoverWhileRelocating should report cluster state when failing to reach green 2014-09-21 20:16:45 +02:00
Boaz Leskes 41fd5d02f4 Discovery: Give a unique id to each ping response
During discovery a node gossips with other nodes to discover the current state of the cluster - what nodes are out there, what version they use and most importantly whether there is an active master out there. During this ping process we may end up in a situation where old information is mixed with new. This is comment if a couple of master election happen in rapid succession.

This commit adds a monotonically increasing id to each ping response. This makes it easy to always select the last ping from every node.

Closes #7769
2014-09-20 12:58:15 +02:00
Martijn van Groningen afcbffbfc1 Core: Check if from + size don't cause overflow and fail with a better error.
Closes #7778
2014-09-20 12:34:48 +02:00
mikemccand dbe4e6e674 Internal: remove ForceSyncDirectory
Historical code, not used anymore.

Closes #7804
2014-09-19 14:15:06 -04:00
Brian Murphy 8e742c2096 Indexed Scripts/Templates : Cleanup
This contains several cleanups to the indexed scripts.
Remove the unused FetchSourceContext from the Get request..
Add lang,_version,_id to the REST GET API.
Removes the routing from GetIndexedScriptRequest since the script index is a single shard that is replicated across all nodes.
Fix backward compatible template file reference
Before 1.3.0 on disk scripts could be referenced by requesting
````
_search/template

{
  "template" : "ondiskscript"
}
````
This was broken in 1.3.0 by requiring
````
{
  "template" :
  {
    "file" : "ondiskscript"
  }
}
````
This commit restores the previous behavior.
Remove support for preference, realtime and refresh
These parameters don't make sense anymore for indexed scripts as we always force the preference to _local and
always refresh after a Put to the indexed scripts index.

Closes #7568
Closes #7559
Closes #7647
Closes #7567
2014-09-19 11:59:08 +01:00
Lee Hinman 4185566e93 Add option to take currently relocating shards' sizes into account
When using the DiskThresholdDecider, it's possible that shards could
already be marked as relocating to the node being evaluated. This commit
adds a new setting `cluster.routing.allocation.disk.include_relocations`
which adds the size of the shards currently being relocated to this node
to the node's used disk space.

This new option defaults to `true`, however it's possible to
over-estimate the usage for a node if the relocation is already
partially complete, for instance:

A node with a 10gb shard that's 45% of the way through a relocation
would add 10gb + (.45 * 10) = 14.5gb to the node's disk usage before
examining the watermarks to see if a new shard can be allocated.

Fixes #7753
Relates to #6168
2014-09-19 12:36:51 +02:00
Brian Murphy 61c21f9a0e Bulk API: Do not fail whole request on closed index
The bulk API request was marked as completely failed,
in case a request with a closed index was referred in
any of the requests inside of a bulk one.

Implementation Note: Currently the implementation is a bit more verbose in order to prevent an instanceof check and another cast - if that is fast enough, we could execute that logic only once at the
beginning of the loop (thinking this might be a bit overoptimization here).

Closes #6410
2014-09-19 10:55:49 +01:00
Brian Murphy 4f791b06db Revert "Bulk API: Do not fail whole request on closed index"
This reverts commit 405e5816b8.
2014-09-19 10:27:28 +01:00
Brian Murphy c7c61bfd91 Revert "Bulk Request : Add Document Request"
This reverts commit 86f575dcea.
2014-09-19 10:27:16 +01:00
Brian Murphy 86f575dcea Bulk Request : Add Document Request
This file was missing.
2014-09-19 10:05:45 +01:00
Brian Murphy 405e5816b8 Bulk API: Do not fail whole request on closed index
The bulk API request was marked as completely failed,
in case a request with a closed index was referred in
any of the requests inside of a bulk one.

Implementation Note: Currently the implementation is a bit more verbose in order to prevent an instanceof check and another cast - if that is fast enough, we could execute that logic only once at the beginning of the loop (thinking this might be a bit overoptimization here).

Closes #6410
2014-09-19 09:56:49 +01:00
javanna 4fa924494d [TEST] move REST tests to their own test group
Closes #7795
2014-09-19 10:34:30 +02:00
Simon Willnauer 9f6d6d540b [ENGINE] try increment store before searcher is acquired
InternalEngine#refreshNeeded must increment the ref count on the
store used before it's checking if the searcher is current since
internally a searcher ref is acquired and if that happens concurrently
to a engine close it might violate the assumption that all files
are closed when the store is closed.

This commit also converts some try / finally into try / with.
2014-09-19 00:37:30 +02:00
javanna 508ff29e0d [TEST] allow to fully disable REST tests included parsing via -Dtests.rest=false
We currently look for REST tests on file system although they are disabled. We should not do that and move the check earlier on. This way third parties using our test infra, which don't have REST tests on file system, can effectively disable the REST tests, otherwise they would get initialization error despite having disabled them.  The downside is that the number of tests visualized is going to be zero instead of the real number of parsed REST tests, but there is nothing we can do about this. Tests get ignored anyways.
2014-09-18 17:05:34 +02:00
Colin Goodheart-Smithe 66417a93a0 Aggregations: Removes isSingleUserCriteria check
This change removes the backwards compatibility workaround that checks that a compoundOrder originated from a single user defined criteria for the purposes of serialising to older versioned nodes.
2014-09-18 15:22:43 +01:00
javanna 5e1f95ca93 [TEST] close REST test execution context only if not null
The context can be null when REST tests are disabled via sysprop.
2014-09-18 16:14:59 +02:00
Colin Goodheart-Smithe f0e9b7b8ef [DOC] Add GET Alias API note to breaking changes
Note explains that GET Alias API now supports IndicesOptions and will error if a index is missing
2014-09-18 15:09:01 +01:00
Simon Willnauer d3e348ef90 [CORE] Add AbstractRunnable support to ThreadPool to simplify async operation on bounded threadpools
today we have to catch rejected operation exceptions in various places
and notify an ActionListener. This pattern is error prone and adds a lot
of boilerplait code. It's also easy to miss catching this exception
which only is relevant if nodes are under high load. This commit adds
infrastructure that makes ActionListener first class citizen on async
actions.

Closes #7765
2014-09-18 15:25:10 +02:00
Simon Willnauer b2477a43c8 [TEST] Reimplement AckTests#testDeleteWarmerNoAcknowledgement
This test was not testing what it was supposed to test. This commit
implements the test as an actual delete warmer test without ack
returned.
2014-09-18 15:21:02 +02:00
javanna 5f97bccb54 Internal: add indices setter to IndicesRequest interface
We currently expose generic getters for indices and indicesOptions on the IndicesRequest interface. This commit adds a generic setter as well, which can be used to set the indices to a request. The setter impl throws `UnsupportedOperationException` if called on internal requests. Also throws exception if called on single index operations, since it accepts an array as argument.

Closes #7734
2014-09-18 14:26:11 +02:00
javanna 6717de9e46 Internal: make sure that update internal requests share the same original headers and request context
Update request internally executes index and delete operations. We need to make sure that those internal operations hold the same headers and context as the original update request. Achieved via copy constructors that accept the current request and the original request.

Closes #7766
2014-09-18 14:00:51 +02:00
javanna b9b5842acc Internal: make sure that all delete mapping internal requests share the same original headers and context
Delete mapping executes flush, delete by query and refresh operations internally. Those internal requests are now initialized by passing in the original delete mapping request so that its headers and request context are kept around.

Closes #7736
2014-09-18 13:58:03 +02:00
Martijn van Groningen f43a8e2961 Aggregations: Fix regression bug for the support of terms aggregation on the `_parent` field. 2014-09-18 12:27:55 +02:00
Simon Willnauer 66421c5a83 [TEST] Only reset test cluster if a test actually failed
Previously we resetted the test cluster for all subsequent tests
even though they didn't fail. This make suites like REST tests faster
and prevents crazy timeouts.

Closes #7775
2014-09-18 10:40:30 +02:00
Simon Willnauer dd97a95b04 [TEST] Wait until warmer is registered when testing timeout 2014-09-18 09:15:17 +02:00
Simon Willnauer 19c969a800 [SNAPSHOT] Minor code cleanup 2014-09-17 19:17:40 +02:00