Commit Graph

5595 Commits

Author SHA1 Message Date
Simon Willnauer cfa4689445 Share test injector creation 2016-06-20 11:38:48 +02:00
Simon Willnauer 7fea5bd8e7 Remove obsolete Modules that can simply be inlined in node creation 2016-06-20 11:28:14 +02:00
Simon Willnauer 260f38fd76 Remove VersionModule and use Version#current consistently.
We pretended to be able to ackt like a different version node for so long it's
time to be honest and remove this ability. It's just confusing and where needed
and tested we should build dedicated extension points.
2016-06-20 10:55:52 +02:00
Tanguy Leroux 98951b1203 Compile each Groovy script in its own classloader
closes #18572
2016-06-20 08:17:09 +02:00
Florian Hopf 6b46bf13f0 Throw if the local node is not set
This commit adds an IllegalStateException if attempting to get the local
node from the cluster service when it is not set.

Relates #18963
2016-06-19 17:25:18 -04:00
Boaz Leskes 8f96dd53aa mark ESIndexLevelReplicationTestCase as abstract so it won't fail naming conventions 2016-06-18 19:39:37 +02:00
Boaz Leskes 61b7f49ed2 ESIndexLevelReplicationTestCase.java fix line lengths 2016-06-18 19:13:50 +02:00
Boaz Leskes 14cd8a6794 Introduce Replication unit tests using real shards (#18930)
This commit introduce unit testing infrastructure to test replication operations using real index shards. This is infra is complementary to the full integration tests and unit testing of ReplicationOperation we already have. The new ESIndexLevelReplicationTestCase base makes it easier to test and simulate failure mode that require real shards and but do not need the full blow stack of a complete node.

The commit also add a simple "nothing is wrong" test plus a test that checks we don't drop docs during the various stages of recovery.

For now, only single doc indexing is supported but this can be easily extended in the future.
2016-06-18 18:53:47 +02:00
Boaz Leskes 535a245354 testSingleBatchSubmission - random numbers don't have to be unique 2016-06-18 18:44:54 +02:00
Simon Willnauer 4cb2bb635b Remove forked joda time BaseDateTime class (#18953)
This class was forked in 0.20 to remove a volatile keyword. While there
is no issue attached to the commit, no evidence of the criticality of the
change nor does it seem to be correct since we set this value internally as well
I think this class should be used as is from joda time even if we have to pay
the price of volatile reads. We can't do 3rd party optimization in our codebase that
way it just not maintainable.

This was added in 2280915d3c
2016-06-18 17:06:18 +02:00
Simon Willnauer 420dc72124 Add ClusterName#value as the default instead of it's toString method 2016-06-18 08:40:01 +02:00
Jeff Evans e9f2548ee0 Include script field even if it value is null
Include script field even if it value is null.

Closes #16408.
2016-06-17 16:41:25 -04:00
Jason Tedor d09d89f8c5 Remove only node preference
This commit removes the search preference _only_node as the same
functionality can be obtained by using the search preference
_only_nodes. This commit also adds a test that ensures that _only_nodes
will continue to support specifying node IDs.

Relates #18875
2016-06-17 15:27:46 -04:00
Areek Zillur 9bca264dcd Merge branch 'master' of https://github.com/elastic/elasticsearch 2016-06-17 11:41:23 -04:00
Areek Zillur 9356a6090f Merge branch 'master' into enhancement/rollover_api 2016-06-17 11:35:57 -04:00
Boaz Leskes 46b40f73b7 Inline reroute with process of node join/master election (#18938)
In the past, we had the semantics where the very first cluster state a node processed after joining could not contain shard assignment to it. This was to make sure the node cleans up local / stale shard copies before receiving new ones that might confuse it. Since then a lot of work in this area, most notably the introduction of allocation ids and #17270 . This means we don't have to be careful and just reroute in the same cluster state change where we process the join, keeping things simple and following the same pattern we have in other places.
2016-06-17 17:32:38 +02:00
Jim Ferenczi fb2a48d0f0 Revert "Remove support for sorting terms aggregation by ascending count"
This is delayed after alpha4 since Kibana relies on it.
2016-06-17 17:14:01 +02:00
Simon Willnauer bdb6dcea3a Cleanup ClusterService dependencies and detached from Guice (#18941)
This change removes some unnecessary dependencies from ClusterService
and cleans up ClusterName creation. ClusterService is now not created
by guice anymore.
2016-06-17 17:07:19 +02:00
Areek Zillur 545ffa7801 Merge branch 'master' into enhancement/rollover_api 2016-06-17 10:33:11 -04:00
Jim Ferenczi 755721953b Remove support for sorting terms aggregation by ascending count
closes #17614
2016-06-17 15:06:49 +02:00
Adrien Grand 712e387058 Rename PipelineAggregatorBuilder to PipelineAggregationBuilder.
This is a follow-up to #18377.
2016-06-17 14:35:49 +02:00
javanna af93533a17 Merge branch 'master' into feature/http_client 2016-06-17 13:50:18 +02:00
Jim Ferenczi 529c2ca13f Add did-you-mean for plugin cli
This commit adds error messages like: `Unknown plugin xpack, did you mean [x-pack]?`

Closes #18896
2016-06-17 12:17:48 +02:00
Boaz Leskes f256769179 Simplify NodeJoinController to make use of new cluster state batching infra (#18832)
The NodeJoinController is responsible for processing joins from nodes, both normally and during master election. For both use cases, the class processes incoming joins in batches in order to be efficient and to accumulated enough joins (i.e., >= min_master_nodes) to seal an election and ensure the new cluster state can be committed. Since the class was written, we introduced a new infrastructure to support batch changes to the cluster state at the `ClusterService` level. This commit rewrites NodeJoinController to use that infra and be simpler.

The PR also introduces a new concept to ClusterService allowing to submit tasks in batches, guaranteeing that all tasks submitted in a batch will be processed together (potentially with more tasks).  On top of that I added some extra safety checks to the ClusterService, around potential double submission of task objects into the queue.

This is done in preparation to revive #17811
2016-06-17 09:22:15 +02:00
Adrien Grand 600cbb6ab0 Upgrade to Lucene 6.1.0. #18926 2016-06-17 09:03:00 +02:00
Areek Zillur 6adffa6b7b Merge branch 'master' into enhancement/rollover_api 2016-06-16 17:27:32 -04:00
Ryan Ernst 8196cf01e3 Merge branch 'master' into plugin_name_api 2016-06-16 13:49:28 -07:00
Ryan Ernst 96321d7749 Remove outtdated comment referring to name/description for Plugin class 2016-06-16 10:18:10 -07:00
Alexander Kazakov 9eea1b6833 Fix flat_settings REST parameter
* Get XContent params from request in Nodes rest actions
* Adding test for nodes info rest api
2016-06-16 10:03:51 -04:00
Simon Willnauer b22c526b34 Cut over settings registration to a pull model (#18890)
Today we have a push model for registering basically anything. All our extension points
are defined on modules which we pass in to plugins. This is harder to maintain and adds
unnecessary dependencies on the modules itself. This change moves towards a pull model
where the plugin offers a getter kind of method to get the extensions. This will also
help in the future if we need to pass dependencies to the extension points which can
easily be defined on the method as arguments if a pull model is used.
2016-06-16 15:52:58 +02:00
Nik Everett 5aa4769b25 Move waitForTaskCompletion into TaskManager
This allows for listening for the waiting to start using
MockTaskManager. This allows us to work around a race condition
in the TasksIT.
2016-06-16 09:45:46 -04:00
Daniel Mitterdorfer 0faa9409b3 Force test infra to use node client in NettyHttpRequestSizeLimitIT 2016-06-16 15:36:29 +02:00
Simon Willnauer e442f07460 Remove dead code 2016-06-16 15:29:49 +02:00
Tanguy Leroux 3c9712794e Merge pull request #18586 from a2lin/msearch_error_fix
Adding status field in _msearch error request bodies
2016-06-16 14:31:39 +02:00
Jim Ferenczi ad232aebbe Set collection mode to breadth_first in the terms aggregation when the cardinality of the field is unknown or smaller than the requested size.
closes #9825
2016-06-16 11:33:40 +02:00
Mike McCandless 3f221bf7cb Add total_indexing_buffer/_in_bytes to nodes info API 2016-06-16 04:39:34 -04:00
Christoph Büscher 01004c72ba Improve TimeZoneRoundingTests error messages
Currently the error messages for failing tests in the TimeZoneRoundingTests test
suite are hard to read because they usually report the actual end expected date
in milliseconds utc (e.g. "Expected: <1414270860000L> but: was <1414270800000L>".
This makes failing tests hard to read.

This change introduces a new Matcher that can be used for equality checks for
long dates but reports the error both as a formated date string according to
some time zone and also as the actual long values, so you get messages like
"Expected: 2014-10-26T00:01:00.000+03:00 [1414270860000] but:  was
"2014-10-26T00:00:00.000+03:00 [1414270800000]".

Also clean cleaning up some helper methods and generally simplifying a few test
cases. Otherwise this change shouldn't affect either the scope of the test or
anything about the rounding implementation itself.
2016-06-16 10:10:04 +02:00
Adrien Grand 9ffb2ff6ba Expose half-floats. #18887
They have been implemented in https://issues.apache.org/jira/browse/LUCENE-7289.
Ranges are implemented so that the accuracy loss only occurs at index time,
which means that if you are searching for values between A and B, the query will
match exactly all documents whose value rounded to the closest half-float point
is between A and B.
2016-06-16 09:46:39 +02:00
Simon Willnauer 18ff051ad5 Simplify ScriptModule and script registration (#18903)
Registering a script engine or native scripts still uses Guice today
and is much more complicated than needed. This change moves to a pull
based model where script plugins have to implement a dedicated interface
`ScriptPlugin` and defines simple getter returning instances rather than
classes.
2016-06-16 09:35:13 +02:00
Alexander Lin 7d42e7e716 Closes #18013. Added status field to _msearch response bodies. 2016-06-16 00:25:17 -07:00
Daniel Mitterdorfer edf010f878 Force single-node cluster in NettyHttpRequestSizeLimitIT 2016-06-16 07:30:55 +02:00
Ryan Ernst a4503c2aed Plugins: Remove name() and description() from api
In 2.0 we added plugin descriptors which require defining a name and
description for the plugin. However, we still have name() and
description() which must be overriden from the Plugin class. This still
exists for classpath plugins. But classpath plugins are mainly for
tests, and even then, referring to classpath plugins with their class is
a better idea. This change removes name() and description(), replacing
the name for classpath plugins with the full class name.
2016-06-15 17:12:22 -07:00
Nik Everett dc2d7a2a6d Test: wait for task to start before waiting for it to finish (#18902) 2016-06-15 18:42:48 -04:00
Tal Levy a26260fb72 new ScriptProcessor for Ingest (#18193)
add new ScriptProcessor for executing ES Scripts within pipelines
2016-06-15 14:57:18 -07:00
Areek Zillur 1a59a8418a Merge branch 'master' into enhancement/shrink_request_parser 2016-06-15 15:49:40 -04:00
Nik Everett ab2a4a0d72 Fix exception on task not found
Silly protected method....
2016-06-15 15:29:42 -04:00
Nik Everett 8cc848f31c Allow FieldStatsRequest to disable cache 2016-06-15 15:10:46 -04:00
Ryan Ernst 8de90a66a1 Relax plugin id url heuristic, since java uses single slash instead of double 2016-06-15 11:43:38 -07:00
Ryan Ernst 9c65bd4ac4 Merge pull request #18876 from rjernst/plugin_install_unknown
Emit nicer error message when trying to install unknown plugin
2016-06-15 09:56:43 -07:00
Daniel Mitterdorfer cca4529b1c Mute NHRSLIT while investigating 2016-06-15 18:02:31 +02:00
Simon Willnauer 7df5d05c62 Simplify SubFetchPhase interface (#18881)
This interface used to have dedicated methods to prevent calling execute
methods. These methods are unnecessary as the checks can simply be
done inside the execute methods itself. This simplifies the interface
as well as its usage.
2016-06-15 15:49:11 +02:00
Nik Everett e09b6d7ba1 Test: Remove and untrue assertion
Task status might change between list and get.
2016-06-15 09:02:36 -04:00
Daniel Mitterdorfer f32b700472 Exclude admin / diagnostic requests from HTTP request limiting
With this commit we exclude certain HTTP requests that are needed to inspect the cluster
from HTTP request limiting to ensure these commands are processed even in critical
memory conditions.

Relates #17951, relates #18145, closes #18833
2016-06-15 14:29:46 +02:00
javanna ace3a7b146 Merge branch 'master' into feature/http_client 2016-06-15 11:44:46 +02:00
Simon Willnauer 0f87afe2bf [TEST] Fix Highlighters assertion - not guice injected anymore 2016-06-15 09:26:34 +02:00
Simon Willnauer 429dd3a876 Simplify FetchSubPhase registration and detach it from Guice (#18862)
this commit removes FetchSubPhrase registration by class to registration
by instance. No Guice binding needed anymore.
2016-06-15 09:13:02 +02:00
Ryan Ernst 1ecf14cee0 Add test for plugin install heuristic 2016-06-14 23:42:49 -07:00
Ryan Ernst 6db323164e Plugins: Emit nicer error message when trying to install unknown plugin
When installing plugins, we first try the elastic download service for
official plugins, then try maven coordinates, and finally try the
argument as a url. This can lead to confusing error messages about
unknown protocols when eg an official plugin name is mispelled. This
change adds a heuristic for determining if the argument in the final
case is in fact a url that we should try, and gives a simplified error
message in the case it is definitely not a url.

closes #17226
2016-06-14 23:42:34 -07:00
Ryan Ernst fa77d4d885 Test: make secure mock setup work with ibm jdk 2016-06-14 18:46:43 -07:00
Jason Tedor e96722d91c Add search preference to prefer multiple nodes
The search preference _prefer_node allows specifying a single node to
prefer when routing a request. This functionality can be enhanced by
permitting multiple nodes to be preferred. This commit replaces the
search preference _prefer_node with the search preference _prefer_nodes
which supplants the former by specifying a single node and otherwise
adds functionality.

Relates #18872
2016-06-14 21:34:24 -04:00
Martijn van Groningen 14d6b04944 test: don't return 0, at least one request must be added to msearch request
also the maxConcurrentSearchRequests() setter only support positive values.
2016-06-14 23:29:29 +02:00
Nik Everett f7f377791f Test: don't use 0 size for terms aggregation
It is no longer allowed.
2016-06-14 17:27:45 -04:00
Nik Everett c8931768ba Clean up after test failure
If the test fails we properly clean up. Also add a toString
implementation so we get useful results on failure.
2016-06-14 16:16:19 -04:00
Nik Everett 3032a7c653 Cache FieldStats
This caches FieldStats at the field level. For one off requests or for
few indicies this doesn't save anything, but when there are 30 indices,
5 shards, 1 replica, 100 parallel requests this is about twice as fast
as not caching. I expect lots of usage won't see much benefit from this
but pointing kibana to a cluster with many indexes and shards, will be
faster.

Closes #18717
2016-06-14 13:57:18 -04:00
Nik Everett e392e0b1df Create get task API that falls back to the .tasks index
This adds a get task API that supports GET /_tasks/${taskId} and
removes that responsibility from the list tasks API. The get task
API supports wait_for_complation just as the list tasks API does
but doesn't support any of the list task API's filters. In exchange,
it supports falling back to the .results index when the task isn't
running any more. Like any good GET API it 404s when it doesn't
find the task.

Then we change reindex, update-by-query, and delete-by-query to
persist the task result when wait_for_completion=false. The leads
to the neat behavior that, once you start a reindex with
wait_for_completion=false, you can fetch the result of the task by
using the get task API and see the result when it has finished.

Also rename the .results index to .tasks.
2016-06-14 13:37:34 -04:00
Simon Willnauer ee2ba13cce Register Highlighter instances instead of classes (#18859)
This change detaches highlighter registration from Guice. It's just a
small step into the right direction.
2016-06-14 17:04:58 +02:00
Colin Goodheart-Smithe d7e3f9e4eb #18854 Remove size 0 options in aggregations
Remove size 0 options in aggregations
2016-06-14 15:32:42 +01:00
Christoph Büscher 32f141223d Merge pull request #18800 from cbuescher/fix-interval-rounding-uneven
Fix invalid rounding value for TimeIntervalRounding close to DST transitions
2016-06-14 16:22:11 +02:00
Christoph Büscher 03f5aa8ea0 Don't throw IllegalInstantException to determine DST gap
By taking the logic from DateTimeZone#convertLocalToUTC(long, boolean) we
can avoid throwing the exception.
2016-06-14 15:36:00 +02:00
Simon Willnauer 4d78f280ed Remove dead code and dead parameters (#18855) 2016-06-14 15:25:44 +02:00
Christoph Büscher 5abe1f7bb2 Fix invalid rounding value for TimeIntervalRounding close to DST transition
There are edge cases where rounding a date to a certain interval using a time
zone with DST shifts can currently cause the rounded date to be bigger than the
original date. This happens when rounding a date closely after a DST start and
the rounded date falls into the DST gap.

Here is an example for CET time zone, where local time is set forward by one
hour at 2016-03-27T02:00:00+01:00 to 2016-03-27T03:00:00.000+02:00:

The date 2016-03-27T03:01:00.000+02:00 (1459040460000) which is just after the
DST change is first converted to local time (1459047660000). If we then apply
interval rounding for a 14m interval in local time, this  takes us to
1459047240000, which unfortunately falls into the DST gap.  When converting
this back to UTC, joda provides options to throw exceptions on illegal dates
like this, or correct this by adjusting the date to the new time zone offset.
We currently do the later, but this leads to converting this illegal date back
to 2016-03-27T03:54:00.000+02:00 (1459043640000), giving us a date that is
larger than the original date we wanted to round.

This change fixes this by using the "strict" option of 'convertLocalToUTC()'
to detect rounded dates that fall into the DST gap. If this happens, we can use
the time of the DST change instead as the interval start.

Even before this change, intervals around DST shifts like this can be shorter
than the desired interval.  This, for example, happens when the requested
interval width doesn't completely fit into the remaining time span when the DST
shift happens. For example, using a 14m interval in UTC+1 (CET before DST
starts) leads to the following valid rounding values around the time where DST
happens:

2016-03-27T01:30:00+01:00
2016-03-27T01:44:00+01:00
2016-03-27T01:58:00+01:00
2016-03-27T02:12:00+01:00
2016-03-27T02:26:00+01:00
...

while the rounding values in UTC+2 (CET after DST start) are placed like this
around the same time:

2016-03-27T02:40:00+02:00
2016-03-27T02:54:00+02:00
2016-03-27T03:08:00+02:00
2016-03-27T03:22:00+02:00
...

From this we can see then when we switch from UTC+1 to UTC+2 at 02:00 the last
rounding value in UTC+1 is at 01:58 and the first valid one in UTC+2 is at
03:08, so even if we decide to put all the dates in between into one rounding
interval, it will only cover 10 minutes. With this change we choose to use the
moment of DST shift as an aditional interval separator, leaving us with a 2min
interval from [01:58,02:00) before the shift and an 8min interval from
[03:00,03:08) after the shift.

This change also adds tests for the above example and adds randomization to the
existing TimeIntervalRounding tests.
2016-06-14 14:59:51 +02:00
Colin Goodheart-Smithe bec621d46f changes from review 2016-06-14 13:45:03 +01:00
Colin Goodheart-Smithe cfd3356ee3 Remove size 0 options in aggregations
This removes the ability to set `size: 0` in the `terms`, `significant_terms` and `geohash_grid` aggregations for the reasons described in https://github.com/elastic/elasticsearch/issues/18838

Closes #18838
2016-06-14 13:07:02 +01:00
Boaz Leskes 7a226122e3 MasterFaultDetection can leak an exception during shutdown 2016-06-14 01:16:17 +03:00
Ryan Ernst 991c2221a1 Set next version back to alpha4 2016-06-13 09:26:45 -07:00
Simon Willnauer 7379b17e61 Revert "Make random UUIDs reproducible in tests"
This reverts commit a25b8ee1bf.
2016-06-13 11:14:30 +02:00
Christoph Büscher f20928b146 Remove redundant parseElementst() method in RescorePhase and SuggestPhase
The default implementation in SearchPhase does the same.
2016-06-13 10:20:23 +02:00
Martijn van Groningen 3b96055b23 msearch: Cap the number of searches the msearch api will concurrently execute
By default the number of searches msearch executes is capped by the number of
nodes multiplied with the default size of the search threadpool. This default can be
overwritten by using the newly added `max_concurrent_searches` parameter.

Before the msearch api would concurrently execute all searches concurrently. If many large
msearch requests would be executed this could lead to some searches being rejected
while other searches in the msearch request would succeed.

The goal of this change is to avoid this exhausting of the search TP.

Closes #17926
2016-06-13 10:13:08 +02:00
Nik Everett 387155559e Make TimeValue Writeable instead of Streamable
Writeable is better for immutable objects like TimeValue.

Switch to writeZLong which takes up less space than the original
writeLong in the majority of cases. Since we expect negative
TimeValues we shouldn't use
writeVLong.
2016-06-10 18:24:16 -04:00
Jason Tedor 86f1bedaab Rename NettyTransportChannel#close
This commit renames the NettyTransportChannel#close method to
NettyTransportChannel#release to clarify the semantics.
2016-06-10 15:26:49 -04:00
Areek Zillur df4a959d6c removed support for customs from create index request 2016-06-10 12:06:50 -03:00
Areek Zillur 62f98767eb removed redundant Fields class 2016-06-10 12:02:36 -03:00
Adrien Grand 44c653f5a8 Upgrade to lucene-6.1.0-snapshot-3a57bea. 2016-06-10 16:18:12 +02:00
Jason Tedor a25b8ee1bf Make random UUIDs reproducible in tests
Today we use a random source of UUIDs for assigning allocation IDs,
cluster IDs, etc. Yet, the source of randomness for this is not
reproducible in tests. Since allocation IDs end up as keys in hash maps,
this means allocation decisions and not reproducible in tests and this
leads to non-reproducible test failures. This commit modifies the
behavior of random UUIDs so that they are reproducible under tests. The
behavior for production code is not changed, we still use a true source
of secure randomness but under tests we just use a reproducible source
of non-secure randomness.

It is important to note that there is a test,
UUIDTests#testThreadedRandomUUID that relies on the UUIDs being truly
random. Thus, we have to modify the setup for this test to use a true
source of randomness. Thus, this is one test that will never be
reproducible but it is intentionally so.

Relates #18808
2016-06-10 10:18:06 -04:00
Ali Beyad 43e07c0c88 Better handling of an empty shard's segments_N file
When trying to restore a snapshot of an index created in a previous
version of Elasticsearch, it is possible that empty shards in the
snapshot have a segments_N file that has an unsupported Lucene version
and a missing checksum.  This leads to issues with restoring the
snapshot.  This commit handles this special case by avoiding a restore
of a shard that has no data, since there is nothing to restore anyway.

Closes #18707
2016-06-10 09:57:09 -04:00
Nik Everett d733fb689b Better error message when mapping configures null
Closes #18803
2016-06-10 09:43:18 -04:00
Yannick Welsch a2c506acd3 Fix sync flush total shards statistics (#18766) 2016-06-10 13:39:47 +02:00
Yannick Welsch 6ea89004cd Make IndicesClusterStateService unit testable (#17270)
Testability of ICSS is achieved by introducing interfaces for IndicesService, IndexService and IndexShard. These interfaces extract all relevant methods used by ICSS (which do not deal directly with store) and give the possibility to easily mock all the store behavior away in the tests (and cuts down on dependencies).
2016-06-10 12:47:41 +02:00
javanna 9cbfa984fa Merge branch 'master' into feature/http_client 2016-06-10 11:18:21 +02:00
Colin Goodheart-Smithe 1d76177510 Adds aggregation profiling (not including reduce phase)
Add Aggregation profiling initially only be for the shard phases (i.e. the reduce phase will not be profiled in this change)

This change refactors the query profiling class to extract abstract classes where it is useful for other profiler types to share code.
2016-06-10 09:02:07 +01:00
Jim Ferenczi 439b2a96e5 Add an index setting to limit the maximum number of slices allowed in a scroll request (default to 1024). 2016-06-10 09:43:32 +02:00
Daniel Mitterdorfer 7229c91289 Remove trace logging from NettyHttpRequestSizeLimitIT
With this commit we revert back to normal behavior as the
underlying issue has been fixed with #18627.
2016-06-10 07:46:04 +02:00
Nik Everett e02d9f0945 Squash a race condition in RefreshListeners
It presented as listeners never being called if you refresh at the same
time as the listener is added. It was caught rarely by
testConcurrentRefresh. mostly this is removing code and adding a comment:

```
Note that it is not safe for us to abort early if we haven't advanced the
position here because we set and read lastRefreshedLocation outside of a
synchronized block. We do that so that waiting for a refresh that has
already passed is just a volatile read but the cost is that any check
whether or not we've advanced the position will introduce a race between
adding the listener and the position check. We could work around this by
moving this assignment into the synchronized block below and double
checking lastRefreshedLocation in addOrNotify's synchronized block but
that doesn't seem worth it given that we already skip this process early
if there aren't any listeners to iterate.
```
2016-06-09 13:48:41 -04:00
Areek Zillur 41d31541a6 Allow users to override the name for the rollover index 2016-06-09 13:43:19 -04:00
gfyoung 6f222b5be1 Support flags in pattern replace char filter
Works just like pattern analyzer's flags param.

Closes #18362.
2016-06-09 12:39:23 -04:00
Areek Zillur a9f24ea2dc fail rollover request if rollover index already exists 2016-06-09 12:38:12 -04:00
Nik Everett fb52c258fd [test] Check if RefreshListeners was called immediately
Return a boolean from RefreshListeners, true if we called the listener
inline and false if we didn't, and check it in the test.
2016-06-09 12:08:36 -04:00
Areek Zillur 9027e8a719 renamed simulated mode to dry_run mode 2016-06-09 11:55:10 -04:00
javanna cf6e713d77 Merge branch 'master' into feature/http_client 2016-06-09 17:43:45 +02:00
Nik Everett bd276ef5f1 [test] Check for listener calling error
Failing to call a refresh listener is logger at WARN but that'll
cause test failure. This adds explicit assertions that there are
no errors.
2016-06-09 11:26:08 -04:00
Areek Zillur ce211119d0 Add Shrink request source parser to parse create index request body
Follow up to https://github.com/elastic/elasticsearch/pull/18732#discussion_r66407196
2016-06-09 10:41:59 -04:00
javanna 437c4f210b rename ElasticsearchResponse to Response and ElasticsearchResponseException to ResponseException 2016-06-09 14:38:32 +02:00
Areek Zillur 94a7978ef6 add documentation 2016-06-08 18:38:02 -04:00
Jason Tedor e9017f619e Improve performance of applyDeletedShards
This commit addresses a performance issue in
IndicesClusterStateService#applyDeletedShards. Namely, the current
implementation is O(number of indices * number of shards). This is
because of an outer loop over the indices and an inner loop over the
assigned shards, all to check if a shard is in the outer index. Instead,
we can group the shards by index, and then just do a map lookup for each
index.

Testing this on a single-node with 2500 indices, each with 2 shards,
creating an index before this optimization takes 0.90s and after this
optimization takes 0.19s.

Relates #18788
2016-06-08 16:08:00 -04:00
Simon Willnauer 9497b704bb [TEST] Fix NodeEnvironmentTests on Windows - use Path.resolve instead of platform dependent path seperator 2016-06-08 21:40:35 +02:00
Areek Zillur ae3eb15caa fix rest tests 2016-06-08 15:03:35 -04:00
Areek Zillur 0c6d19c40c add body support for create index request 2016-06-08 14:16:06 -04:00
Nik Everett 4b21157906 Remove setRefresh
It has been replaced with `setRefreshPolicy` which has support for
waiting until refresh with `setRefreshPolicy(WAIT_FOR)`.

Related to #1063
2016-06-08 13:50:59 -04:00
Lee Hinman 92349f70e2 Merge remote-tracking branch 'dakrone/igs-false2' 2016-06-08 10:49:20 -06:00
Lee Hinman c637fea84b Change the default of `include_global_state` from true to false for restores
This changes the default value to be false *only* for restore operations.

Resolves #18569
2016-06-08 10:48:36 -06:00
Nik Everett 5161afe5e3 Support optional ctor args in ConstructingObjectParser
You declare them like
```
static {
  PARSER.declareInt(optionalConstructorArg(), new ParseField("animal"));
}
```

Other than being optional they follow all of the rules of regular
`constructorArg()`s. Parsing an object with optional constructor args
is going to be slightly less efficient than parsing an object with
all required args if some of the optional args aren't specified because
ConstructingObjectParser isn't able to build the target before the
end of the json object.
2016-06-08 12:38:40 -04:00
Areek Zillur dec0dcc30b minor cleanup 2016-06-08 11:33:55 -04:00
Simon Willnauer bec26015b2 [TEST] add a dedicated test for empty files 2016-06-08 15:40:14 +02:00
Christoph Büscher a2372778dd Fix problem with TimeIntervalRounding on DST end
Due to an error in our current TimeIntervalRounding, two dates can
round to the same key, even when they are 1h apart when using
short interval roundings (e.g. 20m) and a time zone with DST change.

Here is an example for the CET time zone:

On 25 October 2015, 03:00:00 clocks are turned backward 1 hour to
02:00:00 local standard time. The dates
"2015-10-25T02:15:00+02:00" (1445732100000) (before DST end) and
"2015-10-25T02:15:00+01:00" (1445735700000) (after DST end)
are thus 1h apart, but currently they round to the same value
"2015-10-25T02:00:00.000+01:00" (1445734800000).

This violates an important invariant of rounding, namely that the
rounded value must be less or equal to the value that is rounded.
It also leads to wrong histogram bucket counts because documents in
[02:00:00+02:00, 02:20:00+02:00) go to the same bucket as documents
from [02:00:00+01:00, 02:20:00+01:00).

The problem happens because in TimeIntervalRounding#roundKey() we
need to perform the rounding operation in local time, but on
converting back to UTC we don't honor the original values time zone
offset. This fix changes that and adds tests both for DST start and
DST end as well as a test that demonstrates what happens to bucket
sizes when the dst change is not evently divisibly by the interval.
2016-06-08 13:05:52 +02:00
Jim Ferenczi 712c77264d Fix ut: make sure that the number of slices is bigger than 1 in the SliceBuilder tests. 2016-06-08 11:51:46 +02:00
Areek Zillur 134a4e5e52 incorporate feedback 2016-06-07 22:38:47 -04:00
Lee Hinman 762bbdbd0c Revert "Change the default of `include_global_state` from true to false."
This reverts commit 052a62250c.
2016-06-07 15:07:37 -06:00
Lee Hinman 052a62250c Change the default of `include_global_state` from true to false.
Resolves #18569
2016-06-07 15:06:20 -06:00
Nik Everett a405c2ba99 Switch QueryBuilders to new MatchPhraseQueryBuilder
It was doing deprecated things with MatchQueryBuilder.
2016-06-07 14:35:23 -04:00
Lee Hinman 32bd869b28 Merge remote-tracking branch 'dakrone/no-cluster-name-in-path' 2016-06-07 10:14:23 -06:00
Lee Hinman feb244c14a Remove cluster name from data path
Previously Elasticsearch used $DATA_DIR/$CLUSTER_NAME/nodes for the path
where data is stored, this commit changes that to be $DATA_DIR/nodes.

On startup, if the old folder structure is detected it will be used.
This behavior will be removed in Elasticsearch 6.0

Resolves #17810
2016-06-07 10:13:48 -06:00
Jim Ferenczi 43b419b230 rehash the docvalues in DocValuesSliceQuery using BitMixer.mix instead of the naive Long.hashCode. 2016-06-07 17:58:32 +02:00
Martijn van Groningen f611f1c99e ingest: Move processors from core to ingest-common module.
Folded grok processor into ingest-common module.

The rest tests have been moved to ingest-common module as well, because these tests don't run in the rest-api-spec module but in the distribution:integ-test-zip module
and adding a test plugin there felt just wrong to me. I think this is ok. I left a tiny ingest rest test behind in that tests with an empty pipeline.

Removed messy tests, these tests were already covered in the rest tests

Added ingest test plugin in test infra so that each module testing integration with ingest doesn't need write its own plugin

Moved reindex ingest tests to qa module

Closes #18490
2016-06-07 17:32:52 +02:00
trangvh c0da8e4060 Fix some typos (#18746)
* Update java-doc of SearchResponse.getProfileResults()

* Fix a trivial typo in Reference document
2016-06-07 16:41:39 +02:00
Jim Ferenczi 692c42b23a Fix ut 2016-06-07 16:29:18 +02:00
Jim Ferenczi b9030bf6fe Add the ability to partition a scroll in multiple slices.
API:

```
curl -XGET 'localhost:9200/twitter/tweet/_search?scroll=1m' -d '{
    "slice": {
        "field": "_uid", <1>
        "id": 0, <2>
        "max": 10 <3>
    },
    "query": {
        "match" : {
            "title" : "elasticsearch"
        }
    }
}
```

<1> (optional) The field name used to do the slicing (_uid by default)
<2> The id of the slice

By default the splitting is done on the shards first and then locally on each shard using the _uid field
with the following formula:
`slice(doc) = floorMod(hashCode(doc._uid), max)`
For instance if the number of shards is equal to 2 and the user requested 4 slices then the slices 0 and 2 are assigned
to the first shard and the slices 1 and 3 are assigned to the second shard.

Each scroll is independent and can be processed in parallel like any scroll request.

Closes #13494
2016-06-07 16:21:53 +02:00
Jason Tedor c3e3a6337e Use method name in bootstrap check might fork test
This commit modifies the bootstrap check invocations in the might fork
tests to use the underlying test name when setting up the logging prefix
when invoking the bootstrap checks. This is done to give clear logs in
case of failure.
2016-06-07 09:33:17 -04:00
Jason Tedor 75d3b13790 Merge pull request #18756 from jasontedor/on-out-of-memory-error
Bootstrap check for OnOutOfMemoryError and seccomp
2016-06-07 09:26:57 -04:00
Simon Willnauer c72ebba5de checkstyle have your upper L 2016-06-07 11:05:28 +02:00
Simon Willnauer 0a5e06d402 fix javadocs 2016-06-07 10:22:11 +02:00
Simon Willnauer b2c4c323e1 Allow `_shrink` to N shards if source shards is a multiple of N (#18699)
Today we allow to shrink to 1 shard but that might not be possible due to
too many document or a single shard doesn't meet the requirements for the index.
The logic can be expanded to N shards if the source index shards is a multiple of N.
This guarantees that there are not hotspots created due to different number of shards
being shrunk into one.
2016-06-07 10:06:41 +02:00
Jason Tedor acc9cea8f6 Fix compilation issue in RefreshListenersTests
This commit fixes a compilation issue in RefreshListenersTests that
arose from code being integrated into master, and then a large pull
request refactoring the handling of thread pools was later merged into
master.
2016-06-06 23:26:22 -04:00
Jason Tedor e94408c0d2 Bootstrap check for OnError and seccomp
This commit adds a bootstrap check for the JVM option OnError being in
use and seccomp being enabled. These two options are incompatible
because OnError allows the user to specify an arbitrary program to fork
when the JVM encounters an fatal error, and seccomp enables system call
filters that prevents forking.
2016-06-06 22:18:44 -04:00
Jason Tedor da74323141 Register thread pool settings
This commit refactors the handling of thread pool settings so that the
individual settings can be registered rather than registering the top
level group. With this refactoring, individual plugins must now register
their own settings for custom thread pools that they need, but a
dedicated API is provided for this in the thread pool module. This
commit also renames the prefix on the thread pool settings from
"threadpool" to "thread_pool". This enables a hard break on the settings
so that:
 - some of the settings can be given more sensible names (e.g., the max
   number of threads in a scaling thread pool is now named "max" instead
   of "size")
 - change the soft limit on the number of threads in the bulk and
   indexing thread pools to a hard limit
 - the settings names for custom plugins for thread pools can be
   prefixed (e.g., "xpack.watcher.thread_pool.size")
 - remove dynamic thread pool settings

Relates #18674
2016-06-06 22:09:12 -04:00
Areek Zillur d8d60d294e remove put method 2016-06-06 18:54:28 -04:00
Areek Zillur 1e329099f9 enhance rollover request 2016-06-06 18:50:45 -04:00
Jason Tedor 9695caa3fb Bootstrap check for OnOutOfMemoryError and seccomp
This commit adds a bootstrap check for the JVM option OnOutOfMemoryError
being in use and seccomp being enabled. These two options are
incompatible because OnOutOfMemoryError allows the user to specify an
arbitrary program to fork when the JVM encounters an
OutOfMemoryError, and seccomp enables system call filters that prevents
forking.

This commit also adds support for bootstrap checks that are always
enforced, whether or not Elasticsearch is in production mode.
2016-06-06 17:31:42 -04:00
Jared Carey 080946c915 Fix typo in template validation message
This commit addresses a typo in a template validation message in
MetaDataIndexTemplateService.java.

Relates #18754
2016-06-06 17:00:13 -04:00
Areek Zillur d96fe20e3a add named writable registry glue 2016-06-06 16:11:46 -04:00
Areek Zillur 3a2cc22aff simplify conditions and rollover request 2016-06-06 16:10:36 -04:00
Simon Willnauer d594d6c07c [TEST] ensure green before we filter allocations otherwise follow up ensureGreen() will fail 2016-06-06 18:14:05 +02:00
Nik Everett d8056c8213 Add support for waiting until a refresh occurs
This adds support for setting the refresh request parameter to
`wait_for` in the `index`, `delete`, `update`, and `bulk` APIs. When
`refresh=wait_for` is set those APIs will not return until their
results have been made visible to search by a refresh.

Also it adds a `forced_refresh` field to the response of `index`,
`delete`, `update`, and to each item in a bulk response. This will
be true for requests with `?refresh` or `?refresh=true` and will be
true for some requests (see below) with `refresh=wait_for` but ought
to otherwise always be false.

`refresh=wait_for` is implemented as a list of
`Tuple<Translog.Location, Consumer<Boolean>>`s in the new `RefreshListeners`
class that is managed by `IndexShard`. The dynamic, index scoped
`index.max_refresh_listeners` setting controls a maximum number of
listeners allowed in any shard. If more than that many listeners
accumulate in the engine then a refresh will be forced, the thread that
adds the listener will be blocked until the refresh completes, and then the
listener will be called with a `forcedRefresh` flag so it knows that it was
the "straw that broke the camel's back". These listeners are only used by
`refresh=wait_for` and that flag manifests itself as `forced_refresh` being
`true` in the response.

About half of this change comes from piping async-ness down to the appropriate
layer in a way that is compatible with the ongoing with with sequence ids.

Closes #1063

You can look up the winding story of all the commits here:
https://github.com/elastic/elasticsearch/pull/17986

Here are the commit messages in case they are intersting to you:
commit 59a753b89109828d2b8f0de05cb104fc663cf95e
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon Jun 6 10:18:23 2016 -0400

    Replace a method reference with implementing an interface

    Saves a single allocation and forces more commonality
    between the WriteResults.

commit 31f7861a85b457fb7378a6f27fa0a0c171538f68
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon Jun 6 10:07:55 2016 -0400

    Revert "Replace static method that takes consumer with delegate class that takes an interface"

    This reverts commit 777e23a6592c75db0081a53458cc760f4db69507.

commit 777e23a6592c75db0081a53458cc760f4db69507
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon Jun 6 09:29:35 2016 -0400

    Replace static method that takes consumer with delegate class that takes an interface

    Same number of allocations, much less code duplication.

commit 9b49a480ca9587a0a16ebe941662849f38289644
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon Jun 6 08:25:38 2016 -0400

    Patch from boaz

commit c2bc36524fda119fd0514415127e8901d94409c8
Author: Nik Everett <nik9000@gmail.com>
Date:   Thu Jun 2 14:46:27 2016 -0400

    Fix docs

    After updating to master we are actually testing them.

commit 03975ac056e44954eb0a371149d410dcf303e212
Author: Nik Everett <nik9000@gmail.com>
Date:   Thu Jun 2 14:20:11 2016 -0400

    Cleanup after merge from master

commit 9c9a1deb002c5bebb2a997c89fa12b3d7978e02e
Author: Nik Everett <nik9000@gmail.com>
Date:   Thu Jun 2 14:09:14 2016 -0400

    Breaking changes notes

commit 1c3e64ae06c07a85f7af80534fab88279adb30b4
Merge: 9e63ad6 f67e580
Author: Nik Everett <nik9000@gmail.com>
Date:   Thu Jun 2 14:00:05 2016 -0400

    Merge branch 'master' into block_until_refresh2

commit 9e63ad6de52d0b28f0b6d7203721baf1ebf6f56b
Author: Nik Everett <nik9000@gmail.com>
Date:   Thu Jun 2 13:21:27 2016 -0400

    Test for TransportWriteAction

commit 522ecb59d39b3c9e8df0d3b8df34b9e7aeaf0ce9
Author: Nik Everett <nik9000@gmail.com>
Date:   Thu Jun 2 10:30:18 2016 -0400

    Document deprecation

commit 0cd67b947f58867e704a1f0e66928a6fb5a11f11
Author: Nik Everett <nik9000@gmail.com>
Date:   Thu Jun 2 10:26:23 2016 -0400

    Deprecate setRefresh(boolean)

    Users should use `setRefresh(RefreshPolicy)` instead.

commit aeb1be3f2c501990b33fb1f8230d496035f498ef
Author: Nik Everett <nik9000@gmail.com>
Date:   Thu Jun 2 10:12:27 2016 -0400

    Remove checkstyle suppression

    It is fixed

commit 00d09a9caa638b6f90f4896b5502dd98d8fad56e
Author: Nik Everett <nik9000@gmail.com>
Date:   Thu Jun 2 10:08:28 2016 -0400

    Improve comment

commit 788164b898a6ee2878a273961230122b7386c3c9
Author: Nik Everett <nik9000@gmail.com>
Date:   Thu Jun 2 10:01:01 2016 -0400

    S/ReplicatedWriteResponse/WriteResponse/

    Now it lines up with WriteRequest.

commit b74cf3fe778352b140355afcaa08d3d4412d749d
Author: Nik Everett <nik9000@gmail.com>
Date:   Wed Jun 1 18:27:52 2016 -0400

    Preserve `?refresh` behavior

    `?refresh` means the same things as `?refresh=true`.

commit 30f972bdaeaaa0de6fe67746cdb8628aa86f5a8c
Author: Nik Everett <nik9000@gmail.com>
Date:   Wed Jun 1 17:39:05 2016 -0400

    Handle hanging documents

    If a document is added to the index during a refresh we weren't properly
    firing its refresh listener. This happened because the way we detect
    whether a refresh makes something visible or not is imperfect. It is
    ok because it always errs on the side of thinking that something isn't
    yet visible.

    So when a document arrives during a refresh the refresh listeners
    won't think it made it into a refresh when, often, it does. The way
    we work around this is by telling Elasticsearch that it ought to
    trigger a refresh if there are any pending refresh listeners even
    if there aren't pending documents to update. Lucene short circuits
    the refresh so it doesn't take that much effort, but the refresh
    listeners still get the signal that a refresh has come in and they
    still pick up the change and notify the listener.

    This means that the time that a listener can wait is actually slightly
    longer than the refresh interval.

commit d523b5702b60c7ba309fb0dcf3cd3a4798f11960
Author: Nik Everett <nik9000@gmail.com>
Date:   Wed Jun 1 14:34:01 2016 -0400

    Explain Integer.MAX_VALUE

commit 4ffb7c0e954343cc1c04b3d7be2ebad66d3a016b
Author: Nik Everett <nik9000@gmail.com>
Date:   Wed Jun 1 14:27:39 2016 -0400

    Fire all refresh listeners in a single thread

    Rather than queueing a runnable each.

commit 19606ec3bbe612095df45eba734c5b7eb2709c01
Author: Nik Everett <nik9000@gmail.com>
Date:   Wed Jun 1 14:09:52 2016 -0400

    Assert translog ordering

commit 6bb4e5c75e850f4a42518f06fbc955f7ec76d245
Author: Nik Everett <nik9000@gmail.com>
Date:   Wed Jun 1 13:17:44 2016 -0400

    Support null RefreshListeners in InternalEngine

    Just skip using it.

commit 74be1480d6e44af2b354ff9ea47c234d4870b6c2
Author: Nik Everett <nik9000@gmail.com>
Date:   Tue May 31 18:02:03 2016 -0400

    Move funny ShardInfo hack for bulk into bulk

    This should make it easier to understand because it is closer to where it
    matters....

commit 2b771f8dabd488e056cfdc9989608d18264ddfb0
Author: Nik Everett <nik9000@gmail.com>
Date:   Tue May 31 17:39:46 2016 -0400

    Pull listener out into an inner class with javadoc and stuff

commit 058481ad72019c0492b03a7a4ac32a48673697d3
Author: Nik Everett <nik9000@gmail.com>
Date:   Tue May 31 17:33:42 2016 -0400

    Fix javadoc links

commit d2123b1cabf29bce8ff561d4a4c1c1d5b42bccad
Author: Nik Everett <nik9000@gmail.com>
Date:   Tue May 31 17:28:09 2016 -0400

    Make more stuff final

commit 8453fc4f7850f6a02fb5971c17a942a3e3fd9f7b
Author: Nik Everett <nik9000@gmail.com>
Date:   Tue May 31 17:26:48 2016 -0400

    Javadoc

commit fb16d2fc7016c1e8e1621d481e8781c7ef43326c
Author: Nik Everett <nik9000@gmail.com>
Date:   Tue May 31 16:14:48 2016 -0400

    Rewrite refresh docs

commit 5797d1b1c4d233c0db918c0d08c21731ddccd05e
Author: Nik Everett <nik9000@gmail.com>
Date:   Tue May 31 15:02:34 2016 -0400

    Fix forced_refresh flag

    It wasn't being set.

commit 43ce50a1de250a9e073a2ca6cbf55c1b4c74b11b
Author: Nik Everett <nik9000@gmail.com>
Date:   Tue May 31 14:02:56 2016 -0400

    Delay translog sync and flush until after refresh

    The sync might have occurred for us during the refresh so we
    have less work to do. Maybe.

commit bb2739202e084703baf02cfa58f09517598cf14e
Author: Nik Everett <nik9000@gmail.com>
Date:   Tue May 31 13:08:08 2016 -0400

    Remove duplication in WritePrimaryResult and WriteReplicaResult

commit 2f579f89b4867a880396f2e7fcffc508449ff2de
Author: Nik Everett <nik9000@gmail.com>
Date:   Tue May 31 12:19:05 2016 -0400

    Clean up registration of RefreshListeners

commit 87ab6e60ca5ba945bf0fba84784b2bbe53506abf
Author: Nik Everett <nik9000@gmail.com>
Date:   Tue May 31 11:28:30 2016 -0400

    Shorten lock time in RefreshListeners

    Also use null to represent no listeners rather than an empty list.
    This saves allocating a new ArrayList every refresh cycle on every
    index.

commit 0d49d9c5720dadfb67da3fa760397bf6d874601c
Author: Nik Everett <nik9000@gmail.com>
Date:   Tue May 24 10:46:18 2016 -0400

    Flip relationship between RefreshListeners and Engine

    Now RefreshListeners comes to Engine from EngineConfig.

commit b2704b8a39382953f8f91a9743e894ee289f7514
Author: Nik Everett <nik9000@gmail.com>
Date:   Tue May 24 09:37:58 2016 -0400

    Remove unused imports

    Maybe I added them?

commit 04343a22647f19304d9dc716b3fac9b183227f63
Author: Nik Everett <nik9000@gmail.com>
Date:   Tue May 24 09:37:52 2016 -0400

    Javadoc

commit da1e765678890a02d61d8a29aa433274beb5e00c
Author: Nik Everett <nik9000@gmail.com>
Date:   Tue May 24 09:26:35 2016 -0400

    Reply with non-null

    Also move the fsync and flush to before the refresh listener stuff.

commit 5d8eecd0d904b497844b4c81c46477bd6178ed3a
Author: Nik Everett <nik9000@gmail.com>
Date:   Tue May 24 08:58:47 2016 -0400

    Remove funky synchronization in AsyncReplicaAction

commit 1ec71eea0f4e1228ae1497d982307be818ef4b65
Author: Nik Everett <nik9000@gmail.com>
Date:   Tue May 24 08:01:14 2016 -0400

    s/LinkedTransferQueue/ArrayList/

commit 7da36a4ceed2ccf7955138c3b005237fa41efcb4
Author: Nik Everett <nik9000@gmail.com>
Date:   Tue May 24 07:46:38 2016 -0400

    More cleanup for RefreshListeners

commit 957e9b77007c32ee75dde152c6622bab065d5993
Author: Nik Everett <nik9000@gmail.com>
Date:   Tue May 24 07:34:13 2016 -0400

    /Consumer<Runnable>/Executor/

commit 4d8bf5d4a70dcc56150c8d8d14165cd23d308b3c
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon May 23 22:20:42 2016 -0400

    explain

commit 15d948a348089bb2937eec5ac4e96f3ec67dbe32
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon May 23 22:17:59 2016 -0400

    Better....

commit dc28951d02973fc03b4d51913b5f96de14b75607
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon May 23 21:09:20 2016 -0400

    Javadocs and compromises

commit 8eebaa89c0a1ee74982fbe0d56d1485ca2ae09db
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon May 23 20:52:49 2016 -0400

    Take boaz's changes to their logic conclusion and unbreak important stuff like bulk

commit 7056b96ea412f275005b93e3570bcff895859ed5
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon May 23 15:49:32 2016 -0400

    Patch from boaz

commit 87be7eaed09a274cc6a99d1a3da81d2d7bf9dd64
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon May 23 15:49:13 2016 -0400

    Revert "Move async parts of replica operation outside of the lock"

    This reverts commit 13807ad10b6f5ecd39f98c9f20874f9f352c5bc2.

commit 13807ad10b6f5ecd39f98c9f20874f9f352c5bc2
Author: Nik Everett <nik9000@gmail.com>
Date:   Fri May 20 22:53:15 2016 -0400

    Move async parts of replica operation outside of the lock

commit b8cadcef565908b276484f7f5f988fd58b38d8b6
Author: Nik Everett <nik9000@gmail.com>
Date:   Fri May 20 16:17:20 2016 -0400

    Docs

commit 91149e0580233bf79c2273b419fe9374ca746648
Author: Nik Everett <nik9000@gmail.com>
Date:   Fri May 20 15:17:40 2016 -0400

    Finally!

commit 1ff50c2faf56665d221f00a18d9ac88745904bf5
Author: Nik Everett <nik9000@gmail.com>
Date:   Fri May 20 15:01:53 2016 -0400

    Remove Translog#lastWriteLocation

    I wasn't being careful enough with locks so it wasn't right anyway.
    Instead this builds a synthetic Tranlog.Location when you call
    getWriteLocation with much more relaxed equality guarantees. Rather
    than being equal to the last Translog.Location returned it is
    simply guaranteed to be greater than the last translog returned
    and less than the next.

commit 55596ea68b5484490c3637fbad0d95564236478b
Author: Nik Everett <nik9000@gmail.com>
Date:   Fri May 20 14:40:06 2016 -0400

    Remove listener from shardOperationOnPrimary

    Create instead asyncShardOperationOnPrimary which is called after
    all of the replica operations are started to handle any async
    operations.

commit 3322e26211bf681b37132274ee158ae330afc28b
Author: Nik Everett <nik9000@gmail.com>
Date:   Tue May 17 17:20:02 2016 -0400

    Increase default maximum number of listeners to 1000

commit 88171a8322a424e624d48960fb4c98dd43e4d671
Author: Nik Everett <nik9000@gmail.com>
Date:   Tue May 17 16:40:57 2016 -0400

    Rename test

commit 179c27c4f829f2c6ded65967652cf85adaf2ae52
Author: Nik Everett <nik9000@gmail.com>
Date:   Tue May 17 16:35:27 2016 -0400

    Move refresh listeners into their own class

    They still live at the IndexShard level but they live on their
    own in RefreshListeners which interacts with IndexShard using a
    couple of callbacks and a registration method. This lets us test
    the listeners without standing up an entire IndexShard. We still
    test the listeners against an InternalEngine, because the interplay
    between InternalEngine, Translog, and RefreshListeners is complex
    and important to get right.

commit d8926d5fc1d24b4da8ccff7e0f0907b98c583c41
Author: Nik Everett <nik9000@gmail.com>
Date:   Tue May 17 11:02:38 2016 -0400

    Move refresh listeners into IndexShard

commit df91cde398eb720143a85a8c6fa19bdc3a74e07d
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon May 16 16:01:03 2016 -0400

    unused import

commit 066da45b08148b266e4173166662fc1b3f66ed53
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon May 16 15:54:11 2016 -0400

    Remove RefreshListener interface

    Just pass a Translog.Location and a Consumer<Boolean> when registering.

commit b971d6d3301c7522b2e7eb90d5d8dd96a77fa625
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon May 16 14:41:06 2016 -0400

    Docs for setForcedRefresh

commit 6c43be821eaf61141d3ec520f988aad3a96a3941
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon May 16 14:34:39 2016 -0400

    Rename refresh setter and getter

commit e61b7391f91263a4c4d6107bfbc2a828bbcc805c
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon Apr 25 22:48:09 2016 -0400

    Trigger listeners even when there is no refresh

    Each refresh gives us an opportunity to pick up any listeners we may
    have left behind.

commit 0c9b0477085c021f503db775640d25668e02f635
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon Apr 25 20:30:06 2016 -0400

    REST

commit 8250343240de7e63118c663a230a7a314807a754
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon Apr 25 19:34:22 2016 -0400

    Switch to estimated count

    We don't need a linear time count of the number of listeners - a volatile
    variable is good enough to guess. It probably undercounts more than it
    overcounts but it isn't a huge problem.

commit bd531167fe54f1bde6f6d4ddb0a8de5a7bcc18a2
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon Apr 25 18:21:02 2016 -0400

    Don't try and set forced refresh on bulk items without a response

    NullPointerExceptions are bad. If the entire request fails then the user
    has worse problems then "did these force a refresh".

commit bcfded11515af5e0b3c3e36f3c2f73f5cd26512e
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon Apr 25 18:14:20 2016 -0400

    Replace LinkedList and synchronized with LinkedTransferQueue

commit 8a80cc70a76375a7593745884cb987535b37ca80
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon Apr 25 17:38:24 2016 -0400

    Support for update

commit 1f36966742f851b7328015151ef6fc8f95299af2
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon Apr 25 15:46:06 2016 -0400

    Cleanup translog tests

commit 8d121bf35eb265b8a0aee9710afeb1b054a113d4
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon Apr 25 15:40:53 2016 -0400

    Cleanup listener implementation

    Much more testing too!

commit 2058f4a808762c4588309f21b13b677245832f2c
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon Apr 25 11:45:55 2016 -0400

    Pass back information about whether we refreshed

commit e445cb0cb91ebdbcfdbf566696edb2bf1c84a882
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon Apr 25 11:03:31 2016 -0400

    Javadoc

commit 611cbeeaeb458f4b428bfc43a1ee6652adf4baff
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon Apr 25 11:01:40 2016 -0400

    Move ReplicationResponse

    now it is in the same package as its request

commit 9919758b644fd73895fb88cd6a4909a8387eb2e2
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon Apr 25 11:00:14 2016 -0400

    Oh boy that wasn't working

commit 247cb483c4459dea8e95e0e3bd2e4bf8d452c598
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon Apr 25 10:29:37 2016 -0400

    Basic block_until_refresh exposed to java client

    and basic "is it plugged in" style tests.

commit 46c855c9971cb2b748206d2afa6a2d88724be3ba
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon Apr 25 10:11:10 2016 -0400

    Move test to own class

commit a5ffd892d0a352ae7e9757f2640fc2a1fa656bf2
Author: Nik Everett <nik9000@gmail.com>
Date:   Mon Apr 25 07:44:25 2016 -0400

    WIP

commit 213bebb6ece11b85d17e44af9a54fc2e5e332d39
Author: Nik Everett <nik9000@gmail.com>
Date:   Fri Apr 22 21:35:52 2016 -0400

    Add refresh listeners

commit a2bc7f30e6d4857a1224ef5a89909b36c8f33731
Author: Nik Everett <nik9000@gmail.com>
Date:   Fri Apr 22 21:11:55 2016 -0400

    Return last written location from refresh

commit 85033a87551da89f36a23d4dfd5016db218e08ee
Author: Nik Everett <nik9000@gmail.com>
Date:   Fri Apr 22 20:28:21 2016 -0400

    Never reply to replica actions while you have the operation lock

    This last thing was causing periodic test failures because we were
    replying while we had the operation lock. Now, we probably could get
    away with that in most cases but the tests don't like it and it isn't
    a good idea to do network io while you have a lock anyway. So this
    prevents it.

commit 1f25cf35e796835b3827b8a4110e09e5de61784c
Author: Nik Everett <nik9000@gmail.com>
Date:   Fri Apr 22 19:56:18 2016 -0400

    Cleanup

commit 52c5f7c3f04710901f503334239a611c0e21c85a
Author: Nik Everett <nik9000@gmail.com>
Date:   Fri Apr 22 19:33:00 2016 -0400

    Add a listener to shard operations

commit 5b142dc331214c8eef90587144f4b3f959f9eced
Author: Nik Everett <nik9000@gmail.com>
Date:   Fri Apr 22 18:03:52 2016 -0400

    Cleanup

commit 3d22b2d7ceb473db339259452a7c4f117ce86069
Author: Nik Everett <nik9000@gmail.com>
Date:   Fri Apr 22 17:59:55 2016 -0400

    Push the listener into shardOperationOnPrimary

commit 34b378943b8185451acf6350f661c0ad33b5836d
Author: Nik Everett <nik9000@gmail.com>
Date:   Fri Apr 22 17:48:47 2016 -0400

    Doc

commit b42b8da968d42cc7414020c7b199606a5dcce50a
Author: Nik Everett <nik9000@gmail.com>
Date:   Fri Apr 22 17:45:40 2016 -0400

    Don't finish early if the primary finishes early

    We use a "fake" pending shard that we resolve when the replicas have
    all started.

commit 0fc045b56e1e02a48c30383ac50a281d5af7e0b6
Author: Nik Everett <nik9000@gmail.com>
Date:   Fri Apr 22 17:30:06 2016 -0400

    Make performOnPrimary asyncS

    Instead of returning Tuple<Response, ReplicaRequest> it returns
    ReplicaRequest and takes a ActionListener<Response> as an argument.
    We call the listener immediately to preserve backwards compatibility
    for now.

commit 80119b9a26ede96a865af45904c3ac69d5b19b59
Author: Nik Everett <nik9000@gmail.com>
Date:   Fri Apr 22 16:51:53 2016 -0400

    Factor out common code in shardOperationOnPrimary

commit 0642083676702618f900fa842c08802a04c1a53e
Author: Nik Everett <nik9000@gmail.com>
Date:   Fri Apr 22 16:32:29 2016 -0400

    Factor out common code from shardOperationOnReplica

commit 8bdc415fedaaa9f2d0c555590a13ec4699a7c3f7
Author: Nik Everett <nik9000@gmail.com>
Date:   Fri Apr 22 16:23:28 2016 -0400

    Create ReplicatedMutationRequest

    Superclass for index, delete, and bulkShard requests.

commit 0f8fa846a2822c4293df32fed18c9b99660b39ff
Author: Nik Everett <nik9000@gmail.com>
Date:   Fri Apr 22 16:10:30 2016 -0400

    Create TransportReplicatedMutationAction

    It is the superclass of replication actions that mutate data: index, delete,
    and shardBulk. shardFlush and shardRefresh are replication actions but they
    do not extend TransportReplicatedMutationAction because they don't change
    the data, only shuffle it around.
2016-06-06 11:37:53 -04:00
Jason Tedor 200d76e6f0 Throw exception if using a closed transport client
Today when attempting to use a closed transport client, an exception
saying that no nodes are available is thrown. This is because when a
transport client is closed, its internal list of nodes is cleared. But
this exception is puzzling and can be made clearer. This commit changes
the behavior so that attempting to execute a request using a closed
transport client throws an illegal state exception.

Relates #18722
2016-06-06 11:27:26 -04:00
Yannick Welsch e3e8f10103 [TEST] fix assertion that index was fully deleted 2016-06-06 15:45:57 +02:00
Yannick Welsch 0a8afa2e72 Add back pending deletes (#18698)
Triggering the pending deletes logic was accidentally removed in the clean up PR #18602.
2016-06-06 15:14:09 +02:00
Tanguy Leroux 4ca04d6f6c Close SearchContext if query rewrite failed
If a query failed to be rewritten and throws an exception, the SearchContext is not properly closed, skewing the ref count on the underlying Store.
2016-06-06 09:40:56 +02:00
javanna b891c46657 [TEST] remove status matcher and hasStatus assertion
All it does is checking the status code of a response, which can be done with a single line in each test
2016-06-03 23:25:17 +02:00
Jason Tedor be0036542c More complete exception message in settings tests
This commit adds additional details to the exception message assertions
in the YAML settings loader tests.
2016-06-03 16:18:46 -04:00
Areek Zillur a30437f250 adds _rollover api 2016-06-03 16:09:24 -04:00
Jason Tedor bbd5f26d45 Merge branch 'master' into rjernst-placeholder
* master: (911 commits)
  [TEST] wait for yellow after setup doc tests (#18726)
  Fix recovery throttling to properly handle relocating non-primary shards (#18701)
  Fix merge stats rendering in RestIndicesAction (#18720)
  [TEST] mute RandomAllocationDeciderTests.testRandomDecisions
  Reworked docs for index-shrink API (#18705)
  Improve painless compile-time exceptions
  Adds UUIDs to snapshots
  Add test rethrottle test case for delete-by-query
  Do not start scheduled pings until transport start
  Adressing review comments
  Only filter intial recovery (post API) when shrinking an index (#18661)
  Add tests to check that toQuery() doesn't return null
  Removing handling of null lucene query where we catch this at parse time
  Handle empty query bodies at parse time and remove EmptyQueryBuilder
  Mute failing assertions in IndexWithShadowReplicasIT until fix
  Remove allow running as root
  Add upgrade-not-supported warning to alpha release notes
  remove unrecognized javadoc tag from matrix aggregation module
  set ValuesSourceConfig fields as private
  Adding MultiValuesSource support classes and documentation to matrix stats agg module
  ...
2016-06-03 13:32:03 -04:00
javanna f17f0f9247 rename ElasticsearchResponse#getFirstHeader to getHeader 2016-06-03 18:28:31 +02:00
javanna c48b7a7a97 [TEST] create standard RestClient at first request and reuse it
A RestClient instance is now created whenever EsIntegTestCase#getRestClient is invoked for the first time. It is then kept until the cluster is cleared (depending on the cluster scope of the test).

Renamed other two restClient methods to createRestClient, as that instance needs to be closed and managed in the tests.
2016-06-03 18:21:19 +02:00
javanna 23a94bb974 [TEST] create standard RestClient at first request and reuse it
A RestClient instance is now created whenever EsIntegTestCase#getRestClient is invoked for the first time. It is then kept until the cluster is cleared (depending on the cluster scope of the test).

Renamed other two restClient methods to createRestClient, as that instance needs to be closed and managed in the tests.
2016-06-03 18:00:54 +02:00
javanna 4fa824f891 [TEST] add a lot of forgotten try with resources to wrap ElasticsearchResponses 2016-06-03 16:42:07 +02:00
javanna eae914ae8e Replace rest test client with low level RestClient
We still have a wrapper called RestTestClient that is very specific to Rest tests, as well as RestTestResponse etc. but all the low level bits around http connections etc. are now handled by RestClient.
2016-06-03 16:01:07 +02:00
javanna 325b723930 [TEST] add rest client test dependency and replace usage of HttpRequestBuilder with RestClient in integration tests 2016-06-03 16:01:07 +02:00
Yannick Welsch 24a7b7224b Fix recovery throttling to properly handle relocating non-primary shards (#18701)
Relocation of non-primary shards is realized by recovering from the primary shard. Recovery throttling wrongly equates non-primary relocation as recovering a shard from the non-primary relocation source, however.

Closes #18640
2016-06-03 14:11:34 +02:00
Simon Willnauer 6c28235b03 Fix merge stats rendering in RestIndicesAction (#18720)
give the table description:
```
        table.addCell("merges.total", "sibling:pri;alias:mt,mergesTotal;default:false;text-align:right;desc:number of completed merge ops");
        table.addCell("pri.merges.total", "default:false;text-align:right;desc:number of completed merge ops");

        table.addCell("merges.total_docs", "sibling:pri;alias:mtd,mergesTotalDocs;default:false;text-align:right;desc:docs merged");
        table.addCell("pri.merges.total_docs", "default:false;text-align:right;desc:docs merged");

        table.addCell("merges.total_size", "sibling:pri;alias:mts,mergesTotalSize;default:false;text-align:right;desc:size merged");
        table.addCell("pri.merges.total_size", "default:false;text-align:right;desc:size merged");
```
this is how it should be.
2016-06-03 13:42:09 +02:00
Britta Weber 78574d248c [TEST] mute RandomAllocationDeciderTests.testRandomDecisions
we have a pr already: https://github.com/elastic/elasticsearch/pull/18701
2016-06-03 11:37:22 +02:00
Ali Beyad b720216395 Adds UUIDs to snapshots
This commit adds a UUID for each snapshot, in addition to the already
existing repository and snapshot name. The addition of UUIDs will enable
more robust handling of the deletion of previous snapshots and lingering
files from partially failed delete operations, on top of being able to
uniquely track each snapshot.

Closes #18228
Relates #18156
2016-06-02 17:01:48 -04:00
Jason Tedor f67e5807ca Do not start scheduled pings until transport start
Today, scheduled pings in NettyTransport can start before the transport
is started. Instead, these pings should not be scheduled until after the
transport is started. This commit modifies NettyTransport so that this
is the case.

Relates #18702
2016-06-02 13:12:21 -04:00
Christoph Büscher b6f26c0c37 Merge pull request #17624
Handle empty query bodies at parse time and remove EmptyQueryBuilder.
2016-06-02 18:40:36 +02:00
Christoph Büscher 9067407cdd Adressing review comments 2016-06-02 16:19:23 +02:00
Simon Willnauer 22dfc41521 Only filter intial recovery (post API) when shrinking an index (#18661)
Today we use `index.routing.allocation.include._id` to filter the allocation
for the shrink target index. That has the sideeffect that the user has to
delete that setting / change it once the primary has been recovered (shrink is done)
This PR adds a dedicated filter that can only be set internally that only filters
allocation for unassigned shards.
2016-06-02 15:38:51 +02:00
Christoph Büscher cb145aec68 Removing handling of null lucene query where we catch this at parse time 2016-06-02 11:25:56 +02:00
Christoph Büscher 359f45988f Handle empty query bodies at parse time and remove EmptyQueryBuilder
Currently we support empty query clauses like the filter in

"constant_score" : {  "filter" : { } }

How these clauses are handled depends on the surrounding query.
They later are either ignored, converted to match all or no documents or
passed up further in the query hierarchy. During parsing these claues are
currently represented as EmptyQueryBuilders. When not handled anywhere else,
these special cases need to be checked for on the shard when building the
lucene query.

This is trappy, so this PR changes the parsing of compound queries. Instead
of returning QueryBuilder, the core query parsing method
QueryShardContext#parseInnerQueryBuilder() now return an Optional which can
be empty in the case of empty query clauses. This has the advantage of forcing
callers to deal with this sooner or later. When encountering empty Optionals,
compound query builders now have the choice to ignore them, pass them on or
rewrite to a different query, depending on context.
2016-06-02 11:25:56 +02:00
Yannick Welsch b2724c0d08 Mute failing assertions in IndexWithShadowReplicasIT until fix 2016-06-02 11:23:44 +02:00
Jason Tedor 3ccd59592a Remove allow running as root
This commit removes the escape hatch for running Elasticsearch as root.

Relates #18694
2016-06-02 05:03:28 -04:00
Nicholas Knize 54575e55ca set ValuesSourceConfig fields as private 2016-06-01 16:39:42 -05:00
Nicholas Knize 90b8f5d0d8 Adding MultiValuesSource support classes and documentation to matrix stats agg module 2016-06-01 16:39:42 -05:00
Jason Tedor 8e2a7d0fe1 Rename boostrap.mlockall to bootstrap.memory_lock
The setting bootstrap.mlockall is useful on both POSIX-like systems
(POSIX mlockall) and Windows (Win32 VirtualLock). But mlockall is really
a POSIX only thing so the name should not be tied POSIX. This commit
renames the setting to "bootstrap.memory_lock".

Relates #18669
2016-06-01 16:25:51 -04:00
Yannick Welsch 21f0d7da1d [TEST] Increase timeout to wait on folder deletion in IndexWithShadowReplicasIT 2016-06-01 18:47:25 +02:00
Yannick Welsch c20bf5d747 [TEST] Fix tests that rely on assumption that data dirs are removed after index deletion (#18681)
Relates to #18602
2016-06-01 17:02:09 +02:00
Yannick Welsch bdd1d0703d Acknowledge index deletion requests based on standard cluster state acknowledgment (#18602)
Index deletion requests currently use a custom acknowledgement mechanism that wait for the data nodes to actually delete the data before acknowledging the request to the client. This was initially put into place as a new index with same name could only be created if the old index was wiped as we used the index name as data folder on the data nodes. With PR #16442, we now use the index uuid as folder name which avoids collision between indices that are named the same (deleted and recreated). This allows us to get rid of the custom acknowledgment mechanism altogether and rely on the standard cluster state-based acknowledgment instead.

Closes #18558
2016-06-01 15:22:55 +02:00
Adrien Grand 3b8c9d87b4 Fix StoreRecoveryTests after 6.0.1 upgrade. 2016-06-01 10:54:51 +02:00
Jun Ohtani 871aaa102f Merge branch 'fix/validate_index_template' 2016-06-01 17:49:01 +09:00
Adrien Grand d182e171a4 Upgrade to Lucene 6.0.1. 2016-06-01 10:31:10 +02:00
Martijn van Groningen 766789b0f0 ingest: added `ignore_failure` option to all processors
If this option is enabled on a processor it silently catches any processor related failure and continues executing the rest of the pipeline.

 Closes #18493
2016-06-01 10:29:12 +02:00
Simon Willnauer 966bfc3f2d Throw IllegalStateException when handshake fails due to version or cluster mismatch (#18676)
We do throw ConnectTransportException which is logged in trace level hiding a potentially
important information  when an old or wrong node wants to connect. We should throw ISE and
log as warn.
2016-06-01 10:28:35 +02:00
Adrien Grand a78c7d9911 AggregatorBuilder and PipelineAggregatorBuilder do not need generics. #18368
Similar reasoning as #18133 but for the aggs API. One important change is that
I moved the base PipelineAggregatorBuilder class to the o.e.s.aggregations
package instead of o.e.s.aggregations.pipeline so that the create method does
not need to be public.
2016-06-01 10:19:30 +02:00
Simon Willnauer 88800e8e47 Move PageCacheRecycler into BigArrays (#18666)
PageCacheRecycler is really just an implementation detail of
BigArrays. There is no need to leak this class anywhere outside of it.
2016-06-01 09:43:11 +02:00
Jun Ohtani fd76291131 Index Template: parse and validate mappings in template when template puts
Share applying template with MetaDataCreateIndexService and MetaDataIndexTemplateService
Add some unit test
Collapse addMappingsToMapperService and move it to MapperService
Extract validateTemplate method
use expectThrows in testcase
Add TODO comment

Closes #2415
2016-06-01 16:41:48 +09:00
Ali Beyad f12b10c48a Make cluster health classes immutable and have them implement Writeable
instead of Streamable

Closes #18673
2016-05-31 23:04:13 -04:00
Ali Beyad dac322a32d Fix javadoc that stated a throws clause that didn't exist. 2016-05-31 19:56:55 -04:00
Ali Beyad 0efac76f01 Clarify the semantics of the BlobContainer interface
This commit clarifies the behavior that must be adhered to by any
implementors of the BlobContainer interface.  This is done through
expanded Javadocs.

Closes #18157
Closes #15580
2016-05-31 19:22:55 -04:00
Jason Tedor e21d8b31f1 Remove thread pool from page cache recycler
The page cache recycler has a dependency on thread pool that was there
for historical reasons but is no longer needed. This commit removes this
now unneeded dependency.

Relates #18664
2016-05-31 14:51:58 -04:00
Yannick Welsch ad093a78c5 Fix ConcurrentModificationException in ReplicaShardAllocator 2016-05-31 19:42:17 +02:00
Clinton Gormley 7df5fcd4b6 Changed version back to 5.0.0 2016-05-31 18:34:54 +02:00
Robert Muir 0373c62a57 Merge pull request #18658 from rmuir/jodaTime
improve date api for expressions/painless fields
2016-05-31 12:33:22 -04:00
Yannick Welsch 58a103aca2 Remove custom iterators from RoutingNodes (#18615)
Contains a number of cleanups related to recent changes in RoutingNodes:

- PR #17821 (Immutable ShardRouting) changed RoutingNode to use a map indexed by ShardId to manage ShardRouting elements. This means that we can directly select the right ShardRouting without iterating over all elements. This lets us get rid of RoutingNodeIterator and all kind of iterations all over the place.
- Second cleanup is an extension of #18390 (Expose cluster state before reroute in RoutingAllocation instead of RoutingNodes). We should not reexpose RoutingTable in RoutingNodes and only use it in the constructor. This makes it clear that the RoutingTable is only used to construct the RoutingNodes and can diverge from it afterwards (only RoutingNodes is mutable).
- Remove AllocationService.applyNewNodes() (that is already done as part of construction of RoutingNodes)
2016-05-31 16:37:58 +02:00
Simon Willnauer 82645563bf Estimate shard size for shrinked indices (#18659)
When we shrink an index we can estimate the shards size for the primary
from the source index. This is important for allocation decisions since we
should try out best to ensure we have enough space on the node we shrink the
index.
2016-05-31 16:33:38 +02:00
Christoph Büscher 740bdc8d99 Treat zero token in `common` terms query as MatchNoDocsQuery
Currently we return `null` when the query in a common terms query has
zero tokens after analysis. It would be better if query builders
`toQuery()` would never return null and return a meaningful lucene
query instead. Since an ExtendedCommonTermsQuery with no terms gets
rewritten to a MatchNoDocsQuery later, it is enough to leave out the
check for zero tokens.
2016-05-31 16:28:03 +02:00
Michael McCandless 8f0109c2a5 Merge pull request #18651 from mikemccand/remove_iw_max_memory_stat
Remove index_writer_max_memory stat from segment stats
2016-05-31 09:58:55 -04:00
Robert Muir 2d1eb89aef improve date api for expressions/painless fields 2016-05-31 09:32:33 -04:00
Adrien Grand adf4712164 Make ip fields backward-compatible at query time. #18593
The fact that ip fields used a different doc values representation in 2.x causes
issues when querying 2.x and 5.0 indices in the same request. This changes 2.x
doc values on ip fields/2.x to be hidden behind binary doc values that use the
same encoding as 5.0. This way the coordinating node will be able to merge shard
responses that have different major versions.

One known issue is that this makes sorting/aggregating slower on ip fields for
indices that have been generated with elasticsearch 2.x.
2016-05-31 14:50:00 +02:00
Mike McCandless 5c525e6606 Remove index_writer_max_memory stat from segment stats 2016-05-31 06:29:29 -04:00
Simon Willnauer 867d35c084 Never trip circuit breaker in liveness request (#18627)
We don't want transport clients to disconnect due to circuit breakers
preventing the liveness request from executing.

Relates to #17951
2016-05-31 11:31:52 +02:00
Simon Willnauer 502a775a7c Add primitive to shrink an index into a single shard (#18270)
This adds a low level primitive operations to shrink an existing
index into a new index with a single shard. This primitive expects
all shards of the source index to allocated on a single node. Once the target index is initializing on the shrink node it takes a snapshot of the source index shards and copies all files into the target indices data folder. An [optimization](https://issues.apache.org/jira/browse/LUCENE-7300) coming in Lucene 6.1 will also allow for optional constant time copy if hard-links are supported by the filesystem. All mappings are merged into the new indexes metadata once the snapshots have been taken on the merge node.

To shrink an existing index all shards must be moved to a single node (one instance of each shard) and the index must be read-only:

```BASH
$ curl -XPUT 'http://localhost:9200/logs/_settings' -d '{
    "settings" : {
        "index.routing.allocation.require._name" : "shrink_node_name",
        "index.blocks.write" : true 
    }
}
```
once all shards are started on the shrink node. the new index can be created via:

```BASH
$ curl -XPUT 'http://localhost:9200/logs/_shrink/logs_single_shard' -d '{
    "settings" : {
        "index.codec" : "best_compression",
        "index.number_of_replicas" : 1
    }
}'
```

This API will perform all needed check before the new index is created and selects the shrink node based on the allocation of the source index. This call returns immediately, to monitor shrink progress the recovery API should be used since all copy operations are reflected in the recovery API with byte copy progress etc.

The shrink operation does not modify the source index, if a shrink operation should
be canceled or if the shrink failed, the target index can simply be deleted and
all resources are released.
2016-05-31 10:41:44 +02:00
Ryan Ernst c911f4d951 Merge pull request #18646 from rjernst/jarhell_evil_tests
Tests: Remove unnecessary evil jarhell tests
2016-05-31 00:08:36 -07:00
Adrien Grand 9570feaf8d Process dynamic templates in order. #18638
When calling `findTemplateBuilder(context, currentFieldName, "text", null)`,
elasticsearch ignores all templates that have a `match_mapping_type` set since
no dynamic type is provided (the last parameter, which is null in that case).
So this should only be called _last_. Otherwise, if a path-based template
matches, it will have precedence over all type-based templates.

Closes #18625
2016-05-31 09:00:10 +02:00