Commit Graph

4723 Commits

Author SHA1 Message Date
Tanguy Leroux 662c34b888 Clean up some log messages 2016-04-01 14:44:50 +02:00
Boaz Leskes 283eff13aa Merge pull request #17457 from s1monw/make_translog_config_immutable
Make TranslogConfig immutable and pass TranslogGeneration as a ctor arg to Translog

This mutable state is confusing and is easily missed. By default this is null and
wipes all translog. This commit makes the TranslogGeneration mandatory on the Translog
constructor and removes the mutalbe state.
2016-04-01 12:18:28 +02:00
Boaz Leskes 0b12cab07a Cluster Health should run on applied states, even if waitFor=0 #17440
We already protect against making decisions based on an inflight cluster state if someone asks for a waitFor rule (like wait for green). We should do the same for normal health checks as well (unless timeout is set to 0) as it be trappy to debug failures when health says the cluster is in a certain state, but that state wasn't applied yet.

Closes #17440
2016-04-01 11:14:17 +02:00
Martijn van Groningen 9ebc0c8f47 ingest: make concrete processor impl final, like all other processor concrete impls. 2016-04-01 11:05:16 +02:00
Christoph Büscher 7a1b06ce0b Improve some test method comments. 2016-04-01 11:04:56 +02:00
Adrien Grand 4c4bbb3e45 Replace FieldStatsProvider with a method on MappedFieldType. #17334
FieldStatsProvider had to perform instanceof calls to properly handle dates or
ip addresses. By moving the logic to MappedFieldType, each field type can check
whether all values are within bounds its way.

Note that this commit only keeps rewriting support for dates, which are the only
field for which the rewriting mechanism is likely to help (because of time-based
indices).
2016-04-01 10:28:58 +02:00
Martijn van Groningen 11fdd2608f cleanup test 2016-04-01 09:45:28 +02:00
Adrien Grand 8afc0f1748 Remove MathUtils. #17454
It has a single method: mod, which can be replaced with Math.floorMod since
we always coll it with a positive divisor.
2016-04-01 08:31:31 +02:00
Christoph Büscher 1a697a1ae6 Addressing review comments 2016-03-31 21:46:17 +02:00
Simon Willnauer 7f8235b004 fix javadocs 2016-03-31 21:33:01 +02:00
Simon Willnauer 23ea59b06f Make TranslogConfig immutable and pass TranslogGeneration as a ctor arg to Translog
This mutable state is confusing and is easily missed. By default this is null and
wipes all translog. This commit makes the TranslogGeneration mandatory on the Translog
constructor and removes the mutalbe state.
2016-03-31 21:24:48 +02:00
Simon Willnauer baa2d51e59 Merge pull request #17422 from s1monw/recovery_mem_buffer_access
Move translog recover outside of the engine

We changed the way we manage engine memory buffers to an
open model where each shard can essentially has infinite memory.
The indexing memory controller is responsible for moving memory to disk
when it's needed. Yet, this doesn't work today when we recover from store/translog
since the engine is not fully initialized such that IMC has no access to the engine,
neither to it's memory buffer nor can it move data to disk.

The biggest issue here is that translog recovery happends inside the Engine constructor
which is problematic by itself since it might take minutes and uses a not yet fully
initialzied engine to perform write operations on.

This change detaches the translog recovery and makes it the responsibility of the caller
to run it once the engine is fully constructed or skip it if not necessary.
2016-03-31 21:03:00 +02:00
Simon Willnauer 6fb588156c Remove redundant commit - #openTranslog() already commits in that case 2016-03-31 20:46:28 +02:00
Nik Everett 75a9899813 Start to rework query registration
Changes QueryParser into a @FunctionalInterface and provides a way to
register queries using that. Cuts match and function_score queries over
to that registration method as a proof of concept.

Once all queries have been cut over we can remove their PROTOTYPES.
2016-03-31 13:43:07 -04:00
Christoph Büscher bbb6d91147 Add randomization of XContentBuilder output to query tests
Currently our testing of parsing query builders is limited to the
default order of the parameters that each builders toXContent()
method produces. To better test real queries where the order of
parameters can be different, this change adds a helper
method to ESTestCase that takes a XContentBuilder and randomly
shuffles the order of the fields inside an object. This is
used in AbstractQueryTestCase, but it can be used in other similar
places in the future.
2016-03-31 18:17:39 +02:00
Nik Everett c7780e6e0a Use ObjectParser in highlighting 2016-03-31 10:34:26 -04:00
Martijn Laarman 7de2377d38 Merge pull request #13906 from Mpdreamz/fix/remove-deprecated-get-alias-api
Remove deprecated indices.get_aliases
2016-03-31 15:57:03 +02:00
Simon Willnauer ea055d2ac9 add a comment why and when we can ignore skipTranslogRecovery in IndexShard 2016-03-31 15:54:21 +02:00
Simon Willnauer 4657796b24 fix test 2016-03-31 15:15:39 +02:00
Nik Everett 30a1862339 Remove PROTOTYPE from BulkItemResponse.Failure
Closes #17086
2016-03-31 09:10:36 -04:00
Nik Everett f8a67a2622 Remove PROTOTYPEs from ingest 2016-03-31 09:09:35 -04:00
Simon Willnauer f7cbc384f7 add tests to ensure translog ID is baked into commit on all open modes 2016-03-31 15:09:11 +02:00
Martijn Laarman dfe2c0ff0a Remove deprecated indices.get_aliases
This has been deprecated since the first release of Elasticsearch 1.0
2016-03-31 14:46:53 +02:00
Christoph Büscher 2843194635 Merge pull request #17441 from cbuescher/remove-RescoreParseElement
Remove RescoreParseElement after refactoring to RescoreBuilder.
2016-03-31 14:38:56 +02:00
Christoph Büscher 06184ee006 Remove RescoreParseElement
The refactoring of RescoreBuilder and QueryRescoreBuilder moved parsing
previously in RescoreParseElement into the builders. This removes the
left over parse element.
2016-03-31 14:37:21 +02:00
Simon Willnauer 0a277227c3 add back opem mode randomization 2016-03-31 13:06:23 +02:00
Simon Willnauer a356f74d4d Add test for IMC to ensure we also move buffers to disk during translog recovery 2016-03-31 12:58:00 +02:00
Simon Willnauer d006200c77 Make EngineConfig less mutable and ensure we prevent flushes during translog recovery phase. 2016-03-31 12:00:51 +02:00
Luca Cavanna 6b00b4b8dc Merge pull request #17410 from javanna/enhancement/discovery_node_one_getter
Remove duplicate getters from DiscoveryNode and DiscoveryNodes
2016-03-31 11:45:41 +02:00
Christoph Büscher c850b960fb Remove unused 'reverse' parse field from ScoreSortBuilder 2016-03-31 11:18:46 +02:00
javanna 32b6e529f4 Merge branch 'master' into enhancement/discovery_node_one_getter 2016-03-31 10:49:26 +02:00
Martijn van Groningen ccb009e45f percolator: Add scoring support to the percolator query
Percolator query documents are scored based on how well they match with the document being percolated.

Closes #13827
2016-03-31 10:08:43 +02:00
Martijn van Groningen 7600dc9943 percolator: Add support to extract terms from several types of span queries 2016-03-31 09:54:29 +02:00
Lee Hinman e266452cb2 Switch request parsing in ClusterAllocationExplainRequest to ObjectParser
Relates to #17305
2016-03-30 19:29:19 -06:00
Igor Motov e073b0c75d Add ability to group tasks by common parent
By default, tasks are grouped by node. However, task execution in elasticsearch can be quite complex and an individual task that runs on a coordinating node can have many subtasks running on other nodes in the cluster. This commit makes it possible to list task grouped by common parents instead of by node. When this option is enabled all subtask are grouped under the coordinating node task that started all subtasks in the group. To group tasks by common parents, use the following syntax:

 GET /tasks?group_by=parents
2016-03-30 17:50:27 -04:00
Simon Willnauer 1e06139584 Move translog recover outside of the engine
We changed the way we manage engine memory buffers to an
open model where each shard can essentially has infinite memory.
The indexing memory controller is responsible for moving memory to disk
when it's needed. Yet, this doesn't work today when we recover from store/translog
since the engine is not fully initialized such that IMC has no access to the engine,
neither to it's memory buffer nor can it move data to disk.

The biggest issue here is that translog recovery happends inside the Engine constructor
which is problematic by itself since it might take minutes and uses a not yet fully
initialzied engine to perform write operations on.

This change detaches the translog recovery and makes it the responsibility of the caller
to run it once the engine is fully constructed or skip it if not necessary.
2016-03-30 23:24:24 +02:00
Nik Everett 78ab6c5b7f [reindex] Dynamic throttle!
This allows the user to update the reindex throttle on the fly, with changes
that speed up the throttling being applied immediately and changes that
slow down the throttling being applied during the next batch. This means
that if a user throttles reindex in such a way that it tries to sleep for
16 years and then realizes that they've done something wrong then they
can change the throttle and reindex will wake up again. We don't apply
slow downs immediately so we never get in danger of losing the scan context.

Also, if reindex is canceled while it is sleeping (how it honor throttling)
then it'll immediately wake up and cancel itself.
2016-03-30 16:40:42 -04:00
Nik Everett 1ffb072df3 Replace static bwc data for new releases
The ones we had for 2.2.2 and 2.3.0 were broken.
2016-03-30 15:22:23 -04:00
Clinton Gormley bc998fd0c2 Added bwc indices for 2.2.2 and 2.3.0 2016-03-30 18:58:08 +02:00
javanna b9f9b2e3ee Merge branch 'master' into enhancement/discovery_node_one_getter 2016-03-30 17:22:40 +02:00
Simon Willnauer 97eaf6c04b Merge pull request #17414 from s1monw/issues/17412
Don't take recovery indexing into account on indexing stats

Closes #17412
2016-03-30 17:15:00 +02:00
Christoph Büscher f012b73947 Remove HighlighterParseElement
The old HighlightParseElement was only left because it was still
used in tests and some places in InnerHits. This PR removes it
and replaces the tests that checked that the original parse element
and the rafactored highlighter code produce the same output with
new tests that compare builder input to the SearchContextHighlight
that is created.
2016-03-30 16:59:49 +02:00
Adrien Grand d7179cafcc Add a soft limit on the number of shards that can be queried in a single search request. #17396
This commit adds the new `action.search.shard_count.limit` setting which
configures the maximum number of shards that can be queried in a single search
request. It has a default value of 1000.
2016-03-30 16:55:01 +02:00
Simon Willnauer dbcb9a29a5 Don't take recovery indexing into account on indexing stats
Closes #17412
2016-03-30 16:42:09 +02:00
Simon Willnauer e25ccb91ae [TEST] Make type inference simpler 2016-03-30 16:38:07 +02:00
javanna 52dcc9899e fix checkstyle line lenght issue 2016-03-30 15:45:41 +02:00
Simon Willnauer 43b87e8f86 Invoke `IndexingOperationListeners` also when recovering from store or remote
Today we don't invoke `IndexingOperationListeners` when we are running
a recovery form store or replaying translog from remote. This is problematic since
the actual code path for indexing is different between normal indexing and recovery.
An important detail is left out on recovery since we implemented the `IndexingMemoryController`
as an `IndexingOperationListener` we might never flush the `IndexWriter` of a recovering shard
which can lead to `OOMs` on node startup / recovery.
2016-03-30 15:32:43 +02:00
javanna 70573a35e1 remove unused DiscoveryNodes#valid method 2016-03-30 15:28:39 +02:00
javanna 30b57be8f4 Rename DiscoveryNodes#localNodeMaster to isLocalNodeElectedMaster 2016-03-30 15:28:39 +02:00
javanna 2075c7f0a7 Rename DiscoveryNodes#smallestNonClientNodeVersion to getSmallestNonClientNodeVersion 2016-03-30 15:28:39 +02:00
javanna 030420768c Rename DiscoveryNodes#smallestNodeVersion to getSmallestNodeVersion 2016-03-30 15:28:39 +02:00
javanna 62ac7d219f Remove DiscoveryNodes#masterNode in favour of existing DiscoveryNodes#getMasterNode 2016-03-30 15:28:32 +02:00
javanna 7ebc094353 Remove DiscoveryNodes#localNode in favour of existing DiscoveryNodes#getLocalNode 2016-03-30 15:28:23 +02:00
javanna f26d05eac8 Remove DiscoveryNodes#localNodeId in favour of existing DiscoveryNodes#getLocalNodeId 2016-03-30 15:28:15 +02:00
javanna f8b5d1f5b0 Remove DiscoveryNodes#masterNodeId in favour of existing DiscoveryNodes#getMasterNodeId 2016-03-30 15:28:06 +02:00
javanna 97f7aef776 Rename DiscoveryNodes#masterAndDataNodes to getMasterAndDataNodes 2016-03-30 15:28:04 +02:00
javanna 4032859e90 Remove DiscoveryNodes#masterNodes in favour of existing DiscoveryNodes#getMasterNodes 2016-03-30 15:27:56 +02:00
javanna 463fbe45c6 Remove DiscoveryNodes#dataNodes in favour of existing DiscoveryNodes#getDataNodes 2016-03-30 15:27:46 +02:00
javanna c175e07c8a Remove DiscoveryNodes#nodes in favour of existing DiscoveryNodes#getNodes 2016-03-30 15:27:38 +02:00
javanna fc2e6d7ce0 Remove DiscoveryNodes#size in favour of existing DiscoveryNodes#getSize 2016-03-30 15:27:28 +02:00
javanna 131f5b0596 Remove unused emptyDelta method from DiscoveryNodes and related EMPTY_LIST from DiscoveryNode 2016-03-30 15:27:26 +02:00
javanna 2dbba45f2c Rename static DiscoveryNode#localNode(Settings) to DiscoveryNode#isLocalNode(Settings) 2016-03-30 15:27:26 +02:00
javanna 7751b45f0f Rename static DiscoveryNode#ingestNode(Settings) to isIngestNode 2016-03-30 15:27:05 +02:00
javanna 49e952e272 Rename static DiscoveryNode#dataNode(Settings) to isDataNode 2016-03-30 15:26:41 +02:00
javanna 2230fec9ea Rename static DiscoveryNode#masterNode(Settings) to isMasterNode 2016-03-30 15:26:10 +02:00
Martijn van Groningen 7e2696c570 Refactored inner hits parsing and intoduced InnerHitBuilder
Both top level and inline inner hits are now covered by InnerHitBuilder.
Although there are differences between top level and inline inner hits,
they now make use of the same builder logic.

The parsing of top level inner hits slightly changed to be more readable.
Before the nested path or parent/child type had to be specified as encapsuting
json object, now these settings are simple fields. Before this was required
to allow streaming parsing of inner hits without missing contextual information.

Once some issues are fixed with inline inner hits (around multi level hierachy of inner hits),
top level inner hits will be deprecated and removed in the next major version.
2016-03-30 15:15:56 +02:00
Simon Willnauer ee140f4ee2 Remove unused 'node.client' setting 2016-03-30 15:03:05 +02:00
Nik Everett 1c16d63a9a Merge pull request #17394 from camilojd/refactor/replace-getrandom
Refactor: replace all ocurrences of ESTestCase.getRandom() with LuceneTestCase.random()
2016-03-30 08:58:21 -04:00
javanna 3942c9e4df Remove DiscoveryNode#version in favour of existing DiscoveryNode#getVersion 2016-03-30 14:56:15 +02:00
javanna 00f5ca57fa Remove DiscoveryNode#address in favour of existing DiscoveryNode#getAddress 2016-03-30 14:54:42 +02:00
javanna 41039a6431 Remove DiscoveryNode#masterNode in favour of existing DiscoveryNode#isMasterNode 2016-03-30 14:52:39 +02:00
javanna dfeb9379ab Remove DiscoveryNode#dataNode in favour of existing DiscoveryNode#isDataNode 2016-03-30 14:51:36 +02:00
javanna a8bbdff3bc Remove DiscoveryNode#name in favour of existing DiscoveryNode#getName 2016-03-30 14:47:36 +02:00
javanna 9889f10e5e Remove DiscoveryNode#id in favour of existing DiscoveryNode#getId 2016-03-30 14:42:15 +02:00
Simon Willnauer 81801451ad Factor out slow logs into Search and IndexingOperationListeners
This commit introduces SearchOperationListeneres which allow to hook
into search operation lifecycle and execute operations like slow-logs
and statistic collection in a transparent way. SearchOperationListenrs
can be registered on the IndexModule just like IndexingOperationListeners.
The main consumers (slow log) have already been moved out of IndexService
into IndexModule which reduces the dependency on IndexService as well as
IndexShard and makes slowlogging transparent.

Closes #17398
2016-03-30 14:38:37 +02:00
Adrien Grand fc47007e17 Add a soft limit on the mapping depth. #17400
This commit adds the new `index.mapping.depth.limit` setting which controls the
maximum mapping depth that is allowed. It has a default value of 20.
2016-03-30 14:37:00 +02:00
Adrien Grand 068c788ec8 Disable fielddata on text fields by defaults. #17386
`text` fields will have fielddata disabled by default. Fielddata can still be
enabled on an existing index by setting `fielddata=true` in the mappings.
2016-03-30 14:35:32 +02:00
Simon Willnauer af976a6673 Merge pull request #17402 from s1monw/issues/17280
Remove ability to specify arbitrary node attributes with `node.` prefix

Today the basic node settings like `node.data` and `node.master` can't really be fully validated
since we allow to specify custom user attributes on the node level. We have to, in order to
support that, add a wildcard setting for `node.*` to let these setting pass validation.
Instead we should require a more contraint prefix like `node.attr.` that defines a namespace
that is reserved for user attributes.
This commit adds a new namespace for attributes in `node.attr`.

Closes #17280
2016-03-30 14:14:26 +02:00
Simon Willnauer 8b075dbb75 Remove ability to specify arbitrary node attributes with `node.` prefix
Today the basic node settings like `node.data` and `node.master` can't really be fully validated
since we allow to specify custom user attributes on the node level. We have to, in order to
support that, add a wildcard setting for `node.*` to let these setting pass validation.
Instead we should require a more contraint prefix like `node.attr.` that defines a namespace
that is reserved for user attributes.
This commit adds a new namespace for attributes in `node.attr`.

Closes #17280
2016-03-30 13:29:48 +02:00
Isabel Drost-Fromm 453ffebb96 Merge pull request #13825 from MaineC/bug-fix/10021-logging-only
Add stack traces to logged exceptions where missing
2016-03-30 12:31:45 +02:00
Isabel Drost-Fromm d33197c4a2 Add stack traces to logged exceptions where missing
Relates to #10021
2016-03-30 12:30:29 +02:00
Britta Weber eed885eeab [TEST] use random ascii instead of random unicode
Otherwise fields might not have values after analysis and the docCount
and hence the score will be unpredictable.
2016-03-30 11:45:47 +02:00
Isabel Drost-Fromm f27399dc0e Merge pull request #17282 from MaineC/deprecation/sort-option-reverse-removal
Remove deprecated reverse option from sorting
2016-03-30 11:02:19 +02:00
Simon Willnauer 3e67b808a0 Add created flag to IndexingOperationListener#postIndex
This adds a `created` flag to `IndexingOperationListener#postIndex` to
easily differentiate between updates and creates on the listener level.

Closes #17333
Merges #17340
2016-03-30 10:52:59 +02:00
Simon Willnauer 76fd1b4777 Fix test bug 2016-03-30 10:52:38 +02:00
Simon Willnauer 1110e5bdb6 Merge branch 'master' into pr-17340 2016-03-30 10:49:16 +02:00
Alexander Reelsen 48951aeb86 PluginManager: Rename xpack plugin to x-pack 2016-03-30 09:50:01 +02:00
Boaz Leskes 977c93be3c make GsubProcessorFactoryTests.testCreateInvalidPattern windows friendly. 2016-03-30 08:45:28 +02:00
Camilo Diaz Repka 7be11a36cd Refactor: replace all ocurrences of ESTestCase.getRandom() for random().
Remove getRandom().
2016-03-29 23:18:05 -04:00
Jason Tedor aa583c4064 Fix spelling of linear interpolation scorer class
This commit fixes the spelling of LinearInterpoatingScorer to
LinearInterpolatingScorer. Note that there was a missing ell.

Closes #17384
2016-03-29 20:18:29 -04:00
Jason Tedor c63415d091 Cleanup format in LinearInterpoatingScorer.java
This commit fixes a few style issues in LinearInterpoatingScorer.java.
2016-03-29 20:17:55 -04:00
Nik Everett df08854c60 Remove PROTOTYPEs from suggesters
Also stops using guice for suggesters at all and lots of checkstyle.
2016-03-29 17:55:01 -04:00
javanna 19eeb68bc4 Merge branch 'master' into enhancement/remove_node_client_setting 2016-03-29 21:53:22 +02:00
Igor Motov 4f5f3fe895 Increase the total number of allowed fields in UpdateMappingIntegrationIT#testDynamicUpdates
With restriction for the total number of fields introduced in #17357 this test can fail if a large number of records is randomly selected for indexing.
2016-03-29 15:25:21 -04:00
javanna 061f09d9a4 Merge branch 'master' into enhancement/remove_node_client_setting 2016-03-29 20:19:33 +02:00
Yanjun Huang 361adcf387 Add limit to total number of fields in mapping. #17357
This is to prevent mapping explosion when dynamic keys such as UUID are used as field names. index.mapping.total_fields.limit specifies the total number of fields an index can have. An exception will be thrown when the limit is reached. The default limit is 1000. Value 0 means no limit. This setting is runtime adjustable

Closes #11443
2016-03-29 19:39:46 +02:00
javanna 0c70a9d5bd fix bug introduced with refactoring of DiscoveryNode constructors
Transport client was replacing the address of the nodes connecting to with the ones received from the liveness api rather keeping the original listed nodes. Written a test for that.
2016-03-29 18:36:09 +02:00
Jason Tedor 27448dc3d0 Adjust for long random timeout in acked indexing 2016-03-29 12:11:09 -04:00
Jason Tedor 649bcdc4eb Enable acked indexing test 2016-03-29 11:43:15 -04:00
Jason Tedor c4324f9964 Merge branch 'master' into enable_acked
* master: (25 commits)
  Replication operation that try to perform the primary phase on a replica should be retried
  split long line in ConvertProcessorTests
  add  type conversion support to ConvertProcessor
  percolator: Make explain use the two phase iterator
  test: make sure we don't flush during indexing the percolator queries
  Added experimental annotation to the update-by-query and reindex docs
  Fixed bad YAML in reindex REST test: 50_routing.yaml
  Update-by-query rest tests: fixed bad yaml and deleted a client-dependent test
  Prevents exception being raised when ordering by an aggregation which wasn't collected
  The reindex body is now required, which changes the exception thrown by the REST test
  Docs: Included Nodes Task API and tidied reindex/update-by-query
  Rename update-by-query REST tests to update_by_query
  REST: The body is required in the reindex API
  The source parameter should not be defined in the delete-by-query REST spec
  Renamed update-by-query REST spec to update_by_query
  Fix test bug in TypeQueryBuilderTests.
  Add comment why it is safe to check the number of nested fields in MapperService.merge.
  Automatically add a sub keyword field to string dynamic mappings. #17188
  Type filters should not have a performance impact when there is a single type. #17350
  Add API to explain why a shard is or isn't assigned
  ...
2016-03-29 11:42:34 -04:00
Boaz Leskes 48b4f086e0 Replication operation that try to perform the primary phase on a replica should be retried
In extreme cases a local primary shard can be replaced with a replica while a replication request is in flight and the primary action is applied to the shard (via `acquirePrimaryOperationLock()).  #17044 changed the exception used in that method to something that isn't recognized as `TransportActions.isShardNotAvailableException`, causing the operation to fail immediately instead of retrying. This commit fixes this by check the primary flag before
acquiring the lock. This is safe to do as an IndexShard will never be demoted once a primary.

Closes #17358
2016-03-29 17:21:14 +02:00
Jason Tedor 0e5b22a648 Remove pending locks assertions from TRA 2016-03-29 11:20:04 -04:00
Jason Tedor 85d3d51a74 Clarify message on out-of-order state publish 2016-03-29 11:20:03 -04:00
Tal Levy 833fc8420f split long line in ConvertProcessorTests 2016-03-29 08:19:15 -07:00
Tal Levy 16e888fac3 Merge pull request #17260 from talevy/fix-regex-exceptions
Handle regex parsing errors in Gsub and Grok Processors
2016-03-29 08:12:26 -07:00
Tal Levy 9ac3887139 Merge pull request #17263 from talevy/auto-convert
add  type conversion support to ConvertProcessor
2016-03-29 07:57:57 -07:00
Tal Levy 2064fe3985 add type conversion support to ConvertProcessor 2016-03-29 07:56:53 -07:00
Martijn van Groningen 9d37f459b5 percolator: Make explain use the two phase iterator
So that we don't eveluate percolator queries that don't match.

Closes #17314
2016-03-29 16:26:31 +02:00
Martijn van Groningen 60793a848e test: make sure we don't flush during indexing the percolator queries 2016-03-29 16:24:49 +02:00
Isabel Drost-Fromm 778a447ef0 Do not fail if providing coordinates for same field name 2016-03-29 14:38:48 +02:00
Colin Goodheart-Smithe ff3fd99074 Prevents exception being raised when ordering by an aggregation which wasn't collected
If a terms aggregation was ordered by a metric nested in a single bucket aggregator which did not collect any documents (e.g. a filters aggregation which did not match in that term bucket) an ArrayOutOfBoundsException would be thrown when the ordering code tried to retrieve the value for the metric. This fix fixes all numeric metric aggregators so they return their default value when a bucket ordinal is requested which was not collected.

Closes #17225
2016-03-29 13:28:03 +01:00
javanna 8fc9dbbb99 Merge branch 'master' into enhancement/remove_node_client_setting 2016-03-29 14:27:04 +02:00
javanna f9a5e1a03a wrap EnumSet into unmodifiableSet directly, plus minor changes 2016-03-29 14:26:09 +02:00
Isabel Drost-Fromm 46e4ea9d16 Refine dealing with removed reverse sort option.
For geo distance sort parsing: Disallow anything but
VALUE_STRING as geo hash, disallow resetting field
name for geo fields.

Also make error message for wrong lat/lon values more
verbose by including the affected field name.
2016-03-29 13:16:33 +02:00
javanna 36f446759f fix silly serialization mistake in DiscoveryNode 2016-03-29 12:44:31 +02:00
javanna f7becf1f53 Move Set<Role> to EnumSet<Role> in DiscoveryNode 2016-03-29 12:33:14 +02:00
Adrien Grand cb31e591f1 Fix test bug in TypeQueryBuilderTests. 2016-03-29 11:43:27 +02:00
javanna 5794455912 Use regular Map for attributes in DiscoveryNode, get rid of attributes getter in favour of regular getter 2016-03-29 11:42:44 +02:00
Isabel Drost-Fromm 5a913fcc69 Fix build errors after last merge. 2016-03-29 11:26:41 +02:00
javanna 30ac3dcf01 use ordinary incrementing loops in ClusterStatsNodes 2016-03-29 11:05:58 +02:00
Isabel Drost-Fromm 407e2cdcf9 Merge branch 'master' into deprecation/sort-option-reverse-removal
Conflicts:
	core/src/main/java/org/elasticsearch/search/sort/ScoreSortBuilder.java
	core/src/test/java/org/elasticsearch/search/sort/FieldSortBuilderTests.java
2016-03-29 11:04:02 +02:00
javanna 66a8e4efeb less streams more loops 2016-03-29 11:01:49 +02:00
javanna 8034e13365 adapt cluster allocation explain to DiscoveryNode changes 2016-03-29 10:53:42 +02:00
javanna de5cbda8e7 Merge branch 'master' into enhancement/remove_node_client_setting 2016-03-29 10:48:47 +02:00
Adrien Grand c7bdfb1126 Add comment why it is safe to check the number of nested fields in MapperService.merge. 2016-03-29 10:02:29 +02:00
Adrien Grand 0eedc784fe Automatically add a sub keyword field to string dynamic mappings. #17188
If you add a string field to a document, it will have the following default
mapping:

```
{
  "type": "text",
  "fields": {
    "keyword": {
      "type": "keyword",
      "ignore_above": 256
    }
  }
}
```
2016-03-29 09:49:16 +02:00
Adrien Grand 4bd27bc2a0 Type filters should not have a performance impact when there is a single type. #17350
Today, if you call /index/type/_search instead of /index/_search, elasticsearch
will automatically insert a type filter to only match documents of the given
type. This commit uses a new TypeQuery instead of a TermQuery for this filter,
which rewrites to a MatchAllDocsQuery when all documents of a shard match the
filtered type. This is helpful because BooleanQuery has a special rewrite rule
to remove MatchAllDocsQuery as FILTER clauses. So for instance if your query is
`+body:"quick fox" #_type:my_type`, it will be rewritten to
`+body:"quick fox" #*:*` which is then rewritten to `body:"quick fox"`.
2016-03-29 08:52:49 +02:00
Lee Hinman c63cb21745 Merge remote-tracking branch 'dakrone/allocation-explain' 2016-03-28 16:06:49 -06:00
Lee Hinman 80ab366de4 Add API to explain why a shard is or isn't assigned
This adds a new `/_cluster/allocation/explain` API that explains why a
shard can or cannot be allocated to nodes in the cluster. Additionally,
it will show where the master *desires* to put the shard, according to
the `ShardsAllocator`.

It looks like this:

```
GET /_cluster/allocation/explain?pretty
{
  "index": "only-foo",
  "shard": 0,
  "primary": false
}
```

Though, you can optionally send an empty body, which means "explain the
allocation for the first unassigned shard you find".

The output when a shard is unassigned looks like this:

```
{
  "shard" : {
    "index" : "only-foo",
    "index_uuid" : "KnW0-zELRs6PK84l0r38ZA",
    "id" : 0,
    "primary" : false
  },
  "assigned" : false,
  "unassigned_info" : {
    "reason" : "INDEX_CREATED",
    "at" : "2016-03-22T20:04:23.620Z"
  },
  "nodes" : {
    "V-Spi0AyRZ6ZvKbaI3691w" : {
      "node_name" : "Susan Storm",
      "node_attributes" : {
        "bar" : "baz"
      },
      "final_decision" : "NO",
      "weight" : 0.06666675,
      "decisions" : [ {
        "decider" : "filter",
        "decision" : "NO",
        "explanation" : "node does not match index include filters [foo:\"bar\"]"
      } ]
    },
    "Qc6VL8c5RWaw1qXZ0Rg57g" : {
      "node_name" : "Slipstream",
      "node_attributes" : {
        "bar" : "baz",
        "foo" : "bar"
      },
      "final_decision" : "NO",
      "weight" : -1.3833332,
      "decisions" : [ {
        "decider" : "same_shard",
        "decision" : "NO",
        "explanation" : "the shard cannot be allocated on the same node id [Qc6VL8c5RWaw1qXZ0Rg57g] on which it already exists"
      } ]
    },
    "PzdyMZGXQdGhqTJHF_hGgA" : {
      "node_name" : "The Symbiote",
      "node_attributes" : { },
      "final_decision" : "NO",
      "weight" : 2.3166666,
      "decisions" : [ {
        "decider" : "filter",
        "decision" : "NO",
        "explanation" : "node does not match index include filters [foo:\"bar\"]"
      } ]
    }
  }
}
```

And when the shard *is* assigned, the output looks like:

```
{
  "shard" : {
    "index" : "only-foo",
    "index_uuid" : "KnW0-zELRs6PK84l0r38ZA",
    "id" : 0,
    "primary" : true
  },
  "assigned" : true,
  "assigned_node_id" : "Qc6VL8c5RWaw1qXZ0Rg57g",
  "nodes" : {
    "V-Spi0AyRZ6ZvKbaI3691w" : {
      "node_name" : "Susan Storm",
      "node_attributes" : {
        "bar" : "baz"
      },
      "final_decision" : "NO",
      "weight" : 1.4499999,
      "decisions" : [ {
        "decider" : "filter",
        "decision" : "NO",
        "explanation" : "node does not match index include filters [foo:\"bar\"]"
      } ]
    },
    "Qc6VL8c5RWaw1qXZ0Rg57g" : {
      "node_name" : "Slipstream",
      "node_attributes" : {
        "bar" : "baz",
        "foo" : "bar"
      },
      "final_decision" : "CURRENTLY_ASSIGNED",
      "weight" : 0.0,
      "decisions" : [ {
        "decider" : "same_shard",
        "decision" : "NO",
        "explanation" : "the shard cannot be allocated on the same node id [Qc6VL8c5RWaw1qXZ0Rg57g] on which it already exists"
      } ]
    },
    "PzdyMZGXQdGhqTJHF_hGgA" : {
      "node_name" : "The Symbiote",
      "node_attributes" : { },
      "final_decision" : "NO",
      "weight" : 3.6999998,
      "decisions" : [ {
        "decider" : "filter",
        "decision" : "NO",
        "explanation" : "node does not match index include filters [foo:\"bar\"]"
      } ]
    }
  }
}
```

Only "NO" decisions are returned by default, but all decisions can be
shown by specifying the `?include_yes_decisions=true` parameter in the
request.

Resolves #14593
2016-03-28 15:21:02 -06:00
Igor Motov 8a5c19b25f Handle RejectedExecution gracefully in TransportService during shutdown
Today we might run into a rejected execution exception when we shutdown the node while handling a transport exception. The exception is run in a seperate thread but that thread might not be able to execute due to the shutdown. Today we barf and fill the logs with large exception. This commit catches this exception and logs it as debug logging instead.

Extends changes made in 8652cd8
2016-03-28 15:12:50 -04:00
Areek Zillur 976b8d2cee Remove dead code
This commit removes dead DeleteByQuery class from core.
2016-03-28 14:48:40 -04:00
Jason Tedor 4de57fc5aa Simplify test out of order commit messages 2016-03-28 13:48:28 -04:00
Jason Tedor 5576526c91 For now do not guard against already failed engine 2016-03-28 13:44:41 -04:00
Jason Tedor 2a9388912b Add clarifying comment on disrupted in-flight ops 2016-03-28 13:34:18 -04:00
Jason Tedor 4e1f62eae9 Clarify exceptions when failing to fail a replica 2016-03-28 13:19:12 -04:00
Jason Tedor 97be38353a Fix formatting in DWSDIT#TCJDOPI 2016-03-28 12:49:56 -04:00
Jason Tedor 37d739a3cd Simplify doc creation check in acked indexing test
This commit simplifies the doc creation check while indexing in the
acked indexing test.
2016-03-28 12:46:54 -04:00
Jason Tedor 14ba0c31b4 Use longer timeout on nightly tests, but rarely
This commit increases the timeout while indexing during the acked
indexing test when running nightly tests, but only rarely.
2016-03-28 12:44:09 -04:00
Jason Tedor 4793630eb8 Merge branch 'master' into enable_acked
* master: (419 commits)
  Remove PROTOTYPE from ShapeBuilders
  Take filterNodeIds into consideration while sending tasks actions requests to nodes
  test: cleanup imports and method rename
  Remove PROTOTYPE from SortBuilders
  percolator: Add query extract support for the blended term query and the common terms query.
  Don't iterate over shard routing if it's null
  [TEST] Reduce size of random shapes
  Add some debug logging to testPrimaryRelocationWhileIndexing
  Order methods in IndicesClusterStateService according to execution
  Tidied up percolator doc annotations
  In cat.snapshots, repository is required
  Do not retrieve all indices stats when checking for cache resets
  Enforce `discovery.zen.minimum_master_nodes` is set when bound to a public ip #17288
  Port Primary Terms to master #17044
  Revert "Add debug logging for Vagrant upgrade test"
  Ownership for data, logs, and configs for packages
  add on_failure exception metadata to ingest document for verbose simulate
  Revert "Merge pull request #16843 from xuzha/s3-encryption"
  Update Format, add new settings into the setting test
  Update and rebase the init implementation.
  ...
2016-03-28 12:29:53 -04:00
Nik Everett 9402251eaf Remove PROTOTYPE from ShapeBuilders
Also cuts lots of tests over to expectThrows and fixes DistanceUnit's
serialization.
2016-03-26 13:58:33 -04:00
Igor Motov ee49081bc7 Take filterNodeIds into consideration while sending tasks actions requests to nodes
This commit fixes a bug that was causing the result of TransportTasksAction#filterNodeIds to be ignored and as a result the tasks actions were executed on all nodes.
2016-03-26 13:15:39 -04:00
Martijn van Groningen 444641ac55 test: cleanup imports and method rename 2016-03-26 15:13:56 +01:00
Nik Everett 8a89482555 Remove PROTOTYPE from SortBuilders 2016-03-25 22:08:50 -04:00
Martijn van Groningen 6cb82965bf percolator: Add query extract support for the blended term query and the common terms query. 2016-03-25 23:22:44 +01:00
Simon Willnauer 3b753ea4c9 Don't iterate over shard routing if it's null 2016-03-25 23:05:55 +01:00
javanna e6c6632e6f add explicit check for Role enum ordinal when reading from StreamInput 2016-03-25 22:58:37 +01:00
javanna 1852b6627f fix failing ZenDiscoveryIT 2016-03-25 21:48:48 +01:00
javanna 947e9ed2c7 fix checkstyle problems 2016-03-25 21:37:40 +01:00
javanna f9e6163365 fix line length problems 2016-03-25 20:29:24 +01:00
javanna a6211c72eb removed needless statis import changes to simplify review 2016-03-25 20:28:23 +01:00