Commit Graph

26240 Commits

Author SHA1 Message Date
Jim Ferenczi b781a4a176 Add unit tests for FiltersAggregator (#22678)
Adds unit tests for the `filters` aggregation.
This change also adds an helper to search and reduce any aggregator in a unit test.
This is done by dividing a single searcher in sub-searcher, one for each segment.

Relates #22278
2017-01-19 16:22:48 +01:00
David Pilato 0315dcc306 Use now common methods with index/update
Brought by #22229
2017-01-19 16:10:13 +01:00
Jim Ferenczi 3d54258de2 Don't register search response listener in transport clients
Small fix for https://github.com/elastic/elasticsearch/pull/22682
2017-01-19 16:08:24 +01:00
David Pilato 718a6b9be7 Add fromxcontent methods to delete response
This commit adds the parsing fromXContent() methods to the IndexResponse class.

It's a pale copy of what has been done in #22229.
2017-01-19 15:59:24 +01:00
Nicholas Knize b006636aaf unmute FieldStatsIntegrationIT.testGeoPointNotIndexed, fix already pushed 2017-01-19 08:44:00 -06:00
Nicholas Knize 88c78833f0 Mute FieldStatsIntegrationIT.testGeoPointNotIndexed, for now 2017-01-19 08:38:17 -06:00
Nik Everett 3ce41a0e15 Painless: Add augmentation to string for base 64 (#22665)
We don't want to expose `String#getBytes` which is required for
`Base64.getEncoder.encode` to work because we're worried about
character sets. This adds `encodeBase64` and `decodeBase64`
methods to `String` in Painless that are duals of one another
such that:
`someString == someString.encodeBase64().decodeBase64()`.

Both methods work with the UTF-8 encoding of the string.

Closes #22648
2017-01-19 09:31:45 -05:00
Jim Ferenczi d145d459ae Fix NPE on FieldStats with mixed cluster on version pre/post 5.2 (#22688)
* Fix NPE on FieldStats with mixed cluster on version pre/post 5.2

In 5.2 the FieldStats API can return null min/max values.
These values cannot be deserialized by a node with version pre 5.2 so if this node
is pick to coordinate a FieldStats request in a mixed cluster an NPE can be thrown.
This change prevents the NPE by removing the non serializable FieldStats object directly in the field stats shard request.
The filtered fields will not be present in the response when a node pre 5.2 acts as a coordinating node.
2017-01-19 14:20:07 +01:00
Tanguy Leroux 833284cae2 Add parsing methods for UpdateResponse (#22586)
This commit adds the fromXContent() method to the UpdateResponse class, so that it can be used with the high level rest client.
2017-01-19 12:49:45 +01:00
Jim Ferenczi 21dae1924f Add the ability to define search response listeners in plugins (#22682)
This change is a simple adaptation of https://github.com/elastic/elasticsearch/pull/19587 for the current state of master.
It allows to define search response listener in the form of `BiConsumer<SearchRequest, SearchResponse>`s in a search plugin.
2017-01-19 12:48:45 +01:00
Yannick Welsch 652cb7dbf7 Make build Gradle 2.14 / 3.x compatible (#22669)
This changes build files so that building Elasticsearch works with both Gradle 2.13 as well as higher versions of Gradle (tested 2.14 and 3.3), enabling a smooth transition from Gradle 2.13 to 3.x.
2017-01-19 09:56:54 +01:00
Daniel Mitterdorfer ce765f7ad2 Use a proper boolean in FieldStatsIntegrationIT#testGeoPointNotIndexed() 2017-01-19 08:33:08 +01:00
Daniel Mitterdorfer aece89d6a1 Make boolean conversion strict (#22200)
This PR removes all leniency in the conversion of Strings to booleans: "true"
is converted to the boolean value `true`, "false" is converted to the boolean
value `false`. Everything else raises an error.
2017-01-19 07:59:18 +01:00
Nik Everett ee5f8c4522 Consolidate some reindex utility classes (#22666)
Everything that extended `AbstractAsyncBulkByScrollAction` also
extended `AbstractAsyncBulkIndexByScrollAction` so this removes
`AbstractAsyncBulkIndexByScrollAction`, merging it into
`AbstractAsyncBulkByScrollAction`.
2017-01-18 16:58:39 -05:00
Nicholas Knize 51e80e7176 remove unnecessary text from exception message 2017-01-18 14:51:56 -06:00
Nicholas Knize 84e4f91253 Add geo_point to FieldStats
This commit adds a new GeoPoint class to FieldStats for computing field stats over geo_point field types.
2017-01-18 14:37:03 -06:00
Nik Everett 1fe74a6b4b Better error when can't auto create index (#22488)
Changes the error message when `action.auto_create_index` or
`index.mapper.dynamic` forbids automatic creation of an index
from `no such index` to one of:
* `no such index and [action.auto_create_index] is [false]`
* `no such index and [index.mapper.dynamic] is [false]`
* `no such index and [action.auto_create_index] contains [-<pattern>] which forbids automatic creation of the index`
* `no such index and [action.auto_create_index] ([all patterns]) doesn't match`

This should make it more clear *why* there is `no such index`.

Closes #22435
2017-01-18 15:18:32 -05:00
Ali Beyad cd52065871 [TEST] testAckedIndexing waits for all nodes to stabilize
testAckedIndexing now waits for all nodes to stabilize in the cluster
state through an assertBusy before final validation that all documents
are found in tehir respective shards in the cluster.  Before, what could
happen is that the ensureGreen check passes but only after that is a
ping failure from the network disruption processed by the master,
thereby rendering the cluster RED again.  This assertBusy waits up to 30
seconds for all nodes to have stabilized and all get document actions to
succeed.
2017-01-18 13:51:25 -05:00
Tim Brooks 2766b08ff4 Add doPrivilege blocks for socket connect operations in plugins (#22534)
This is related to #22116. Certain plugins (discovery-azure-classic, 
discovery-ec2, discovery-gce, repository-azure, repository-gcs, and 
repository-s3) open socket connections. As SocketPermissions are 
transitioned out of core, these plugins will require connect 
permission. This pull request wraps operations that require these 
permissions in doPrivileged blocks.
2017-01-18 10:12:18 -06:00
Michael McCandless 1d1bdd476c Finish exposing FlattenGraphTokenFilter (#22667) 2017-01-18 11:05:34 -05:00
Nik Everett e71b26f480 Improve unit test coverage of aggs (#22668)
Add tests for `GlobalAggregator`, `MaxAggregator`, and `InternalMax`.

Relates to #22278
2017-01-18 10:33:45 -05:00
Simon Willnauer 24e2847af2 Streamline foreign stored context restore and allow to perserve response headers (#22677)
Today we do not preserve response headers if they are present on a transport protocol
response. While preserving these headers is not always desired, in the most cases we
should pass on these headers to have consistent results for depreciation headers etc.
yet, this hasn't been much of a problem since most of the deprecations are detected early
ie. on the coordinating node such that this bug wasn't uncovered until #22647

This commit allow to optionally preserve headers when a context is restored and also streamlines
the context restore since it leaked frequently into the callers thread context when the callers
context wasn't restored again.
2017-01-18 16:17:54 +01:00
Ali Beyad 8a0a1140a9 [TEST] add logging to MockRepository to help debug index-N blob reading 2017-01-18 08:53:29 -05:00
Boaz Leskes 1227044ddd Add a deprecation notice to shadow replicas (#22647)
Relates to #22024

On top of documentation, the PR adds deprecation loggers and deals with the resulting warning headers.

The yaml test is set exclude versions up to 6.0. This is need to make sure bwc tests pass until this is backported to 5.2.0 . Once that's done, I will change the yaml test version limits
2017-01-18 12:28:09 +01:00
Ke Li 797d105177 Remove unnecessary class cast 2017-01-18 11:09:09 +01:00
Christoph Büscher 9ed867ea83 [DOCS] Fix inconsistent formatting for fieldnames in profile.asciidoc 2017-01-18 10:41:22 +01:00
Clinton Gormley e415170989 Docs: Fix link syntax in cross-cluster docs
Also tidy up settings list
2017-01-18 10:13:48 +01:00
Clinton Gormley 5d70df855d Docs: Cross-cluster search doc wasn't being included 2017-01-18 10:02:51 +01:00
Simon Willnauer 6f7caed148 Add federated cross cluster search capabilities (#22502)
Today Elasticsearch can act as a tribe node that fully joins another cluster to support operations across all different clusters. While tribe is a popular feature it also has it's downsides like:
 * non-trivial testing
 * special configuration
 * receives a non trivial amount of cluster-state updates
 * maintains connections to all nodes in all clusters and vice versa
 * has to be restarted to join another cluster
 * needs special strategies to disambiguate indices with identical names

That said, from a functionality standpoint the only feature in elasticsearch that needs cross cluster communication is in-fact the search layer. Everything else can be done on the client side by communicating with each cluster individually. For `_search` the merge of aggregations etc. is non trivial and can't be done on the client side. 

The feature added in this PR is called `cross cluster search` and allows any node to act as a federated search node without joining any of the other clusters. There are 2 basic modes of operations: 

 * globally configured remote clusters via cluster state settings
 * locally configured remote clusters via elasticsearch.yaml

A node with such a configuration will `connect` to the remote cluster and discover a set of nodes that it can communicate with for federated search (Default # of nodes is 3). It won't connect to all nodes but only eligible nodes in the remote cluster (depending on their version and optionally on a node attribute (`node.addr`).

Remote clusters can be configured and updated at any time via cluster settings without the need of a node-restart.  Each remote cluster specified via a `name` -> `seed node IP list` ie:

 * `search.remote.my_cluster.seeds: 127.0.0.1:9300, 127.0.0.1:9301`

indices of this cluster can then be addressed via the clusters alias: `GET index_1,my_cluster:index_1, my_cluster:other*/_search` 


Closes #21473
2017-01-18 09:28:04 +01:00
Simon Willnauer 19f9cb307a Merge branch 'master' into feature/multi_cluster_search 2017-01-18 09:24:35 +01:00
Scott Somerville 372812da98 Allow an index to be partitioned with custom routing (#22274)
This change makes it possible for custom routing values to go to a subset of shards rather than
just a single shard. This enables the ability to utilize the spatial locality that custom routing can
provide while mitigating the likelihood of ending up with an imbalanced cluster or suffering
from a hot shard.

This is ideal for large multi-tenant indices with custom routing that suffer from one or both of
the following:
- The big tenants cannot fit into a single shard or there is so many of them that they will likely
end up on the same shard
- Tenants often have a surge in write traffic and a single shard cannot process it fast enough

Beyond that, this should also be useful for use cases where most queries are done under the context
of a specific field (e.g. a category) since it gives a hint at how the data can be stored to minimize
the number of shards to check per query. While a similar solution can be achieved with multiple
concrete indices or aliases per value today, those approaches breakdown for high cardinality fields.

A partitioned index enforces that mappings have routing required, that the partition size does not
change when shrinking an index (the partitions will shrink proportionally), and rejects mappings
that have parent/child relationships.

Closes #21585
2017-01-18 08:51:23 +01:00
Elijah 3b92179e09 Improve wording in recipes docs
This commit improves some of the wording the recipes docs.

Relates #22661
2017-01-17 21:00:36 -05:00
Elijah b56605d68c Correct grammar in list in how-to docs
This commit corrects the grammar in a list in the how-to docs; namely,
the word "and" was missing preceding the final element in a list.

Relates #22663
2017-01-17 20:57:22 -05:00
Igor Motov 500548fcda Remove taskManager.registerChildTask
Instead of forcing each task to register all nodes where its children are running, this commit runs cancellation on all nodes. The task cancellation operation doesn't run too frequently, so this optimization doesn't seem to be worth additional complexity of the interface.
2017-01-17 18:07:31 -05:00
Ali Beyad 099d229138 [TEST] fix documentation checking tests to account for possible
pending tasks in the cluster state
2017-01-17 17:29:14 -05:00
Ryan Ernst 621643a5c3 Build: Only add ASL license to pom for elasticsearch project (#22664)
The extra plugins that may be attached to the elasticsearch build
contain their own license. In the past, the ASL license elasticsearch
uses was avoided by specially checking for the gradle project prefix of
`:x-plugins`. However, since refactoring to the elasticsearch-extra dir
structure, this mechanism was broken. This change fixes the pom license
adding to only be applied to projects that fall under the root project
(ie elasticsearch).
2017-01-17 13:38:32 -08:00
Ali Beyad ce811feba7 [TEST] testAckedIndexing waits for the cluster state to have propogated
to all nodes in the cluster before checking the existance of documents
on each node
2017-01-17 15:36:31 -05:00
Nik Everett 1169cd936e Fix compilation in eclipse
Eclipse needs a bit of extra special help with type parameters
in `TransportReplicationActionTests` now.
2017-01-17 14:53:54 -05:00
Elijah 297b1b7d9a Capitalize "Elasticsearch" in indexing speed docs
This commit fixes the capitalization of "Elasticsearch" in the indexing
speed docs.

Relates #22659
2017-01-17 12:33:01 -05:00
Ali Beyad 554a5e3039 [TEST] add retries to MockRepository getRepositoryData to try to
diagnose a NotXContentException being thrown
2017-01-17 12:17:29 -05:00
Simon Willnauer 69f1ffb1f8 fix exception message 2017-01-17 17:29:43 +01:00
Simon Willnauer 292e3a60d1 apply review comments 2017-01-17 17:20:52 +01:00
Alex a0c83c4511 Minor doc changes to clarify mapping index param for string type (#22652)
* Grammatical correction

* Add note for legacy string mapping type

* Update truncate token filter to not mention the keyword tokenizer

The advice predates the existence of the keyword field

Closes #22650
2017-01-17 16:43:11 +01:00
Luca Cavanna bc5b604cbd [TEST] parse global parameters from _common.json (#22655)
Replace the hardcoded global parameters in the yaml test suite with parameters parsed from the newly added _common.json file.

Relates to #22569
2017-01-17 16:13:09 +01:00
Ali Beyad e2977889b8 Allow comma delimited array settings to have a space after each entry (#22591)
Previously, certain settings that could take multiple comma delimited
values would pick up incorrect values for all entries but the first if
each comma separated value was followed by a whitespace character.  For
example, the multi-value "A,B,C" would be correctly parsed as
["A", "B", "C"] but the multi-value "A, B, C" would be incorrectly parsed
as ["A", " B", " C"].

This commit allows a comma separated list to have whitespace characters
after each entry.  The specific settings that were affected by this are:

  cluster.routing.allocation.awareness.attributes
  index.routing.allocation.require.*
  index.routing.allocation.include.*
  index.routing.allocation.exclude.*
  cluster.routing.allocation.require.*
  cluster.routing.allocation.include.*
  cluster.routing.allocation.exclude.*
  http.cors.allow-methods
  http.cors.allow-headers

For the allocation filtering related settings, this commit also provides
validation of each specified entry if the filtering is done by _ip,
_host_ip, or _publish_ip, to ensure that each entry is a valid IP
address.

Closes #22297
2017-01-17 08:51:04 -06:00
Tanguy Leroux f5542ed47f Simplify ElasticsearchException rendering as a XContent (#22611)
This commit tries to simplify the way ElasticsearchException are rendered to xcontent. It adds some documentation and renames and merges some methods. Current behavior is preserved, the goal is to be more readable and centralize everything in the ElasticsearchException class.
2017-01-17 15:44:49 +01:00
Simon Willnauer 197cd7d7a9 Add test for the grouping error message if indices and cluster can't be disambiguated 2017-01-17 14:13:09 +01:00
Simon Willnauer 88f6ae55f5 Improve remote / local indices filtering by not modifying external state 2017-01-17 14:05:36 +01:00
Greg Marzouka e0f8d88d5c Include global query string parameters in the REST spec
Closes #11638
2017-01-17 07:35:14 -05:00
Simon Willnauer 709cb9a39e Merge branch 'master' into feature/multi_cluster_search 2017-01-17 12:34:36 +01:00