Commit Graph

6965 Commits

Author SHA1 Message Date
Isabel Drost-Fromm 3e0aedd7da Merge branch 'master' into feature/rank-eval 2016-11-29 11:09:22 +01:00
Simon Willnauer f5ff69fabe Remove connectToNodeLight and replace it with a connection profile (#21799)
The Transport#connectToNodeLight concepts is confusing and not very flexible.
neither really testable on a unittest level. This commit cleans up the code used
to connect to nodes and simplifies transport implementations to share more code.
This also allows to connect to nodes with custom profiles if needed, for instance
future improvements can be added to connect to/from nodes that are non-data nodes without
dedicated bulks and recovery connections.
2016-11-29 09:35:07 +01:00
Ali Beyad a884573898 [TEST] fixes FilterAllocationDecider test for decision explanation
when the initial recovery is LOCAL_SHARDS
2016-11-28 20:37:19 -05:00
Ali Beyad 07bd0a30f0 Improves allocation decider decision explanation messages (#21771)
This commit improves the decision explanation messages,
particularly for NO decisions, in the various AllocationDecider
implementations by including the setting(s) in the explanation
message that led to the decision.

This commit also returns a THROTTLE decision instead of a NO
decision when the concurrent rebalances limit has been reached
in ConcurrentRebalanceAllocationDecider, because it more accurately
reflects a temporary throttling that will turn into a YES decision
once the number of concurrent rebalances lessens, as opposed to a
more permanent NO decision (e.g. due to filtering).
2016-11-28 20:23:16 -05:00
Matt Weber 04e07bcdb6 Synonym Graph Support (LUCENE-6664) (#21517)
Integrate the patch from LUCENE-6664 into elasticsearch and
add support for handling a graph token stream in match/multi-match
queries.

This fixes longstanding bugs with multi-token synonyms returning
incorrect results with proximity queries.
2016-11-28 09:25:49 -08:00
Jim Ferenczi 8affb7c845 Fix FiltersFunctionScoreQuery highlighting (#21827)
This is a cleanup of the fix pushed in https://github.com/elastic/elasticsearch/pull/20400.
FiltersFunctionScoreQuery sub query should be extracted in CustomQueryScorer.extract (and not in CustomQueryScorer.extractUnknownQuery).
This does not fix any bug in this branch (it's just a cleanup) but the intent is first to clean up and then to backport in 2.x where there is a real bug.
The bug is in 2.x only because the backport of https://github.com/elastic/elasticsearch/pull/20400 in 2.x mistakenly renamed the FiltersFunctionScoreQuery to FunctionScoreQuery.
This leads to incorrect highlighting on FiltersFunctionScoreQuery in 2.x.
2016-11-28 17:56:24 +01:00
Nik Everett 145d0813b5 Log ScriptException's xcontent if file script compilation fails (#21767)
When a file script fails to compile, rather than logging the
exception that caused the failure this logs the xcontent of
that exception. This is both shorter and has the script stack
which is useful for figuring out why the compilation failed.

Still logs the entire stacktrace at debug level just in case
you need it.

Relates to #21733
2016-11-28 11:36:06 -05:00
Ali Beyad db7362da67 Fixes shard level snapshot metadata loading when index-N file is missing (#21813)
In making changes for the 5.0 version of snapshots, a bug was
introduced where if an index-N file could not be found for an
individual shard, the backup was to iterate over all snap-*.dat
files in the shard folder to know which snapshots contain that
shard's data, but in 5.0, reading the snap-*.dat files as backup
was incorrectly passing in the blob name for the snap-*.dat file,
thereby failing to load all index files for a given snapshot
when the index-N file is missing.  This condition should be rare
as there is no reason an index-N file should be absent (unless
it was deleted or there was corruption reading the file), but
nevertheless, this situation can be encountered and this commit
fixes the bug by reading the correct snap-*.dat blob name in the
shard data folder.
2016-11-28 10:46:33 -05:00
Simon Willnauer b7292a6005 Remove TcpTransport#addressSupported since TransportAddress is now final
TransportAddress used to be customizable per transport but this has been removed
a while ago. Therefore we can remove all usage of this method as well.

Relates to #20695
2016-11-28 16:06:59 +01:00
Jim Ferenczi 69f35aa07f Fix cross_fields type on multi_match query with synonyms (#21638)
* Fix cross_fields type on multi_match query with synonyms

This change fixes the cross_fields type of the multi_match query when synonyms are involved.
Since 2.x the Lucene query parser creates SynonymQuery for words that appear at the same position.
For simple term query the CrossFieldsQueryBuilder expands the term to all requested fields and creates a BlendedTermQuery.
This change adds the same mechanism for SynonymQuery which otherwise are not expanded to all requested fields.
As a side note I wonder if we should not replace the BlendedTermQuery with the SynonymQuery. They have the same purpose and behave similarly.

Fixes #21633

* Fallback to SynonymQuery for blended terms on a single field
2016-11-28 14:14:01 +01:00
Isabel Drost-Fromm 2b24091361 Merge branch 'master' into feature/rank-eval 2016-11-28 11:45:29 +01:00
Yannick Welsch 7e198f0e41 Detect nodes being blocked by GC-disrupted node (#21797)
The disruption type LongGCDisruption simulates GCs on a node by suspending all the threads of that node. If the suspended threads are in a code section with shared JVM locks, however, it can prevent the other nodes from doing their thing. The class LongGCDisruption has a list of class names for which we know that this can occur. Whenever a test using the GC disruption type fails in mysterious ways, it becomes a long guessing game to find the offending class. This commit adds code to LongGCDisruption to automatically detect these situations, fail the test early and report the offending class and all relevant context.
2016-11-28 11:24:25 +01:00
Adrien Grand 243a788289 Fail to index fields with dots in field names when one of the intermediate objects is nested. (#21787)
Closes #21726
2016-11-28 09:57:32 +01:00
Clinton Gormley c1fa80d40f Log failure to connect to node at info instead of debug (#21809)
Closes #6468
2016-11-26 13:18:26 +01:00
Ali Beyad efba64d60a Removing unused AllocationExplanation class (#21805)
This commit removes the unused AllocationExplanation class.  The
RoutingAllocation class only created an empty instance of it and
never used it anywhere else.  The allocation explanations will be
encompassed in the various decision classes exposed via the cluster
allocation explain API.  Therefore, there is no reason to keep the
AllocationExplanation class.
2016-11-25 12:18:23 -05:00
Luca Cavanna 720b165350 Search shards to print out aliases array together with alias filter (#21784)
With #21738 we added an indices section to the search shards api, that will return the concrete indices hit by the request, and eventually the corresponding alias filter.

The java API returns the AliasFilter object, which holds the filter itself and an array of aliases that pointed to the index in the original request. The REST layer doesn't print out the aliases array though. This commit adds the aliases array as well and tests for this.
2016-11-25 10:58:06 +01:00
Simon Willnauer 9809760eb0 Fix settings diff generation for affix, list and group settings (#21788)
Group, List and Affix settings generate a bogus diff that turns the actual
diff into a string containing a json structure for instance:

```
"action" : {
  "search" : {
    "remote" : {
      "" : "{\"my_remote_cluster\":\"[::1]:60378\"}"
    }
  }
}
```

which make reading the setting impossible. This happens for instance
if a group or affix setting is rendered via `_cluster/settings?include_defaults=true`
This change fixes the issue as well as several minor issues with affix settings that
where not accepted as valid setting today.
2016-11-24 21:53:04 +01:00
Simon Willnauer 72ef6fa0d7 Handle spaces in `action.auto_create_index` gracefully (#21790)
Today if a comma-separated list is passed to action.auto_create_index
leading and trailing whitespaces are not trimmed but since the values are
index expressions whitespaces should be removed for convenience.

Closes #21449
2016-11-24 21:43:58 +01:00
markharwood aa60e5cc07 Aggregations - support for partitioning set of terms used in aggregations so that multiple requests can be done without trying to compute everything in one request.
Closes #21487
2016-11-24 15:10:46 +00:00
Luca Cavanna ac2aa56350 Cluster search shards improvements: expose ShardId, adjust visibility of some members (#21752)
* ClusterSearchShardsGroup to return ShardId rather than the int shard id

This allows more info to be retrieved, like the index uuid which is exposed through the ShardId object but was not available before

* Make ClusterSearchShardsResponse empty constructor public

This allows to receive such responses when sending ClusterSearchShardsRequests directly through TransportService (not using ClusterSearchShardsAction via Client), otherwise an empty response cannot be created unless the class that does it is in org.elasticsearch.action, admin.cluster.shards package

* adjust visibility of ClusterSearchShards members
2016-11-24 09:46:57 +01:00
Luca Cavanna d8c934a7fa Use index uuid as key in the alias filter map rather than the index name (#21749)
The index uuid is unique across multiple clusters, while the index name is not. Using the index uuid to look up filters in the alias filters map is better and will be needed for multi cluster search.
2016-11-24 09:43:42 +01:00
Christoph Büscher e1fe0dc462 Merge branch 'master' into feature/rank-eval 2016-11-24 08:57:26 +01:00
Luca Cavanna 6a16a60c7e Remove unused assignedReplicasIncludingRelocating from ShardsIterator interface (#21687) 2016-11-23 22:25:51 +01:00
Luca Cavanna 887ada4819 Move SearchTransportService and SearchPhaseController creation outside of TransportSearchAction constructor (#21754)
This commit makes sure that there is only one instance of the two services rather than one per transport action that uses it.
Also, we take their initialization out of guice's hands by binding it to a specific instance. Otherwise those two objects would get created within a constructor that is called by guice. That may cause problem for instance when throwing an exception from such constructors as guice tries all over again to re-initialize objects and fills up logs with stacktraces.
2016-11-23 22:18:50 +01:00
Luca Cavanna a84353d2d6 Don't carry ShardRouting around when not needed in AbstractSearchAsyncAction (#21753)
* replace ShardRouting argument in AbstractSearchAsyncAction#onFirstPhaseResult with more contained String nodeId

There is no need to pass in ShardRouting if the only info read from it is the current node id, the shard id can be read directly from the ShardIterator that's already provided as an argument.

* avoid creating a new ShardId when creating a SearchShardTarget in SnapshotsService
2016-11-23 22:18:02 +01:00
Jason Tedor 8416b16dfd Improve handling of unreleased versions
Today when handling unreleased versions for backwards compatilibity
support, we scatted version constants across the code base and add some
asserts to support removing these constants when the version in question
is actually released. This commit improves this situation, enabling us
to just add a single unreleased version constant that can be renamed
when the version is actually released. This should make maintenance of
these versions simpler.

Relates #21760
2016-11-23 15:49:05 -05:00
Luca Cavanna 033eece6d4 ShardSearchRequest to take ShardId constructor argument rather than the whole ShardRouting (#21750)
ShardSearchRequest was previously taking in the whole ShardRouting as a constructor argument while it only needs the ShardsId, changed that to carry over only the needed bits.
2016-11-23 15:34:55 +01:00
Ryan Ernst 10a945ae72 Plugins: Remove support for onModule (#21416)
All plugin extension points have been converted to pull based
interfaces. This change removes the infrastructure for the black-magic
onModule methods.
2016-11-22 23:12:14 -08:00
Ryan Ernst d8808210f1 Transport client: Fix remove address to actually work (#21743)
* Transport client: Fix remove address to actually work

The removeTransportAddress method of TransportClient removes the address
from the list of nodes that the client pings to sniff for nodes.
However, it does not remove it from the list of existing connected
nodes. This means removing a node is not possible, as long as that node
is still up.

This change removes the node from the connected nodes list before
triggering sampling (ie sniffing).  While the fix is simple, testing was
not because there were no existing tests for sniffing. This change also
modifies the mocks used by transport client unit tests in order to allow
mocking sniffing.
2016-11-22 22:50:11 -08:00
Igor Motov c7b69a0133 Add search task descriptions
Since we added ability to cancel searches it would be nice to see which searches we are actually cancelling.
2016-11-22 23:15:49 -05:00
Ryan Ernst 6940b2b8c7 Remove groovy scripting language (#21607)
* Scripting: Remove groovy scripting language

Groovy was deprecated in 5.0. This change removes it, along with the
legacy default language infrastructure in scripting.
2016-11-22 19:24:12 -08:00
Nik Everett 1791623700 Document `error_trace`
The `error_trace` parameter turns on the `stack_trace` field
in errors which returns stack traces.

Removes documentation for `camelCase` because it hasn't worked
in a while....

Documents the internal parameters used to render stack traces as
internal only.

Closes #21708
2016-11-22 19:16:07 -05:00
Jason Tedor c7b70fc770 Mark Security#addBindPermissions as private
This commit marks the method Security#addBindPermissions as private,
it's package-private visibility was not used anywhere.
2016-11-22 18:40:18 -05:00
Jason Tedor 41ae784a6f Refactor handling of bind permissions
This commit refactors the handling of bind permissions, which is in need
of a little cleanup. For example, in its current state, the code for
handling permissions for transport profiles is split across two
methods. This commit refactors this code hopefully making it easier to
work with in future changes. This change is mostly mechanical, no
functionality is changed.

Relates #21742
2016-11-22 18:39:14 -05:00
Jason Tedor 1576eaba25 Increase lower bound for random resolve timeout in test
The test UnicastZenPing#testResolveTimeout chooses a random resolve
timeout between 1ms and 100ms. Close to the lower bound, this is far too
short and the test races against the concurrent resolves executing
before the timeout elapses. This commit increases the timeout to
something that is far less likely to race, yet will not slow the test
down since we are not doing resolves against a real DNS service anyway.
Note that we still want a short resolve timeout since we are testing
whether or not timeouts really work here (by latching one of the
resolves to respond slowly).
2016-11-22 18:35:57 -05:00
Luca Cavanna db5a72774b Add indices and filter information to search shards api output (#21738)
Add indices and filter information to search shards api output

The search shards api returns info about which shards are going to be hit by executing a search with provided parameters: indices, routing, preference. Indices can also be aliases, which can also hold filters. The output includes an array of shards and a summary of all the nodes the shards are allocated on. This commit adds a new indices section to the search shards output that includes one entry per index, where each index can be associated with an optional filter in case the index was hit through a filtered alias.

This is relevant since we have moved parsing of alias filters to the coordinating node.

Relates to #20916
2016-11-22 23:00:25 +01:00
Nik Everett 29e68323a2 Clean up ScriptQuerySearchIT
Shorten line and remove forbidden API.

Relates to #21484
2016-11-22 16:23:33 -05:00
umeshdangat f37db2fe17 Support binary field type in script values (#21484)
Add ScriptDocValues.BytesRefs for reading binary fieldtype
2016-11-22 16:23:23 -05:00
Simon Willnauer a9a2753f0b Add a HostFailureListener to notify client code if a node got disconnected (#21709)
Today there is no way to get notified if a node is disconnected. Client code
must poll the TransportClient constantly to detect that a node is not connected
anymore in order to react and add new nodes or notify altering etc. For instance
if a hostname  gets resolved to an IP but that host is disconnected clients want
to reconnect by resolving the hostname again which is a common situation in cloud
environments.

Closes #21424
2016-11-22 20:46:28 +01:00
Jason Tedor 9dc65037bc Lazy resolve unicast hosts
Today we eagerly resolve unicast hosts. This means that if DNS changes,
we will never find the host at the new address. Moreover, a single host
failng to resolve causes startup to abort. This commit introduces lazy
resolution of unicast hosts. If a DNS entry changes, there is an
opportunity for the host to be discovered. Note that under the Java
security manager, there is a default positive cache of infinity for
resolved hosts; this means that if a user does want to operate in an
environment where DNS can change, they must adjust
networkaddress.cache.ttl in their security policy. And if a host fails
to resolve, we warn log the hostname but continue pinging other
configured hosts.

When doing DNS resolutions for unicast hostnames, we wait until the DNS
lookups timeout. This appears to be forty-five seconds on modern JVMs,
and it is not configurable. If we do these serially, the cluster can be
blocked during ping for a lengthy period of time. This commit introduces
doing the DNS lookups in parallel, and adds a user-configurable timeout
for these lookups.

Relates #21630
2016-11-22 14:17:04 -05:00
Clinton Gormley 3ff8faf514 Added version 2.4.2 and bwc indices 2016-11-22 19:45:59 +01:00
Yannick Welsch a44655763e Allow master to assign primary shard to node that has shard store locked during shard state fetching (#21656)
PR #19416 added a safety mechanism to shard state fetching to only access the store when the shard lock can be acquired. This can lead to the following situation however where a shard has not fully shut down yet while the shard fetching is going on, resulting in a ShardLockObtainFailedException. PrimaryShardAllocator that decides where to allocate primary shards sees this exception and treats the shard as unusable. If this is the only shard copy in the cluster, the cluster stays red and a new shard fetching cycle will not be triggered as shard state fetching treats exceptions while opening the store as permanent failures.

This commit makes it so that PrimaryShardAllocator treats the locked shard as a possible allocation target (although with the least priority).
2016-11-22 19:35:47 +01:00
Jay Modi 080d55a393 Rethrow ExecutionException from the loader to concurrent callers of Cache#computeIfAbsent
This commit clarifies the contract of Cache#computeIfAbsent so that an exception that occurs during the execution
of the loader is thrown to all callers. Prior to this commit, the first caller would get the ExecutionException
and other callers that called during the load execution would get null, which is confusing.
2016-11-22 13:24:15 -05:00
Luca Cavanna db8b2dceea Remove ignored type parameter in search_shards api (#21688)
The `type` parameter has always been accepted by the search_shards api, probably to make the api and its urls the same as search. Truth is that the type never had any effect, it's been ignored from day one while accepting it may make users think that we actually do something with it.

This commit removes support for the type parameter from the REST layer and the Java API. Backwards compatibility is maintained on the transport layer though.

The new added serialization test also uncovered a bug in the java API where the `ClusterSearchShardsRequest` could be created with no arguments, but the indices were required to be not null otherwise the request couldn't be serialized as `writeTo` would throw NPE. Fixed by setting a default value (empty array) for indices.
2016-11-22 17:22:33 +01:00
Jason Tedor 775638c281 Die with dignity on the Lucene layer
When a fatal error tragically closes an index writer, such an error
never makes its way to the uncaught exception handler. This prevents the
node from being torn down if an out of memory error or other fatal error
is thrown in the Lucene layer. This commit ensures that such events
bubble their way up to the uncaught exception handler.

Relates #21721
2016-11-22 11:21:24 -05:00
Jason Tedor 221caa1c5e Refactor handling for bad default permissions
This commit refactors the handling of bad default permissions that come
from the system security policy.

Relates #21735
2016-11-22 10:26:36 -05:00
Yannick Welsch 50e25912c8 Split main ClusterService method into smaller chunks #21666
Splits the main method in ClusterService into smaller chunks so that it's easier to understand and simpler to modify in subsequent PRs.
2016-11-22 12:20:53 +01:00
Yannick Welsch c521219b2f Adapt BWC layer checks for Exceptions to include v5.0.2 support
The PR #21694 was initially planned to go into v6.0.0 and v5.1.0. Due to another PR relying on this one though for backport to v5.0.2, #21694 must go to v5.0.2
as well. As such, the initial backward compatibility rules established by the PR must be changed to include v5.0.2 and above.
2016-11-22 12:02:56 +01:00
Christoph Büscher 1d3e58ab9f Merge branch 'master' into feature/rank-eval 2016-11-22 11:31:46 +01:00
Lee Hinman dd1012d570 Merge remote-tracking branch 'dakrone/fix-lenient-overriding' 2016-11-21 22:10:19 -07:00