Commit Graph

6421 Commits

Author SHA1 Message Date
Lee Hinman 3f77eacab1 Revert "Default `include_in_all` for numeric-like types to false"
This reverts commit 6666892038.
2016-09-28 07:07:46 -06:00
Jason Tedor 6013234560 Include complex settings in settings requests
Today when getting setting via an API like the cluster settings API,
complex settings are excluded (e.g.,
discovery.zen.ping.unicast.hosts). This commit adds these settings to
the output of such APIs.

Relates #20622
2016-09-27 20:25:03 -04:00
Jason Tedor 3c8ff45917 Add production warning for pre-release builds
This commit adds a usage warning when Elasticsearch is started with a
pre-release build.

Relates #20674
2016-09-27 20:13:12 -04:00
Lee Hinman e7ebfa0c69 Clean up confusing error message on unhandled endpoint
It currently returns something like:

```
"No feature for name [_siohgjoidfhjfihfg]"
```

Which is not the most understandable message, this changes it to be a
little more readable.

Resolves #10946
2016-09-27 08:42:47 -06:00
Lee Hinman e6e517a7a4 [TEST] Increase logging level in testDelayShards() 2016-09-27 03:01:38 -06:00
Jason Tedor a6e33494ab Provide error message when plugin id is missing
Today when executing the install plugin command without a plugin id, we
end up throwing an NPE because the plugin id is null yet we just keep
going (ultimatley we try to lookup the null plugin id in a set, the
direct cause of the NPE). This commit modifies the install command so
that a missing plugin id is detected and help is provided to the user.

Relates #20660
2016-09-26 08:09:15 -04:00
Nik Everett 370afa371b Make reindex-from-remote ignore unknown fields
reindex-from-remote should ignore unknown fields so it is mostly
future compatible. This makes it ignore unknown fields by adding an
option to `ObjectParser` and `ConstructingObjectParser` that, if
enabled, causes them to ignore unknown fields.

Closes #20504
2016-09-26 00:55:46 +02:00
Boaz Leskes ee76c1a5c9 Remove NoopGatewayAllocator in favor of a more realistic mock (#20637)
Many of our unit tests instantiate an `AllocationService`, which requires having a `GatewayAllocator`. Today almost all of our test use a class called `NoopGatewayAllocator` which does nothing, effectively leaving all shard assignments to the balanced allocator. This is sad as it means we test a system that behaves differently than our production logic in very basic things. For example, a started primary that is lost will be assigned to a node that didn't use to have it.

This PR removes `NoopGatewayAllocator` in favor of a new `TestGatewayAllocator` that inherits the standard `GatewayAllocator` and overrides shard information fetching to return information based on historical assignments the allocator has done. The only exception is `BalanceConfigurationTests` which does test only the balancer and I opted to not have it work around the `GatewayAllocator` being in it's way.
2016-09-25 20:15:30 +02:00
Ali Beyad ac1b13dde7 Changes the API of GatewayAllocator#applyStartedShards and (#20642)
Changes the API of GatewayAllocator#applyStartedShards and 
GatewayAllocator#applyFailedShards to take both a RoutingAllocation
and a list of shards to apply. This allows better mock allocators
to be created as being done in #20637.

Closes #20642
2016-09-23 09:31:46 -04:00
Ali Beyad 029fc909b5 Removes FailedRerouteAllocation and StartedRerouteAllocation
Removes the FailedRerouteAllocation class and StartedRerouteAllocation
class, as they were just wrappers for RerouteAllocation that stored
started and failed shards, but these started and failed shards can
be passed in directly to the methods that needed them, removing the
need for this wrapper class and extra level of indirection.

Closes #20626
2016-09-23 09:02:36 -04:00
Boaz Leskes 65356697ac IndexRoutingTable.initializeEmpty shouldn't override supplied primary RecoverySource (#20638)
When initializing a new index routing table, we make a decision where the primary shards should be recovered from. This can be an empty folder for new indices, a set of specific allocation ids for old indices or a snapshot. We currently allow callers of `IndexRoutingTable.initializeEmpty` to supply the source but also set it automatically if null is given. Sadly the current logic is reusing the supplied parameter to store the result of the automatic decision. This is flawed if some of the decision should be *different* between the different index shard (as the first decision that is maid sticks).

 This commit fixes this but also simplifies the API to always make an automatic decision.

This was discovered while working on #20637 which strengthens the testing infra and caused this to bubble up. I put it as a separate commit to make sure it is not lost as part of a bigger test only PR.
2016-09-23 13:17:54 +02:00
Simon Willnauer fe1803c957 Remove AnalysisService and reduce it to a simple name to analyzer mapping (#20627)
Today we hold on to all possible tokenizers, tokenfilters etc. when we create
an index service on a node. This was mainly done to allow the `_analyze` API to
directly access all these primitive. We fixed this in #19827 and can now get rid of
the AnalysisService entirely and replace it with a simple map like class. This
ensures we don't create a gazillion long living objects that are entirely useless since
they are never used in most of the indices. Also those objects might consume a considerable
amount of memory since they might load stopwords or synonyms etc.

Closes #19828
2016-09-23 08:53:50 +02:00
Tanguy Leroux ab2e067ef5 Make ByteSizeUnit implements Writeable (#20557)
This commit makes ByteSizeUnit implement Writeable.
2016-09-22 14:42:13 +02:00
Jay Modi 0573e03aa1 Pass classpath plugins to tribe nodes
When testing tribe nodes in an integration test, we should pass the classpath
plugins of the node down to the tribe client nodes. Without this the tribe client
nodes could be prevented from communicating with the tribes.
2016-09-22 07:29:07 -04:00
Ali Beyad 86c3bdb8a5 Fixes a test bug for the DeprecationRestHandler where an empty string
could be randomly produced as a header to test against, which is rejected
by header validation as an invalid header.
2016-09-21 13:07:40 -04:00
Yannick Welsch d55bd707a3 Update incoming recoveries stats when shadow replica is reinitialized (#20612)
When an active shadow replica is reinitialized during primary promotion, the recovery stats that are used by the allocation decider settings `cluster.routing.allocation.node_concurrent_recoveries` and `cluster.routing.allocation.node_concurrent_incoming_recoveries` have to be updated.
2016-09-21 18:13:26 +02:00
Adrien Grand 45469a5570 Native scripts should be created once per index, not per segment. (#20609)
If your native script needs to do some heavy computation on initialization,
the fact that we create a new one for every segment rather than for the whole
index could have a negative performance impact.
2016-09-21 17:39:43 +02:00
Yannick Welsch bf5d425ab9 Fix wrong logger usages
These misusages were found by the logger usage checker that was re-enabled in the previous commit.
2016-09-21 14:45:28 +02:00
Simon Willnauer 0151974500 `_flush` should block by default (#20597)
This commit changes the default behavior of `_flush` to block if other flushes are ongoing.
This also removes the use of `FlushNotAllowedException` and instead simply return immediately
by skipping the flush. Users should be aware if they set this option that the flush might or might
not flush everything to disk ie. no transactional behavior of some sort.

Closes #20569
2016-09-21 14:20:24 +02:00
Simon Willnauer 6dc03ecb10 Remove unused Translog#read method (#20598)
Translog#read is a left-over from realtime-get that allows to read
from an arbitrary location in the transaction log. This method is unused
and can be replaced with snapshots in tests.
2016-09-21 14:19:49 +02:00
Boaz Leskes b3e5e6a0ba `index.routing.allocation.initial_recovery` limits replica allocation (#20589)
`index.routing.allocation.initial_recovery` is used with index shrinking to make sure the new index's primary is assigned to the node that holds a copy of each of the source index shards. Sadly with the introduction of `RecoverySource` a regression was introduced that limits the allocation of replicas of the new index.
2016-09-21 12:40:37 +02:00
Nik Everett 92417c6ae3 Better explain Plugin (#20573)
Adds some javadoc with more explanation on how to extend Plugin and
why we have all these `@Deprecated public final` `onModule` methods.

Closes #20564
2016-09-20 11:20:13 -04:00
Jason Tedor 12234c067a Ensure logging is initialized in CLI tools
Today when CLI tools are executed, logging statements can intentionally
or unintentionally be executed when logging is not configured. This
leads to log messages that the status logger is not configured. This
commit reworks logging configuration for CLI tools so that logging is
always configured.

Relates #20575
2016-09-20 08:28:27 -04:00
Tanguy Leroux 7645abaad9 Remove duplicate methods in ByteSizeValue (#20560)
This commit removes `ByteSizeValue`'s methods that are duplicated (ex: `mbFrac()` and `getMbFrac()`) in order to only keep the `getN` form.
    
It also renames `mb()` -> `getMb()`, `kb()` -> `getKB()` in order to be more coherent with the `ByteSizeUnit` method names.
2016-09-20 14:07:23 +02:00
Alexander Lin d31a8e6558 Provides a cat api endpoint for templates. (#20545)
Adds a cat api endpoint: /_cat/templates and its more specific version, /_cat/templates/{name}.

It looks something like:

$ curl "localhost:9200/_cat/templates?v"
name                  template     order version
sushi_california_roll *avocado*    1     1
pizza_hawaiian        *pineapples* 1
pizza_pepperoni       *pepperoni*  1

The specified version (only allows * globs) looks like:

$ curl "localhost:9200/_cat/templates/pizza*"
name            template     order version
pizza_hawaiian  *pineapples* 1
pizza_pepperoni *pepperoni*  1

Partially specified columns:

$ curl "localhost:9200/_cat/templates/pizza*?v=true&h=name,template"
name            template
pizza_hawaiian  *pineapples*
pizza_pepperoni *pepperoni*

The help text:

$ curl "localhost:9200/_cat/templates/pizza*?help"
name     | n | template name
template | t | template pattern string
order    | o | template application order number
version  | v | version

Closes #20467
2016-09-20 10:40:23 +02:00
Tanguy Leroux 38dce6384f Improve TribeIT tests
This commit adds a new test TribeIT#testClusterStateNodes() to verify that the tribe node correctly reflects the nodes of the remote clusters it is connected to.

It also changes the existing tests so that they really use two remote clusters now.
2016-09-20 10:31:32 +02:00
Luca Cavanna b7314c8721 fix IndexResponse#toString to print out shards info (#20562)
IndexResponse#toString method outputs an error caused by the shards object needing to be wrapped into another object. It is fixed by calling a different variant of Strings.toString(XContent) which accepts a second boolean argument that makes sure that a new object is created before outputting ShardInfo. I didn't change ShardInfo#toString directly as whether it needs a new object or not very much depends on where it is printed out. IndexResponse seemed a specific case as the rest of the info were not json, hence the shards object was the first one, but it is usually not the case.
2016-09-20 08:57:49 +02:00
Ryan Ernst b77c16086c Convert old download links to unified release urls (#20574)
With the unified release process across the elastic stack, download
links for all products are changing. This change updates docs referring
to the old download and packages urls.

Note that this change also updates the plugin installation command as
the url for downloads is being changed to be consistent with that for
packages (both plural).
2016-09-19 18:10:45 -07:00
Jason Tedor 05b4e0c0e3 Add serial collector bootstrap check
The serial collector is not suitable for running with a server
application like Elasticsearch and can decimate performance and lead to
cluster instability. This commit adds a bootstrap check to prevent usage
of the serial collector when Elasticsearch is running in production
mode.

Relates #20558
2016-09-19 20:25:50 -04:00
Jason Tedor aa510159c2 Avoid unnecessary creation of prefix loggers
Today when acquiring a prefix logger for a logger info stream, we obtain
a new prefix logger per invocation. This can lead to contention on the
markers lock in the constructor of PrefixLogger. Usually this is not a
problem (because the vast majority of callers hold on to the logger they
obtain). Unfortunately, under heavy indexing with multiple threads, the
contention on the lock can be devastating. This commit modifies
LoggerInfoStream to hold on to the loggers it obtains to avoid
contending over the lock there.

Relates #20571
2016-09-19 20:22:07 -04:00
Ryan Ernst 85b8f29415 Build: Remove old maven deploy support (#20403)
* Build: Remove old maven deploy support

This change removes the old maven deploy that we have in parallel to
maven-publish, and makes maven-publish fully work with publishing to
maven local. Using `gradle publishToMavenLocal` should be used to
publish to .m2.

Note that there is an unfortunate hack that means for
zip artifacts we must first create/publish a dummy pom file, and then
follow that with the real pom file. It would be nice to have the pom
file contains packaging=zip, but maven central then requires sources and
javadocs. But our zips are really just attached artifacts, so we already
set the packaging type to pom for our zip files. This change just works
around a limitation of the underlying maven publishing library which
silently skips attached artifacts when the packaging type is set to pom.

relates #20164
closes #20375

* Remove unnecessary extra spacing
2016-09-19 15:10:41 -07:00
Simon Willnauer ee8d14798f Unguice Transport and friends (#20526)
This change removes all guice interaction from Transport, HttpServerTransport,
HttpServer and TransportService. All these classes as well as their subclasses
or extended version configured via plugins are now created by using plain old
bloody java constructors. YAY!
2016-09-19 22:10:47 +02:00
Simon Willnauer bd93b3dce6 Call onNewEngine before the engine is actually published 2016-09-19 17:00:17 +02:00
Simon Willnauer 817d55cf93 Take refresh IOExceptions into account when catching ACE in InternalEngine (#20546)
Since #19975 we are aggressively failing with AssertionError when we catch an ACE
inside the InternalEngine. We treat everything that is neither a tragic even on
the IndexWriter or the Translog as a bug and throw an AssertionError. Yet, if the
engine hits an IOException on refresh of some sort and the IW doesn't realize it since
it's not fully under it's control we fail he engine but neither IW nor Translog are marked
as failed by tragic event while they are already closed.
This change takes the `failedEngine` exception into account and if it's set we know
that the engine failed by some other even than a tragic one and can continue.

This change also uses the `ReferenceManager#RefreshListener` interface in the engine rather
than it's concrete implementation.

Relates to #19975
2016-09-19 15:15:51 +02:00
Jason Tedor 7874463db0 Merge pull request #20553 from jasontedor/node-attributes
Node attributes and REST API spec README
2016-09-19 08:31:09 -04:00
Boaz Leskes 2ee9ab25d9 Remove `RoutingAllocation.Result` (#20538)
Currently all the reroute-like methods of `AllocationService` return a result object of type `RoutingAllocation.Result`. The result object contains the new `RoutingTable` and `MetaData` plus an indication whether those were changed. The caller is then responsible of updating a cluster state with these. These means that things can easily go wrong and one can take one of these but not the other causing inconsistencies. We already have a utility method on the `ClusterState` builder that does but no one forces you to do so. Also 99% of the callers do the same thing: i.e., check if the result was changed and if so update the very same cluster state that was passed to `AllocationService`.  This PR folds this pattern into `AllocationService` and changes almost all it's methods to return a new cluster state (potentially the original one).  This saves some 500 lines of code.

The one exception here is the reroute API which executes allocation commands and potentially returns an explanation as well (next to the routing table and metadata). That API now returns a `CommandsResult` object which encapsulate a cluster state and the explanation.
2016-09-19 13:54:35 +02:00
Jason Tedor b456823434 Fix leftover node attributes usage
Previously node attributes could be set via node.* but this now requires
using node.attr.*. This commit fixes some leftover usages of the old
way.
2016-09-19 07:45:31 -04:00
javanna 7097f4943c [TEST] delete specific index boost serialization test, already covered by testSerialization
indexBoost is already randomly set, we don't need a specific test for it in SearchSourceBuilderTests
2016-09-16 21:03:27 +02:00
javanna 20badcdbab [TEST] set back the visibility of our search ext plugin in FetchSubPhasePluginIT to private
This plugin is not used in other tests anymore, it should be private to make sure its usage doesn't spread again
2016-09-16 21:03:27 +02:00
javanna 4c726311e2 [TEST] introduce test plugin to inject random search ext elements in search request tests
A few of our unit tests generate a random search request body nd run tests against it. The source can optionally contain ext elements under the ext sections, which can be parsed by plugins. With this commit we introduce a plugin so that the tests don't use the one from FetchSubPhasePluginIT anymore. They rather generate multiple search ext elements. The plugin can parse and deal with all those. This extends the test coverage as we may have multiple elements with random names.

Took the chance to introduce a common test base class for search requests, called AbstractSearchTestCase, given that the setup phase is the same for all three tests around search source. Then we can have the setup isolated to the base class and the subclasses relying on it.

Closes #17685
2016-09-16 21:03:27 +02:00
Luca Cavanna 629e2b2aff Throw error if query element doesn't end with END_OBJECT (#20528)
* Throw error if query element doesn't end with END_OBJECT

Followup to #20515 where we added validation that after we parse a query within a query element, we should not get a field name. Truth is that the only token allowed at that point is END_OBJECT, as our DSL allows only one single query within the query object:

```
{
  "query" : {
    "term" : { "field" : "value" }
  }
}
```

We can then check that after parsing of the query we have an end_object that closes the query itself (which we already do). Following that we can check that the query object is immediately closed, as there are no other tokens that can be present in that position.

Relates to #20515
2016-09-16 21:02:12 +02:00
Nik Everett 697adfb3c4 Fix up tasks integ test
I'd made some mistakes that hadn't caused the test to fail but did
slow it down and partially invalidate some of the assertions. This
fixes those mistakes.
2016-09-16 12:18:41 -04:00
Jim Ferenczi d0f4bc16ca Fix FieldStats deserialization of `ip` field (#20522)
* Fix FieldStats deserialization of `ip` field

Add missing readBytes in `ip` field deserialization
Add (de)serialization tests for all types
This change also removes the ability to set FieldStats.minValue or FieldStats.maxValue to null.
This is not required anymore since the stats are built on fields with values only.

Fixes #20516
2016-09-16 16:15:46 +02:00
Nicholas Knize 01a6b7c408 [TEST] Refactor Geo test names to follow naming and style convention
This commit closes a stale issue where GeoJson parsing tests used a combination of underscore and camel case.

closes #8998
2016-09-16 09:13:38 -05:00
Simon Willnauer 3aabda6752 Ensure elasticsearch doesn't start with unuspported indices (#20514)
If an index was created with pre 2.0 we should not treat it as supported
even if all segments have been upgraded to a supported lucene version.

Closes #20512
2016-09-16 16:11:53 +02:00
Simon Willnauer f5daa165f1 Remove ability to plug-in TransportService (#20505)
TransportService is such a central part of the core server, replacing
it's implementation is risky and can cause serious issues. This change removes the ability to
plug in TransportService but allows registering a TransportInterceptor that enables
plugins to intercept requests on both the sender and the receiver ends. This is a commonly used
and overwritten functionality but encapsulates the custom code in a contained manner.
2016-09-16 09:47:53 +02:00
Boaz Leskes 577dcb3237 Add current cluster state version to zen pings and use them in master election (#20384)
During a networking partition, cluster states updates (like mapping changes or shard assignments)
are committed if a majority of the masters node received the update correctly. This means that the current master has access to enough nodes in the cluster to continue to operate correctly. When the network partition heals, the isolated nodes catch up with the current state and get the changes they couldn't receive before. However, if a second partition happens while the cluster
is still recovering from the previous one *and* the old master is put in the minority side, it may be that a new master is elected which did not yet catch up. If that happens, cluster state updates can be lost.

This commit fixed 95% of this rare problem by adding the current cluster state version to `PingResponse` and use them when deciding which master to join (and thus casting the node's vote). 

Note: this doesn't fully mitigate the problem as a cluster state update which is issued concurrently with a network partition can be lost if the partition prevents the commit message (part of the two phased commit of cluster state updates) from reaching any single node in the majority side *and* the partition does allow for the master to acknowledge the change. We are working on a more comprehensive fix but that requires considerate work  and is targeted at 6.0.
2016-09-15 23:39:11 +02:00
Simon Willnauer 1e2ef192c9 [TEST] Reduce the number of docs per indexRandom in FieldSortIT#testIssue6614 2016-09-15 22:56:26 +02:00
Areek Zillur 8c12b7c3b6 Merge pull request #20515 from areek/fix/query_validation
Fix silently accepting malformed queries
2016-09-15 16:53:17 -04:00
Areek Zillur bdad62a2f8 Fix silently accepting malformed queries
Currently, we silently accept malformed query where more
than one key is defined at the top-level for query object.
If all the keys have a valid query body, only the last query
is executed, besides throwing off parsing for additional suggest,
aggregation or highlighting defined in the search request.

This commit throws a parsing exception when we encounter a query
with multiple keys.

closes #20500
2016-09-15 16:32:28 -04:00
Boaz Leskes 2233d48235 add a reduced TCP_CONNECT_TIMEOUT setting to DiscoveryWithServiceDisruptionsIT
The default of 30s causes some tests to timeout when running ensureGreen and similar. This is because network delays simulation blocks connect until either the connect timeout expires or the disruption configured time stops. We do *not* immediately connect when the disruption is stopped.
2016-09-15 21:50:24 +02:00
Areek Zillur 16ed2fb423 [TEST] ensure context filtering with valid utf-8 characters work with completion suggester 2016-09-15 14:38:52 -04:00
Luca Cavanna 37489c3274 Add clusterUUID to RestMainAction output (#20503)
Add clusterUUID to RestMainAction output

GET / now returns the clusterUUID as well as part of its output for monitoring purposes
2016-09-15 16:25:17 +02:00
gfyoung b03c807368 Rename service.bat to elasticsearch-service.bat (#20496)
Closes gh-17528.
2016-09-15 15:53:44 +02:00
Jason Tedor 7132fcd7ac Give useful error message if log config is missing
Today when starting Elasticsearch without a Log4j 2 configuration file,
we end up throwing an array index out of bounds exception. This is
because we are passing no configuration files to Log4j. Instead, we
should throw a useful error message to the user. This commit modifies
the Log4j configuration setup to throw a user exception if no Log4j
configuration files are present in the config directory.

Relates #20493
2016-09-15 07:44:05 -04:00
Boaz Leskes a5f03b4bc5 testCanNotPublishWithoutMinMastNodes could time out if disruption hit just before cluster was fully formed
The test uses a NetworkDelay that drops requests and slows down connecting. Next to that it disable node fault detection to make sure nodes are not removed before we check our publishing. Sadly that can lead to huge slow downs if the disruption hits while a node is still pinging (and tries to connect, which is slowed down). Instead we can start the disruption on the cluster state thread, making sure the result of fault detection won't be processed before we publish
2016-09-15 08:50:58 +02:00
Lee Hinman d9f83a6d74 Change 'norelease' in MappedFieldType to 'TODO'
This should not block any sort of release, it is only to be more
protective towards fields not having any settings changed.
2016-09-14 14:37:02 -06:00
Lee Hinman 1e39a12fb0 Merge remote-tracking branch 'dakrone/remove-idx-shard-count-norelease' 2016-09-14 14:10:49 -06:00
Lee Hinman e462edc6d6 Validate index settings differently when registering index template
This was actually a byproduct of trying to remove a //norelease for
index shard setting validation in MetaDataIndexService. This //norelease
is now removed. Previously this check was *only* used by the template
service, so we validated twice, once in the Settings infrastructure and
once when actually creating the index. We now instead use the Settings
infrastructure to validate the settings for shard count.
2016-09-14 13:25:34 -06:00
Simon Willnauer 17ddee7011 Remove TransportService#registerRequestHandler leniency (#20469)
`TransportService#registerRequestHandler` allowed to register
handlers more than once and issues an annoying warn log message when
this happens. This change simple throws an exception to prevent regsitering
the same handler more than once. This commit also removes the ability
to remove request handlers.

Relates to #20468
2016-09-14 20:32:29 +02:00
javanna e7e27d9ffa [TEST] remove unused currentTypes variable from SearchSourceBuilderTests 2016-09-14 19:33:23 +02:00
Simon Willnauer bb6e7eeb7a [TEST] Don't use transport client if we are blocking internal actions we might run into disconnects 2016-09-14 17:50:14 +02:00
Simon Willnauer b35f7446ce Remove unused imports 2016-09-14 17:44:45 +02:00
Simon Willnauer d402ca0dd7 Remove poor-mans compression in InternalSearchHit and friends (#20472)
We still use some crazy poor mans compression in InternalSearchHit that
uses a thread local and an unordered map as a lookup table if requested.
Stuff like this should be handled by compression on the transport layer
rather than in-line in the serialization code. This code is complex enough.
2016-09-14 15:25:25 +02:00
Simon Willnauer c1e84618a6 Only try to read new segments info if we really flushed the index (#20474)
There is no reason to read the current segments info unless we flushed
/ committed the lucene index.
2016-09-14 15:23:17 +02:00
Boaz Leskes 74fc074e5e fix styling 2016-09-14 10:52:10 +02:00
Simon Willnauer a1cd6be777 Don't register SearchTransportService handlers more than once (#20468)
This utility class is used in 3 places while we only need to register
the handlers once per node. Otherwise we will see nasty `WARN` logs like:
`registered two transport handlers for action indices:data/read/search[phase/fetch/id/scroll]...`

This change will only register handlers inside the main TransportSearchAction.
2016-09-14 10:34:40 +02:00
Simon Willnauer 89640965d2 Unguice SearchModule (#20456)
After this change SearchModule doesn't subclass AbstractModule anymore and all wiring
happens in `Node.java`. As a side-effect several tests don't need a guice injector anymore.
2016-09-14 10:07:53 +02:00
Jason Tedor 7560101ec7 Complete Elasticsearch logger names
This commit modifies the logger names within Elasticsearch to be the
fully-qualified class name as opposed removing the org.elasticsearch
prefix and dropping the class name. This change separates the root
logger from the Elasticsearch loggers (they were equated from the
removal of the org.elasticsearch prefix) and enables log levels to be
set at the class level (instead of the package level).

Relates #20457
2016-09-13 22:46:54 -04:00
Jason Tedor 0eff7daf5b Fix logging hierarchy configs
Today when setting the logging level via the command-line or an API
call, the expectation is that the logging level should trickle down the
hiearchy to descendant loggers. However, this is not necessarily the
case. For example, if loggers x and x.y are already configured then
setting the logging level on x will not descend to x.y. This is because
the logging config for x.y has already been forked from the logging
config for x. Therefore, we must explicitly descend the hierarchy when
setting the logging level and that is what this commit does.

Relates #20463
2016-09-13 22:46:14 -04:00
Ali Beyad 4431720c3d File-based discovery plugin (#20394)
This commit introduces a new plugin for file-based unicast hosts
discovery. This allows specifying the unicast hosts participating
in discovery through a `unicast_hosts.txt` file located in the
`config/discovery-file` directory. The plugin will use the hosts 
specified in this file as the set of hosts to ping during discovery.

The format of the `unicast_hosts.txt` file is to have one host/port
entry per line. The hosts file is read and parsed every time
discovery makes ping requests, thus a new version of the file that
is published to the config directory will automatically be picked
up.

Closes #20323
2016-09-13 20:52:39 -04:00
Nicholas Knize 87b06c75b0 [TEST] Fix geo_point backcompat tests
This commit fixes the following geo_point bwc tests:

* GeoDistanceIT to test deprecated GeoDistanceRangeQuery on legacy indexes only.
* ExternalFieldMapperTests to correctly handle LatLonPoint type
* GeoPointFieldMapperTests to correctly test stored geo_point fields
2016-09-13 16:27:55 -05:00
Nicholas Knize 821004d5cd [TEST] Refactor LegacyGeohashMappingGeoPointTests to 2.x indices only
These tests should only exist to ensure backcompat with 2.x indices.
2016-09-13 15:27:39 -05:00
Jim Ferenczi 1764ec56b3 Fixed naming inconsistency for fields/stored_fields in the APIs (#20166)
This change replaces the fields parameter with stored_fields when it makes sense.
This is dictated by the renaming we made in #18943 for the search API.

The following list of endpoint has been changed to use `stored_fields` instead of `fields`:
* get
* mget
* explain

The documentation and the rest API spec has been updated to cope with the changes for the following APIs:
* delete_by_query
* get
* mget
* explain

The `fields` parameter has been deprecated for the following APIs (it is replaced by _source filtering):
* update: the fields are extracted from the _source directly.
* bulk: the fields parameter is used but fields are extracted from the source directly so it is allowed to have non-stored fields.

Some APIs still have the `fields` parameter for various reasons:
* cat.fielddata: the fields paramaters relates to the fielddata fields that should be printed.
* indices.clear_cache: used to indicate which fielddata fields should be cleared.
* indices.get_field_mapping: used to filter fields in the mapping.
* indices.stats: get stats on fields (stored or not stored).
* termvectors: fields are retrieved from the stored fields if possible and extracted from the _source otherwise.
* mtermvectors:
* nodes.stats: the fields parameter is used to concatenate completion_fields and fielddata_fields so it's not related to stored_fields at all.

Fixes #20155
2016-09-13 20:54:41 +02:00
Jason Tedor fbe27664a6 Fix prefix logging
Today we add a prefix when logging within Elasticsearch. This prefix
contains the node name, and index and shard-level components if
appropriate.

Due to some implementation details with Log4j 2 , this does not work for
integration tests; instead what we see is the node name for the last
node to startup. The implementation detail here is that Log4j 2 there is
only one logger for a name, message factory pair, and the key derived
from the message factory is the class name of the message factory. So,
when the last node starts up and starts setting prefixes on its message
factories, it will impact the loggers for the other nodes.

Additionally, the prefixes are lost when logging an exception. This is
due to another implementation detail in Log4j 2. Namely, since we log
exceptions using a parameterized message, Log4j 2 decides that that
means that we do not want to use the message factory that we have
provided (the prefix message factory) and so logs the exception without
the prefix.

This commit fixes both of these issues.

Relates #20429
2016-09-13 14:46:34 -04:00
Nicholas Knize 1a60e1c3d2 Update docs for LatLonPoint cut over
This commit removes documentation for:

* geohash cell query
* lat_lon parameter
* geohash parameter
* geohash_precision parameter
* geohash_prefix parameter

It also updates failing tests that reference these parameters for backcompat.
2016-09-13 12:18:21 -05:00
Nicholas Knize ef926894f4 Cut over geo_point field and queries to new LatLonPoint type
This commit cuts over geo_point fields to use Lucene's new point-based LatLonPoint type for indexes created in 5.0. Indexes created prior to 5.0 continue to use their respective encoding type. Below is a description of the changes made to support the new encoding type:

* New indexes use a new LatLonPointFieldMapper which provides a parse method for the new type
* The new LatLonPoint parse method removes support for lat_lon and geohash parameters
* Backcompat testing for deprecated lat_lon and geohash parameters is added to all unit and integration tests
* LatLonPointFieldMapper provides DocValues support (enabled by default) which uses Lucene's new LatLonDocValuesField type
* New LatLonPoint field data classes are added for aggregation support (wraps LatLonPoint's Numeric Doc Values)
* MultiFields use the geohash as the string value instead of the lat,lon string making it easier to perform geo string queries on the geohash instead of a lat,lon comma delimited string.

Removed Features:

* With the removal of geohash indexing, GeoHashCellQuery support is removed for all new indexes (still supported on existing indexes)
* LatLonPoint does not support a Distance Range query because it is super inefficient. Instead, the geo_distance_range query should be accomplished using either the geo_distance aggregation, sorting by descending distance on a geo_distance query, or a boolean must not of the excluded distance (which is what the distance_range query did anyway).

TODO:

* fix/finish yaml changes for plugin and rest integration tests
* update documentation
2016-09-13 12:17:36 -05:00
javanna e0074ee9d4 [TEST] fix MultiMatchQueryIT random docs generation so that they don't interfere in score tests
When generating random bogus documents, it could happen that they contain both the terms "the" and "ultimate", which would match the query "the ultimate" better than all the other non bogus documents, which would cause testCrossFieldMode to fail. "the" is a term that's relatively likely to be randomly generated given its length; we can simply increase the minimum length of randomly generated terms to 5, so that there are no collisions, as "the" cannot be generated anymore (nor can "ultimate" as the lenght doesn't go up to 8).

Also made some assertions more accurate to check how many hits match a query rather than checking only that the first or second hits are there.

Closes #18873
2016-09-13 18:25:53 +02:00
Nik Everett afbd7cbeb8 Rework the basic IT for GETing running tasks
This integ test relied on the false assumption that
`MockTaskManagerListener#onTaskUnregistered` was called *before* the
task was unregistered. It is in fact called after the task is unregistered.
This mistake led the test to *rarely* miss the task it was looking
for and fail.

Found by https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+5.x+multijob-unix-compatibility/os=ubuntu/4/consoleText
2016-09-13 11:59:50 -04:00
Simon Willnauer c84bc25500 Cleanup version constant for unsupported version in QuerySearchResult 2016-09-13 17:21:04 +02:00
Nik Everett 7888dbfb31 Add second test case for two fields in range query
In this test one field is a number and the other is a date.

Closes #20447
2016-09-13 09:26:29 -04:00
Britta Weber 444c4f1af8 remove workaround for highlighter bug with geo queries (#20418)
This has been fixed in Lucene
https://issues.apache.org/jira/browse/LUCENE-7293
This commit also adds the tests from #20412
2016-09-13 14:59:56 +02:00
Tanguy Leroux 6090c51fc5 Add quiet option to disable console logging (#20422)
This commit adds a -q/--quiet option to Elasticsearch so that it does not log anything in the console and closes stdout & stderr streams. This is useful for SystemD to avoid duplicate logs in both journalctl and /var/log/elasticsearch/elasticsearch.log while still allows the JVM to print error messages in stdout/stderr if needed.

closes #17220
2016-09-13 14:08:24 +02:00
Jason Tedor c7bfbe3e69 Add health status parameter to cat indices API
This commit adds a health status parameter to the cat indices API for
filtering on indices that match the specified status (green|yellow|red).

Relates #20393
2016-09-13 07:57:18 -04:00
Michael Nitschinger 9ee6624fd1 Network: Allow to listen on virtual interfaces.
Previously when trying to listen on virtual interfaces during
bootstrap the application would stop working - the interface
couldn't be found by the NetworkUtils class.

The NetworkUtils utilize the underlying JDK NetworkInterface
class which, when asked to lookup by name only takes physical
interfaces into account, failing at virtual (or subinterfaces)
ones (returning null).

Note that when interating over all interfaces, both physical and
virtual ones are taken into account.

This changeset asks for all known interfaces, iterates over them
and matches on the given name as part of the loop, allowing it
to catch both physical and virtual interfaces.

As a result, elasticsearch can now also serve on virtual
interfaces.

A test case has been added which  makes sure that all
iterable interfaces can be found by their respective name.

Note that this PR is a second iteration over the previously
merged but later reverted #19537 because it causes tests
to fail when interfaces are down. The test has been modified
to take this into account now.

Closes #17473
Closes #19568
Relates #19537
2016-09-13 13:40:09 +02:00
javanna 7894eba2b3 [TEST] add test for match query parsing error when providing an array of terms
Match query throws parsing errors when an array of terms is provided, we should test that to make sure this behaviour doesn't change.

Relates to #15741
2016-09-13 12:46:35 +02:00
Boaz Leskes 10dcfa3304 Fix concurrency issues between cancelling a relocation and marking shard as relocated (#20443)
Once a primary is marked as relocated, we can not safely move it back to started (as we have no way of waiting on inflight operations that are performed on the target primary). If the master cancels the relocation in that state, we fail the primary. Sadly, there is a racing condition between the `updateRoutingEntry` method (which is called when the relocation is cancelled by the master) and the `relocated` method. That racing condition can leave the shard as marked "relocated" but have the routing entry not reflect the target relocation. This in turns causes NPEs in TransportReplicationAction:

```
java.util.Objects requireNonNull Objects.java 203
org.elasticsearch.action.support.replication.TransportReplicationAction$ConcreteShardRequest <init> TransportReplicationAction.java 982
```

Sadly, once we end up in this state, we will never recover.

This commit fixes that race condition by making sure `updateRoutingEntry` acquires the mutex when checking for the relocated status.  While at it, I also tightened up the code and added lots of assertions/hard checks.
2016-09-13 12:44:40 +02:00
makeyang 1ae8d6123f Add node name to decider trace logging (#20437)
Adds the entire DiscoveryNode object to the trace log in AllocationDeciders.
The allocation decider logging at TRACE level can sometimes be helpful to determine why a shard is not getting allocated on specific nodes. Currently, we only log the node id for these messages. It will be helpful to also include the node name (esp. when dealing with a lot of nodes in the cluster).
2016-09-13 11:17:39 +02:00
Lee Hinman 3439796df3 Merge branch 'pr/18683' 2016-09-12 16:24:09 -06:00
Lee Hinman 44278db1bc Merge pull request #20433 from dakrone/remove-cluster-name-folder-fallback
No longer allow cluster name in data path
2016-09-12 17:01:49 -05:00
Lee Hinman 94625d74e4 No longer allow cluster name in data path
In 5.x we allowed this with a deprecation warning. This removes the code
added for that deprecation, requiring the cluster name to not be in the
data path.

Resolves #20391
2016-09-12 15:47:01 -06:00
Simon Willnauer 686994ae2d Deguice SearchService and friends (#20423)
This change removes the guice dependency handling for SearchService and
several related classes like SearchTransportController and SearchPhaseController.
The latter two now have package private constructors and dependencies like FetchPhase
are now created by calling their constructors explicitly. This also cleans up several users
of the DefaultSearchContext and centralized it's creation inside SearchService.
2016-09-12 22:42:55 +02:00
Boaz Leskes 7f92971f26 remove assumeX methods from IndexShardTests
The cause early termination of tests, which means we don't clean up and close shards, but also don't cause a failure. This in turns makes TestRuleTemporaryFilesCleanup fail on windows (because it does try to clean up, but the files are referenced). Getting stuff like:

 ```
 >    C:\jenkins\workspace\es_core_master_windows-2012-r2\core\build\testrun\test\J3\temp\org.elasticsearch.index.shard.IndexShardTests_68B5E1103D78A58B-001\tempDir-006\indices\_na_\0\translog\translog-1.tlog: java.nio.file.AccessDeniedException: C:\jenkins\workspace\es_core_master_windows-2012-r2\core\build\testrun\test\J3\temp\org.elasticsearch.index.shard.IndexShardTests_68B5E1103D78A58B-001\tempDir-006\indices\_na_\0\translog\translog-1.tlog
 ```
2016-09-12 22:29:42 +02:00
Ali Beyad b1e87aa13c Split allocator decision making from decision application (#20347)
Splits the PrimaryShardAllocator and ReplicaShardAllocator's decision
making for a shard from the implementation of that decision on the
routing table. This is a step toward making it easier to use the same
logic for the cluster allocation explain APIs.
2016-09-12 16:21:39 -04:00
Luca Cavanna 119d198cc5 Merge pull request #20424 from javanna/enhancement/error_fetch_source_disabled
Throw error when trying to fetch fields from source and source is disabled
2016-09-12 18:36:43 +02:00
Boaz Leskes b08352047d Introduce IndexShardTestCase (#20411)
Introduce a base class for unit tests that are based on real `IndexShard`s. The base class takes care of all the little details needed to create and recover shards. 

This commit also moves `IndexShardTests` and `ESIndexLevelReplicationTestCase` to use the new base class. All tests in `IndexShardTests` that required a full node environment were moved to a new `IndexShardIT` suite.
2016-09-12 18:20:25 +02:00
Ali Beyad f39f9b9760 Update discovery nodes after cluster state is published (#20409)
Before, when there was a new cluster state to publish,
zen discovery would first update the set of nodes to
ping based on the new cluster state, then publish the new
cluster state. This is problematic because if the cluster
state failed to publish, then the set of nodes to ping
should not have been updated.

This commit fixes the issue by updating the set of
nodes to ping for fault detection only *after* the new
cluster state has been published.
2016-09-12 12:07:51 -04:00
javanna 2a1ed80262 With #20093 we fixed a NPE thrown when using _source include/exclude and source is disabled in the mappings. Fixing meant ignoring the _source parameter in the request as no fields can be extracted from it.
We should rather throw a clear exception to clearly point out that we cannot extract fields from _source. Note that this happens only when explicitly trying to extract fields from source. When source is disabled and no _source parameter is specified, no errors will be thrown and no source will be returned.

Closes #20408
Relates to #20093
2016-09-12 17:36:48 +02:00
Jim Ferenczi 82fd95fd24 Merge pull request #20400 from jimferenczi/function_score_highlight
Fix highlighting of MultiTermQuery within a FunctionScoreQuery
2016-09-12 15:56:06 +02:00
Luca Cavanna b1a2768d7d Merge pull request #20188 from qwerty4030/fix/3839_multi_index_add_remove
Fix IndexNotFoundException in multi index search request.
2016-09-12 14:42:56 +02:00