Commit Graph

5824 Commits

Author SHA1 Message Date
Colin Goodheart-Smithe b717ad8eb6 Enable option to use request cache for size > 0
Previously if the size of the search request was greater than zero we would not cache the request in the request cache.

This change retains the default behaviour of not caching requests with size > 0 but also allows the `request_cache=true` query parameter
to enable the cache for requests with size > 0
2016-07-18 13:33:59 +01:00
Adrien Grand 398d70b567 Add `scaled_float`. #19264
This is a tentative to revive #15939 motivated by elastic/beats#1941.
Half-floats are a pretty bad option for storing percentages. They would likely
require 2 bytes all the time while they don't need more than one byte.

So this PR exposes a new `scaled_float` type that requires a `scaling_factor`
and internally indexes `value*scaling_factor` in a long field. Compared to the
original PR it exposes a lower-level API so that the trade-offs are clearer and
avoids any reference to fixed precision that might imply that this type is more
accurate (actually it is *less* accurate).

In addition to being more space-efficient for some use-cases that beats is
interested in, this is also faster that `half_float` unless we can improve the
efficiency of decoding half-float bits (which is currently done using software)
or until Java gets first-class support for half-floats.
2016-07-18 12:36:23 +02:00
Adrien Grand bde99bad2e Use a static default precision for the cardinality aggregation. #19215
Today the default precision for the cardinality aggregation depends on how many
parent bucket aggregations it had. The reasoning was that the more parent bucket
aggregations, the more buckets the cardinality had to be computed on. And this
number could be huge depending on what the parent aggregations actually are.

However now that we run terms aggregations in breadth-first mode by default when
there are sub aggregations, it is less likely that we have to run the cardinality
aggregation on kagilions of buckets. So we could use a static default, which will
be less confusing to users.
2016-07-18 11:30:41 +02:00
Boaz Leskes 9ededa46bc Make static Store access shard lock aware (#19416)
We currently have concurrency issue between the static methods on the Store class and store changes that are done via a valid open store. An example of this is the async shard fetch which can reach out to a node while a local shard copy is shutting down (the fetch does check if we have an open shard and tries to use that first, but if the shard is shutting down, it will not be available from IndexService).

Specifically, async shard fetching tries to read metadata from store, concurrently the shard that shuts down commits to lucene, changing the segments_N file. this causes a file not find exception on the shard fetching side. That one in turns makes the master think the shard is unusable. In tests this can cause the shard assignment to be delayed (up to 1m) which fails tests. See https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+java9-periodic/570 for details.

This is one of the things #18938 caused to bubble up.
2016-07-18 11:22:58 +02:00
Adrien Grand dd95dc7a0f Fix potential AssertionError with include/exclude on terms aggregations. #19252
We call `LongBitSet.set(start, end)`, which fails when `start >= length`
(0 in that case).

Closes #18575
2016-07-18 11:03:24 +02:00
Martijn van Groningen e0ebf5da1c Template cleanup:
* Removed `Template` class and unified script & template parsing logic. Templates are scripts, so they should be defined as a script. Unless there will be separate template infrastructure, templates should share as much code as possible with scripts.
* Removed ScriptParseException in favour for ElasticsearchParseException
* Moved TemplateQueryBuilder to lang-mustache module because this query is hard coded to work with mustache only
2016-07-18 10:16:01 +02:00
Boaz Leskes 798ee177ed mute testAckedIndexing
pending the merge of https://github.com/elastic/elasticsearch/pull/19416
2016-07-18 10:07:18 +02:00
Ali Beyad 6acb8b31fc Removes ensureYellow() calls after index creation in the (#19452)
integration tests, as they are no longer needed with
index creation now waiting for shards to be started before
returning from the index creation call (by default, it waits
for the primary of each shard to be started before returning,
which is what ensureYellow() was ensuring anyway).

Closes #19452
Relates #19450
2016-07-15 15:37:35 -04:00
Jason Tedor e772b6d924 Add log message about enforcing bootstrap checks
This commit adds a log message when bootstrap checks are enforced
informing the user that they are enforced because they are bound to an
external network interface. We also log if bootstrap checks are being
enforced but system checks are being ignored.

Relates #19451
2016-07-15 14:29:36 -04:00
Yannick Welsch f5b5fbcf1d Strengthen assertions when random failures are not injected by AbstractIndicesClusterStateServiceTestCase (#19358)
The unit tests for IndicesClusterStateService currently inject random failures upon shard creation/ routing upate / mapping update etc. This commit makes injecting failures optional so that stronger assertions can be made about the local indices / shard state in case of no failures.
2016-07-15 18:32:03 +02:00
Ali Beyad 687e2e12b3 Merge pull request #19450 from elastic/feature/friendly-index-creation
Makes index creation more friendly
2016-07-15 11:48:21 -04:00
Ali Beyad d78f40fb1e Index creation waits for active shard copies before returning (#18985)
Before returning, index creation now waits for the configured number
of shard copies to be started. In the past, a client would create an
index and then potentially have to check the cluster health to wait
to execute write operations. With the cluster health semantics changing
so that index creation does not cause the cluster health to go RED,
this change enables waiting for the desired number of active shards
to be active before returning from index creation.

Relates #9126
2016-07-15 11:19:27 -04:00
Jason Tedor 917fea7c5d Reset Priority values
For historical reasons, the value associated with Priority.IMMEDIATE is
-1. Yet, with a full-cluster restart required on major version upgrades,
we can reset these values so they are conceptually simpler. This commit
resets the values associated with Priority instances.
2016-07-15 09:34:31 -04:00
Jason Tedor 220a510d65 Make Priority an enum
Today we have an abstraction Priority for representing
priorities. Ideally, these values are a fixed set of constants with a
well-defined ordering which sounds perfect for an enum. This commit
changes Priority so that it is an enum instead of a class.
2016-07-15 08:55:49 -04:00
Jason Tedor ac39e73183 Priority values should be unmodifiable
In Priority there is a field named values that represents an ordered, by
priority, list of all priorities. Yet, this collection is modifiable and
this collection is exposed via the public API. This means that consumers
can modify this list potentially leading to complete chaos. This commit
modifies this field so that it is unmodifiable, documents that the
returned collection is unmodifiable, and returns total order to the
world. We also punish the bad consumer here by making them make a copy
of the returned collection with which they can do as they please. This
fixes a puzzling test failure which only arises if the two tests
(PrioritizedExecutorsTests#testPriorityQueue and
PriorityTests#testCompareTo run in the same JVM, and run in the right
order).

Relates #19447
2016-07-15 08:36:59 -04:00
Martijn van Groningen d0069f0fbb Provide access to ThreadContext in ingest plugins
Also introduced a `Processor.Parameters` class that is holder for several services processors rely on,
the  IngestPlugin#getProcessors(...) method has been changed to accept `Processor.Parameters` instead
of each service seperately.
2016-07-15 08:16:15 +02:00
Ryan Ernst 9b6e2a8e2f Merge pull request #19440 from rjernst/rest_headers
Plugins: Make rest headers registration pull based
2016-07-14 20:33:44 -07:00
Jason Tedor a5b8cb87be Log one plugin info per line
Today we log all loaded modules and installed plugins in a single
line. The number of modules has grown, and when plugins are installed a
single log line containing the loaded modules and plugins is
lengthy. With this commit, we log a single module or plugin per line,
log these in sorted order, and also log if no modules or no plugins were
loaded.

Relates #19441
2016-07-14 22:46:35 -04:00
Ryan Ernst 4b9932d4a8 Merge branch 'master' into rest_headers 2016-07-14 19:03:53 -07:00
Jason Tedor 31c648eee8 Rename transport-netty to transport-netty3
This commit renames the Netty 3 transport module from transport-netty to
transport-netty3. This is to make room for a Netty 4 transport module,
transport-netty4.

Relates #19439
2016-07-14 22:03:14 -04:00
Ryan Ernst 0b514f82a0 Plugins: Make rest headers registration pull based
Currently custom headers that should be passed through rest requests are
registered by depending on the RestController in guice and calling a
registration method. This change moves that registration to a getter for
plugins, and makes the RestController take the set of headers on
construction.
2016-07-14 18:45:53 -07:00
Ali Beyad b96695396a Adds debug logging to RepositoryUpgradabilityIT test to help figure out
failures in recovery reset/retry.
2016-07-14 11:31:28 -04:00
Zachary Tong c950ea0023 Record method counts while profiling (#18302)
Invocation counts can be used to help judge the selectivity of individual query components in the context of the entire query.  E.g. a query may not look selective when run by itself (matches most of the index), but when run in context of a full search request, is evaluated only rarely due to execution order

Since this is modifying the base timing class, it'll enrich both query and agg profiles (as well as future profile results)
2016-07-14 09:46:24 -04:00
Zachary Tong 8fec348880 Don't recursively count children profile timings (#19397)
The breakdown is already inclusive of children timing, also counting the child times will
double-count and inflate the final time.

Closes #18693
2016-07-14 09:29:43 -04:00
Simon Willnauer 5616251f22 Remove `node.mode` and `node.local` settings (#19428)
Today `node.mode` and `node.local` serve almost the same purpose, they
are a shortcut for `discovery.type` and `transport.type`. If `node.local: true`
or `node.mode: local` is set elasticsearch will start in _local_ mode which means
only nodes within the same JVM are discovered and a non-network based transport
is used. The _local_ mode it only really used in tests or if nodes are embedded.
For both, embedding and tests explicit configuration via `discovery.type` and `transport.type`
should be preferred.

This change removes all the usage of these settings and by-default doesn't
configure a default transport implemenation since netty is now a module. Yet, to make
the user expericence flawless, plugins or modules can set a `http.type.default` and
`transport.type.default`. Plugins set this via `PluginService#additionalSettings()`
which enforces _set-once_ which prevents node startup if set multiple times. This means
that our distributions will just startup with netty transport since it's packaged as a
module unless `transport.type` or `http.transport.type` is explicitly set.

This change also found a bunch of bugs since several NamedWriteables were not registered if a
transport client is used. Now that we don't rely on the `node.mode` leniency which is inherited
instead of using explicit settings, `TransportClient` uses `AssertingLocalTransport` which detects these problems since it serializes all messages.

Closes #16234
2016-07-14 13:21:10 +02:00
Simon Willnauer 4156a4bebb Add support for `wait_for_events` to the `_cluster/health` REST endpoint (#19432)
The Java API supports this while mostly used for tests it can also be useful in
production environments. For instance if something is automated like a settings change
and we execute some health right after it the settings update might have some consequences
like a reroute which hasn't been fully applied since the preconditions are not fulfilled yet.
For instance if not all shards started the settings update is applied but the reroute won't move
currently initializing shards like in the shrink API test. Sure this could be done by waiting for
green before but if the cluster moves shards due to some side-effects waiting for all events is
still useful. I also took the chance to add unittests to Priority.java

Closes #19419
2016-07-14 12:33:29 +02:00
Mathias Fussenegger 8c0b954466 Complete load-settings error message 2016-07-13 23:39:16 +02:00
Tal Levy ed768b101f show ignored errors in verbose simulate result (#19404)
Closes #19319.
2016-07-13 13:32:10 -07:00
Tal Levy 8fd01554bc update foreach processor to only support one applied processor. (#19402)
Closes #19345.
2016-07-13 13:13:00 -07:00
gfyoung 3f2e1066d3 Removed duplicate deleteBlob methods (#18813)
Removed the following methods from the
BlobContainer interface to clean up the interface:

1) deleteBlobs
2) deleteBlobsByPrefix

Closes #18529
2016-07-13 14:36:23 -04:00
Chris Earle ce65ab6eb7 Add RestController method for deprecating in one step
This adds an extra method, registerWithDeprecatedHandler, to register both a normal handler and a deprecated handler at the same time. This helps with renaming methods as opposed to _just_ deprecated methods.
2016-07-13 13:03:23 -04:00
Nik Everett 2422b969c1 Migrate matrix_stats to NamedWriteable
This is the last consumer of the old style register method so I removed
the method.
2016-07-13 10:48:20 -04:00
Nik Everett d95fbba8cb Switch remaining builtin aggs to new registration method 2016-07-13 10:48:20 -04:00
Martijn van Groningen 2c3165d080 Removed deprecated 1.x script and template syntax
Closes #13729
2016-07-13 15:07:36 +02:00
Nik Everett 88d3527178 Migrate derivative pipeline aggregation to NamedWriteable
This is another step in the effort to remove AggregationStreams and
instead use NamedWriteableRegistry like the rest of the code base.
2016-07-13 07:12:22 -04:00
Simon Willnauer 29fd0f1bd8 [TEST] Remove wrong transportName from MockTcpTransport#ctor 2016-07-13 12:50:52 +02:00
Simon Willnauer ae98d59899 Don't assert that files exists if recovery has been cancled
Today we assert that the tmp files are present but if the recovery
was canceled this might not be the case while still a valid state.
This chance only throws the AssertionError if the recovery is still active.
2016-07-13 10:52:17 +02:00
Simon Willnauer 814c7224f9 Merge pull request #19392 from elastic/modularize_netty
This moves all netty related code into modules/transport-netty the module is build as a zip file as well as a JAR to serve as a dependency for transport client. For the time being this is required otherwise we have no network based impl. for transport client users. This might be subject to change given that we move forward http client.
2016-07-13 09:52:03 +02:00
Martijn van Groningen 2bdc55c9ff fvh: Also extract terms from the nested query' inner query.
Closes #19265
2016-07-13 08:15:46 +02:00
Nik Everett d14e06ce51 Migrate top_hits, histogram, and ip_range aggregations to NamedWriteable
This is just another step towards removing AggregationStreams
in favor of NamedWriteable.
2016-07-12 23:02:32 -04:00
Nik Everett f2978f41b9 Migrate nested, reverse_nested, and children aggregations to NamedWriteable
Just another step in removing AggregationStreams in favor of NamedWriteable.
2016-07-12 22:38:51 -04:00
Nik Everett 06bd896ce0 Migrate geohash_grid and geo_bounds to NamedWriteable
Just another small step in removing Aggregation's custom streams
implementation in favor of NamedWriteable.
2016-07-12 22:22:51 -04:00
Nik Everett f479219ca7 Clean up significant terms aggregation results
* Clean up the generics around significant terms aggregation results
* Reduce code duplicated between `SignificantLongTerms` and
`SignificantStringTerms` by creating `InternalMappedSignificantTerms`
and moving common things there where possible.
* Migrate to `NamedWriteable`
* Line length fixes while I was there
2016-07-12 22:08:09 -04:00
Ryan Ernst 920bd0cf68 Merge pull request #19401 from rjernst/more_plugin_services
Plugins: Add resource watcher to services available for plugin components
2016-07-12 15:28:17 -07:00
Lee Hinman 95cf2407ee Merge remote-tracking branch 'dakrone/include-cluster-info-in-explain-api' 2016-07-12 16:26:46 -06:00
Jason Tedor ce5a382c69 Remove support for properties
This commit removes support for properties syntax and config files:
 - removed support for elasticsearch.properties
 - removed support for logging.properties
 - removed support for properties content detection in REST APIs
 - removed support for properties content detection in Java API

Relates #19398
2016-07-12 17:55:18 -04:00
Lee Hinman 58db63b610 Expose the ClusterInfo object in the allocation explain output
This adds an optional parameter to the cluster allocation explain API
that will return the cluster info object, `include_disk_info`, the
output looks like:

GET /_cluster/allocation/explain?include_disk_info -d'
{"index": "i", "shard": 0, "primary": false}'

{
  ... other info ...

  "cluster_info" : {
    "nodes" : {
      "7Uws-vL7R6WVm3ZwQA1n5A" : {
        "node_name" : "Kraven the Hunter",
        "least_available" : {
          "path" : "/path/to/data1",
          "total_bytes" : 165999570944,
          "used_bytes" : 118180614144,
          "free_bytes" : 47818956800,
          "free_disk_percent" : 28.80667493781158,
          "used_disk_percent" : 71.19332506218842
        },
        "most_available" : {
          "path" : "/path/to/data2",
          "total_bytes" : 165999570944,
          "used_bytes" : 118180614144,
          "free_bytes" : 47818956800,
          "free_disk_percent" : 28.80667493781158,
          "used_disk_percent" : 71.19332506218842
        }
      }
    },
    "shard_sizes" : {
      "[i][2][p]_bytes" : 0,
      "[i][4][p]_bytes" : 130,
      "[i][1][p]_bytes" : 0,
      "[i][3][p]_bytes" : 0,
      "[i][0][p]_bytes" : 130
    },
    "shard_paths" : {
      "[i][3], node[7Uws-vL7R6WVm3ZwQA1n5A], [P], s[STARTED], a[id=LegZLDniTVaw0Y1urv7s3g]" : "/path/to/data1/nodes/0",
      "[i][1], node[7Uws-vL7R6WVm3ZwQA1n5A], [P], s[STARTED], a[id=lAU_4vf_SKmoRdtg0ACnjQ]" : "/path/to/data1/nodes/0",
      "[i][2], node[7Uws-vL7R6WVm3ZwQA1n5A], [P], s[STARTED], a[id=Aurpeuj7SeGeyPDDpCtRgg]" : "/path/to/data1/nodes/0",
      "[i][0], node[7Uws-vL7R6WVm3ZwQA1n5A], [P], s[STARTED], a[id=Vgg8GlQTQ82C2j6HYBq8DQ]" : "/path/to/data1/nodes/0",
      "[i][4], node[7Uws-vL7R6WVm3ZwQA1n5A], [P], s[STARTED], a[id=t8hQlVSxQe-58fSeaXcAqg]" : "/path/to/data1/nodes/0"
    }
  }
}

Resolves #14405
2016-07-12 15:52:20 -06:00
Ryan Ernst 9b5ac4f68e Plugins: Add resource watcher to services available for plugin components 2016-07-12 14:51:50 -07:00
Simon Willnauer 4fb79707bd Fix remaining tests that either need access to the netty module or require explict configuration
Some tests still start http implicitly or miss configuring the transport clients correctly.
This commit fixes all remaining tests and adds a depdenceny to `transport-netty` from
`qa/smoke-test-http` and `modules/reindex` since they need an http server running on the nodes.

This also moves all required permissions for netty into it's module and out of core.
2016-07-12 16:29:57 +02:00
Martijn van Groningen 075cb970c0 inner_hits: Ensure that that InnerHitBuilder uses rewritten queries
If a nested, has_child or has_parent query's inner query gets rewritten then the InnerHitBuilder should use that rewritten form too, otherwise this can cause exceptions in a later phase.

Also fixes a bug that HasChildQueryBuilder's rewrite method overwrites max_children with min_children value.

Closes #19353
2016-07-12 16:26:57 +02:00
Britta Weber fed6b72460 Add test for #19389 new type always creates _parent#null field 2016-07-12 15:45:31 +02:00
Boaz Leskes 081d04afac Make NotMasterException a first class citizen (#19385)
That exception is currently serialized as its current base class IllegalStateException which confuses code supposed to deal with the stepping down of a master. This is an important exception and we should be able to serialize it correctly. This commit fixes it by moving the exception to inherit from ElasticsearchException and properly register it.

As a bonus I adapted CapturingTransport to properly simulate serialized exceptions.
2016-07-12 12:44:40 +02:00
Simon Willnauer 199a5a1f04 Fix TcpTransport#sendRequest to raise NotConnectedExcepiton if we get disconnected while sending
This also fixes a race in AbstractSimpleTransportTestCase where we never wait long enough
for all response to finish causing expected failures.
2016-07-12 10:56:20 +02:00
Ryan Ernst 93aebbef0f Merge branch 'master' into modularize_netty 2016-07-11 23:49:00 -07:00
Ryan Ernst 81429aed8c Remove createComponents from transport client 2016-07-11 23:33:49 -07:00
Ryan Ernst 86d0d67036 Plugins: Add some basic services to createComponents
This adds the first few basic services needed for any plugin to create
its own components that interact with the rest of the system.
2016-07-11 23:22:20 -07:00
Ryan Ernst 7195d1e0ff Fix plugins service to not double bind plugin components 2016-07-11 17:05:56 -07:00
Nik Everett 8263873783 Switch search extension from push to pull
Switches most search behavior extensions from push (`onModule(SearchModule)`)
to pull (`implements SearchPlugin`). This effort in general gives plugin
authors a much cleaner view of how to extend Elasticsearch and starts to
set up portions of Elasticsearch as "the plugin API". This commit in
particular does that for search-time behavior like customized suggesters,
highlighters, score functions, and significance heuristics.

It also switches most such customization to being done at search module
construction time which is much, much easier to reason about from a testing
perspective. It also helps significantly in the process of de-guice-ing
Elasticsearch's startup.

There are at least two major search time extensions that aren't covered in
this commit that will simply have to wait for the next commit on the topic
because this one has already grown large: custom aggregations and custom
queries. These will likely live in the same SearchPlugin interface as well.
2016-07-11 18:49:05 -04:00
Ryan Ernst 535b60cb2b Merge pull request #19371 from rjernst/plugin_components
Add components getter as bridge between guice and new plugin init world
2016-07-11 14:23:45 -07:00
Ryan Ernst 6b4d0001a2 Remove unnecessary warning suppression 2016-07-11 14:20:47 -07:00
Ryan Ernst 05ea943def Rename local plugin componenents list for clarity 2016-07-11 14:16:23 -07:00
Ryan Ernst 99ac65931a Plugins: Add components creator as bridge between guice and new plugin init world
This change adds a createComponents() method to Plugin implementations
which they can use to return already constructed componenents/services.
Eventually this should be just services ("components" don't really do
anything), but for now it allows any object so that preconstructed
instances by plugins can still be bound to guice. Over time we should
add basic services as arguments to this method, but for now I have left
it empty so as to not presume what is a necessary service.
2016-07-11 14:14:06 -07:00
Simon Willnauer 048e4416e7 Move netty transport and http into a module
This moves all netty code and it's dependency into a module.
2016-07-11 22:21:29 +02:00
Nik Everett c680bd57da Fix scroll test
It was relying on unreasonably large windows crashing. Those large windows
abort the request immediately now.
2016-07-11 16:04:17 -04:00
Ali Beyad 7759c23272 Fix line length formatting for ClusterStateHealthTests 2016-07-11 15:32:13 -04:00
Ali Beyad 0faf638710 Blocked allocations on primary causes RED health
If the allocation decision for a primary shard was NO, this should
cause the cluster health for the shard to go RED, even if the shard
belongs to a newly created index or is part of cluster recovery.

Relates #9126
2016-07-11 15:32:13 -04:00
Ali Beyad 417bd0cd63 Index creation does not cause the cluster health to go RED
Previously, index creation would momentarily cause the cluster health to
go RED, because the primaries were still being assigned and activated.
This commit ensures that when an index is created or an index is being
recovered during cluster recovery and it does not have any active
allocation ids, then the cluster health status will not go RED, but
instead be YELLOW.

Relates #9126
2016-07-11 15:30:47 -04:00
Nik Everett 3ea1360625 Limit batch size when scrolling
Limits the batch size from scrolling using the same setting as interactive
search: `index.max_result_window`.

Closes #19249
2016-07-11 15:29:45 -04:00
Simon Willnauer 47bd2f9ca5 More cleanups aroung tests that require HTTP to be enalbed. (#19363)
this commit moves the most of the http related integ tests out into it's own 
`qa/smoke-test-http` project where most of the test can run against the external cluster.
2016-07-11 20:44:57 +02:00
Nik Everett 89614586e9 Migrate range, date_range, and geo_distance aggregations to NamedWriteable 2016-07-11 13:00:36 -04:00
Christoph Büscher 0d428b6ba8 Add test for GeoHashUtils#bbox() 2016-07-11 10:46:31 -05:00
Nicholas Knize f77f79c24a GeoBoundingBoxQueryBuilder should fail when topLeft and bottomRight are the same coordinate 2016-07-11 10:25:33 -05:00
Jason Tedor df7ad9970b Batch process node left and node failure
Today when a node is removed the cluster (it leaves or it fails), we
submit a cluster state update task. These cluster state update tasks are
processed serially on the master. When nodes are removed en masse (e.g.,
a rack is taken down or otherwise becomes unavailable), the master will
be slow to process these failures because of the resulting reroutes and
publishing of each subsequent cluster state. We improve this in this
commit by processing the node removals using the cluster state update
task batch processing framework.

Relates #19289
2016-07-11 08:30:09 -04:00
Simon Willnauer 3f3c93ec65 Add blocking socket based MockTcpTransport (#19332)
Today we have a bunch of tests that use netty transport for several reasons
these tests use it because they need to run some tcp based transport. Yet, this
couples our tests tightly to the netty implementation which should be tested on it's own.
This change adds a plain socket based blocking TcpTransport implementation that is used by
default in tests if local transport is suppressed or if network is selected.
It also adds another tcp network implementation as a showcase how the interface works.
2016-07-11 12:17:52 +02:00
Simon Willnauer 1d03a1409c Catch assertion errors on commit and turn it into a real exception (#19357)
Lucene IndexWriter asserts on files existing on the filesystem but
some tests throw IOException explicitly on those operatiosn such that
some tests trip asserts. We had this before on InternalEngine#ctor
and added some logic there to catch only a specific assertions based
on some excepition stack analysis. This change applies the same logic
to the IndexWriter#commit part of the engine since it can hit the same
issue.
This also fixes a self-suppression issue in Store.java.

Closes #19356
2016-07-11 11:20:56 +02:00
javanna 942e342662 Rest Client: use short performRequest methods when possible 2016-07-11 10:36:26 +02:00
Simon Willnauer 36dbe7250f Cleanup usage of http.enabled (#19351)
Several tests required http.enabled where it was unnecessary.
We also had RestMainActionIT which tests what two of our REST tests
test already so I removed it.
The explicit use of http.enabled: false is also obsolet since our
test do that by default.
2016-07-11 10:21:03 +02:00
Ryan Ernst 25ed93dd28 Fix test edge case for random bytes reference iter.
Getting an offset to the last byte means we can only stream one byte and
then we are done, we can't get another offset after it.
2016-07-10 09:11:32 -07:00
Nicholas Knize ab8b577aea Use GeoDistanceIT.testDuelOptimizations for bwc testing only
This test is only relevant for testing the GeoDistanceBuilder with old indices (pre GeoPointV2).

closes #19263
2016-07-09 19:08:06 -05:00
Ryan Ernst 3832825a87 Merge pull request #19348 from rjernst/deguice_attrs
Remove CustomNodeAttributes extension point
2016-07-09 12:46:26 -07:00
Ryan Ernst 2b9d4bdf85 Plugins: Remove CustomNodeAttributes extension point
The DiscoveryNodeService exists to register CustomNodeAttributes which
plugins can add. This is not necessary, since plugins can already add
additional attributes, and use the node attributes prefix.

This change removes the DiscoveryNodeService, and converts the only
consumer, the ec2 discovery plugin, to add the ec2 availability zone
in additionalSettings().
2016-07-08 21:39:11 -07:00
Ryan Ernst 7e59181e58 Internal: Remove child injectors from guice
This change removes the ability for guice to have child injectors (and
the entire concept of parent injectors) from our fork of guice. The
methodology for removing was simple: I removed createChildInjector, and
continued to remove methods and members that were unused until my head
was spinning. The motivation for this change is to limit what our fork
of guice gives us access to, so we don't regress and start adding back
more complicated uses.
2016-07-08 15:22:50 -07:00
Ryan Ernst dea00a0b16 Merge pull request #19324 from rjernst/repository_deguice2
Add RepositoryPlugin interface for registering snapshot repositories
2016-07-08 14:38:07 -07:00
Nicholas Knize 8dd4a6473e Remove radial restriction for GeoDistanceQuery
As of lucene 6.1 GeoDistanceQuery no longer requires restricting the radial distance in GeoPointDistanceQuery.

closes #17578
2016-07-08 12:13:22 -05:00
Nicholas Knize 72fa345f5e Mute GeoDistanceIT while fixing 2016-07-08 10:02:15 -05:00
Simon Willnauer f6ac147b1d Add a unit test that sends random requests among 3 nodes (#19329)
This adds a test that uses transport implementation and sends random requests
to 3 different nodes, the request handlers maybe forwarding the requests to yet another node
etc. until returning the response. This test basically tests that nodes are not deadlocking
in a distributed fashion.
2016-07-08 14:13:36 +02:00
Martijn van Groningen 7b8ae54f0f percolator: Also support query term extract for queries wrapped inside a FunctionScoreQuery
Additionally for highlighting percolator hits, also extract percolator query from FunctionScoreQuery and DisjunctionMaxQuery
2016-07-08 10:51:48 +02:00
Simon Willnauer 1cb1373722 [TEST] Test analyzer alias works
Relates to #19163
2016-07-08 10:33:12 +02:00
Ryan Ernst e6be4af014 Plugins: Add RepositoryPlugin interface for registering snapshot repositories
Repository plugins currently use a lot of custom classes like
RepositoryName and RepositorySettings in order to use guice to construct
repository implementations. But repositories now only really need their
settings to be constructed. Anything else they need (eg a cloud client)
can be constructed within the plugin, instead of via guice.

This change makes repository plugins use the new pull model. It removes
guice from the construction of Repository objects (no more child
injectors) and also from all repository plugins.
2016-07-08 00:10:03 -07:00
Nik Everett 81fcdfcee9 Expose task information from NodeClient
This exposes a method to start an action and return a task from
`NodeClient`. This allows reindex to use the injected `Client` rather
than require injecting `TransportAction`s
2016-07-07 18:02:09 -04:00
Nik Everett fe0f28965a Clean up serialization of terms aggregation results
Move to NamedWriteable and remove a lot of duplication.
2016-07-07 17:01:09 -04:00
Nik Everett 7da753a4d7 Migrate sampler and missing aggregations to NamedWriteable
This is another step down the path to removing aggregation's special
"streams" which reimplement NamedWriteable.
2016-07-07 16:40:38 -04:00
Nik Everett d83e1cccac Fix checkstyle in test 2016-07-07 16:40:11 -04:00
Ryan Ernst 89d69ea5a2 Merge pull request #19292 from rjernst/repository_deguice
Simplified repository api for snapshot/restore
2016-07-07 13:03:58 -07:00
Ryan Ernst 593f8bdf0c Rename repository api methods for clarity and tweak documentation. 2016-07-07 12:54:10 -07:00
Jason Tedor e86aa29f67 Die with dignity
Today when a thread encounters a fatal unrecoverable error that
threatens the stability of the JVM, Elasticsearch marches on. This
includes out of memory errors, stack overflow errors and other errors
that leave the JVM in a questionable state. Instead, the Elasticsearch
JVM should die when these errors are encountered. This commit causes
this to be the case.

Relates #19272
2016-07-07 14:44:03 -04:00
Jason Tedor d3f8329a3d Tighten ensure atomic move cleanup
This commit tightens the cleanup after possible errors while ensuring
the filesystem supports atomic move.

Relates #19309
2016-07-07 14:40:05 -04:00
Tanguy Leroux 3267fc4e0c Clean up more messy tests
After #13834 many tests that used Groovy scripts (for good or bad reason) in their tests have been moved in the lang-groovy module and the issue #13837 has been created to track these messy tests in order to clean them up.

This commit moves more tests back in core, removes the dependency on Groovy, changes the scripts in order to use the mocked script engine, and change the tests  to integration tests.
2016-07-07 17:50:23 +02:00
Clinton Gormley ec3807d426 Added 2.3.4 version and bwc indices 2016-07-07 16:36:23 +02:00
Tanguy Leroux b58f2eb5c2 Move back some messy tests from Groovy plugin to core
This commit moves back some messy tests that have been placed in lang-groovy module in https://github.com/elastic/elasticsearch/pull/13834. It removes the dependency on Groovy plugin as well as change back the tests to integration tests (IT suffix).

It also changes the current MockScriptEngine and MockScriptPlugin to make it easier to use.
2016-07-07 15:26:36 +02:00
Alexander Reelsen 71b48fb16c Dependencies: Update to jopt-5.0 (#19278)
The new version of jopt allows us to remove a couple of TODOs in the code.

Closes #12368
2016-07-07 08:50:10 +02:00
Jason Tedor b3105bd316 Fix modifier order in BytesStreamsTests
This commit fixes an issue with the ordering of modifiers on a static
nested class in BytesStreamsTests.
2016-07-06 20:11:38 -04:00
Ryan Ernst dd7be74bcf Plugins: Simplified repository api for snapshot/restore
The api for snapshot/restore was split up between two interfaces,
Repository and IndexShardRepository. There was also complex
initialization and injection between the two. However, there is always a
one to one relationship between the two.

This change moves the IndexShardRepository api into Repository, as well
as updates the API so as not to require any services to be injected for
sublcasses.
2016-07-06 17:09:30 -07:00
Chris Earle b927cfe1de Add DeprecationRestHandler to automatically log deprecated REST calls
This adds a new proxy for RestHandlers and RestControllers so that requests made
to deprecated REST APIs can be automatically logged in the ES logs via the
DeprecationLogger as well as via a "Warning" header (RFC-7234) for all responses.
2016-07-06 19:52:00 -04:00
Yannick Welsch 76057e6b05 Fix test issue where index is explicitly deleted during cluster state update
Calling indicesService.deleteIndex() can trip an assertion if there is an ongoing cluster state applied in
IndicesClusterStateService. This means that the index is possibly deleted after the failMissingShards
check and before we try creating new and updated shards, tripping an assertion that non-existing shards must
have shard state initializing (started in this case).
2016-07-06 16:51:22 +02:00
Jim Ferenczi 37725f640c Add missing field type in the FieldStats response.
This change adds the type of the field in the fieldstats response.
It can be one of the following:
  * "integer" for byte, short, integer and long
  * "float" for float, half-float and double
  * "date" for date
  * "ip" for ip
  * "text" for string, keyword and text.

Closes #17750
2016-07-06 15:24:09 +02:00
Ryan Ernst f49ce8e6fe Add basic tests for ingest plugins setup 2016-07-05 21:03:39 -07:00
Ryan Ernst 2fc41adeb5 Merge branch 'master' into ingest_plugin_api 2016-07-05 20:53:03 -07:00
Ryan Ernst 18c9e7adaf Merge branch 'master' into unused 2016-07-05 20:46:36 -07:00
Ryan Ernst 14eefb7607 Internal: Remove guice from transport client helper classes
This change removes injection for constructors of
TransportClientNodesService and TransportProxyClient.
2016-07-05 19:51:03 -07:00
Igor Motov 74af0e36f3 [TEST] Fix rare OBOE in AbstractBytesReferenceTestCase 2016-07-05 16:40:03 -04:00
Nik Everett b3c015e2bb Reindex from remote
This adds a remote option to reindex that looks like

```
curl -POST 'localhost:9200/_reindex?pretty' -d'{
  "source": {
    "remote": {
      "host": "http://otherhost:9200"
    },
    "index": "target",
    "query": {
      "match": {
        "foo": "bar"
      }
    }
  },
  "dest": {
    "index": "target"
  }
}'
```

This reindex has all of the features of local reindex:
* Using queries to filter what is copied
* Retry on rejection
* Throttle/rethottle
The big advantage of this version is that it goes over the HTTP API
which can be made backwards compatible.

Some things are different:

The query field is sent directly to the other node rather than parsed
on the coordinating node. This should allow it to support constructs
that are invalid on the coordinating node but are valid on the target
node. Mostly, that means old syntax.
2016-07-05 16:13:17 -04:00
Jason Tedor 96f283c195 Rename writeThrowable to writeException
This commit renames writeThrowable to writeException. The situation here
stems from the fact that the StreamOutput method for serializing
Exceptions needs to accept Throwables too as Throwables can be the cause
of serialized Exceptions. Yet, we do not serialize Throwables in the
Error sub-hierarchy in a way that they can be deserialized into their
initial type. This leads to an asymmetry in the StreamOutput method for
serializing Exceptions and the StreamInput method for writing
Excpetions. Namely, the former will accept Throwables but the latter
will only return Exceptions. A goal with the stream methods has always
been symmetry in the method names so that serialization/deserialization
routines appear symmetrical in code. It is this asymmetry on the
input/output types for Exceptions on StreamOutput/StreamInput that
clashes with the desired symmetry of naming. Despite this, we should
favor symmetry in the naming of the methods. This commit renames
StreamOutput#writeThrowable to StreamOutput#writeException which leaves
us with Exception StreamInput#readException and void
StreamOutput#writeException(Throwable).
2016-07-05 14:37:01 -04:00
Justin Patrin ebe616988a Start transport client round-robin randomly
This commit modifies the initial value of the transport client
round-robin index to a random value so that initial requests are more
likely to not all hit the same node.

Relates #14143
2016-07-05 13:17:17 -04:00
LeeDr 036b8ff177 Fix stored_fields message 2016-07-05 09:11:29 -05:00
Nik Everett 3269c84c7a Remote BucketStreams
It isn't used.
2016-07-05 09:02:36 -04:00
Adrien Grand 4b0d317e63 Bump version to 5.0.0-alpha5. 2016-07-05 14:34:23 +02:00
Simon Willnauer d08812d839 [TEST] fix test to account for internal empyt reference optimization 2016-07-05 11:23:43 +02:00
Simon Willnauer a4ec0ac22f Upgrade to netty 3.10.6.Final (#19235) 2016-07-05 11:11:55 +02:00
Simon Willnauer cbbc8790a5 Remove redundant modifier 2016-07-05 08:39:08 +02:00
Simon Willnauer 44ccf67e33 Simplify TcpTransport interface by reducing send code to a single send method (#19223)
Due to some optimization on the netty layer we had quite some code / cruft
added to the TcpTransport to allow for those optimizations. After cleaning
up BytesReference we can now move this optimization into TcpTransport and
have a simple send method on the implementation layer instead. This commit
adds a CompositeBytesReference that also allows message headers to be written
separately which simplify the header code as well since no skips are needed
anymore.
2016-07-05 08:33:19 +02:00
Jason Tedor a00a54ebda Fix style violation in InstallPluginCommand.java
This commit fixes a checkstyle violation in InstallPluginCommand.java
added after renaming UserError to UserException in
f9d55be1ed.
2016-07-04 19:45:46 -04:00
Jason Tedor f9d55be1ed Rename UserError
The top-level class Throwable represents all errors and exceptions in
Java. This hierarchy is divided into Error and Exception, the former
being serious problems that applications should not try to catch and the
latter representing exceptional conditions that an application might
want to catch and handle. This commit renames
org.elasticsearch.cli.UserError to org.elasticsearch.UserException to
make its name consistent with where it falls in this hierarchy.

Relates #19254
2016-07-04 19:22:29 -04:00
Jason Tedor 36b887ee7c Throw translog corrupted exception on malformed op
Today when reading a malformed operation from the translog, we throw an
assertion error that is immediately caught and wrapped into a translog
corrupted exception. This commit replaces this by electing to directly
throw a translog corrupted exception instead.

Additionally, this cleanup also addressed a double-wrapped translog
corrupted exception. Namely, verifying the checksum can throw a translog
corrupted exception which the existing code would catch and wrap again
in a translog corrupted exception.

Relates #19256
2016-07-04 19:22:12 -04:00
Ali Beyad f5b07e438f Fixes getting snapshot status checks for preventing duplicate entries
from both current snapshots (cluster state) and repository snapshots
(on storage).
2016-07-04 17:40:44 -04:00
Boaz Leskes 37c8c0fa03 Improve logging for batched cluster state updates (#19255)
We've been slowly improving batch support in `ClusterService` so service won't need to implement this tricky logic themselves. These good changes are blessed but our logging infra didn't catch up and we now log things like:

```
[2016-07-04 21:51:22,318][DEBUG][cluster.service          ] [node_sm0] processing [put-mapping [type1],put-mapping [type1]]:
```

Depending on the `source` string this can get quite ugly (mostly in the ZenDiscovery area). 

This PR adds some infra to improve logging, keeping the non-batched task the same. As result the above line looks like:

```
[2016-07-04 21:44:45,047][DEBUG][cluster.service          ] [node_s0] processing [put-mapping[type0, type0, type0]]: execute
```

ZenDiscovery waiting on join moved from:

```
[2016-07-04 17:09:45,111][DEBUG][cluster.service          ] [node_t0] processing [elected_as_master, [1] nodes joined),elected_as_master, [1] nodes joined)]: execute
```

To

```
[2016-07-04 22:03:30,142][DEBUG][cluster.service          ] [node_t3] processing [elected_as_master ([3] nodes joined)[{node_t2}{R3hu3uoSQee0B6bkuw8pjw}{p9n28HDJQdiDMdh3tjxA5g}{127.0.0.1}{127.0.0.1:30107}, {node_t1}{ynYQfk7uR8qR5wKIysFlQg}{wa_OKuJHSl-Oyl9Gis-GXg}{127.0.0.1}{127.0.0.1:30106}, {node_t0}{pweq-2T4TlKPrEVAVW6bJw}{NPBSLXSTTguT1So0JsZY8g}{127.0.0.1}{127.0.0.1:30105}]]: execute
```

As a bonus, I removed all `zen-disco` prefixes to sources from that area.
2016-07-04 22:54:43 +02:00
Boaz Leskes 6861d3571e Persistent Node Ids (#19140)
Node IDs are currently randomly generated during node startup. That means they change every time the node is restarted. While this doesn't matter for ES proper, it makes it hard for external services to track nodes. Another, more minor, side effect is that indexing the output of, say, the node stats API results in creating new fields due to node ID being used as keys.

The first approach I considered was to use the node's published address as the base for the id. We already [treat nodes with the same address as the same](https://github.com/elastic/elasticsearch/blob/master/core/src/main/java/org/elasticsearch/discovery/zen/NodeJoinController.java#L387) so this is a simple change (see [here](https://github.com/elastic/elasticsearch/compare/master...bleskes:node_persistent_id_based_on_address)). While this is simple and it works for probably most cases, it is not perfect. For example, if after a node restart, the node is not able to bind to the same port (because it's not yet freed by the OS), it will cause the node to still change identity. Also in environments where the host IP can change due to a host restart, identity will not be the same. 

Due to those limitation, I opted to go with a different approach where the node id will be persisted in the node's data folder. This has the upside of connecting the id to the nodes data. It also means that the host can be adapted in any way (replace network cards, attach storage to a new VM). I

It does however also have downsides - we now run the risk of two nodes having the same id, if someone copies clones a data folder from one node to another. To mitigate this I changed the semantics of the protection against multiple nodes with the same address to be stricter - it will now reject the incoming join if a node exists with the same id but a different address. Note that if the existing node doesn't respond to pings (i.e., it's not alive) it will be removed and the new node will be accepted when it tries another join.

Last, and most importantly, this change requires that *all* nodes persist data to disk. This is a change from current behavior where only data & master nodes store local files. This is the main reason for marking this PR as breaking.

Other less important notes:
- DummyTransportAddress is removed as we need a unique network address per node. Use `LocalTransportAddress.buildUnique()` instead.
- I renamed `node.add_lid_to_custom_path` to `node.add_lock_id_to_custom_path` to avoid confusion with the node ID which is now part of the `NodeEnvironment` logic.
- I removed the `version` paramater from `MetaDataStateFormat#write` , it wasn't really used and was just in the way :)
- TribeNodes are special in the sense that they do start multiple sub-nodes (previously known as client nodes). Those sub-nodes do not store local files but derive their ID from the parent node id, so they are generated consistently.
2016-07-04 21:09:25 +02:00
Tanguy Leroux ed444cd276 Fix CompletionTokenStream modifier redundancy 2016-07-04 16:29:26 +02:00
Tanguy Leroux d72134b46e Fix CompletionSuggestSearchIT and CompletionSuggestSearch2xIT 2016-07-04 15:58:27 +02:00
Tanguy Leroux 0e7faf1005 Enable Checkstyle RedundantModifier 2016-07-04 15:22:12 +02:00
Nik Everett a728c4bb5c Migrate global, filter, and filters aggs to NamedWriteable
Once all of these are migrated we'll be able to remove aggregation's
custom "streams" which function that same as NamedWriteable. It also
allows us to make most of the fields on aggregations final which is
rather nice.

Also starts to migrate MultiBucketAggregation.Bucket to Writeable,
allowing the buckets to have immutable parts.
2016-07-04 08:53:02 -04:00
Nik Everett c02de9227c Migrate remaining calc aggs to NamedWriteable
Once all of these are migrated we'll be able to remove aggregation's
custom "streams" which function that same as NamedWriteable. It also
allows us to make most of the fields on aggregations final which is
rather nice.
2016-07-04 08:46:00 -04:00
Jason Tedor 3343ceeae4 Do not catch throwable
Today throughout the codebase, catch throwable is used with reckless
abandon. This is dangerous because the throwable could be a fatal
virtual machine error resulting from an internal error in the JVM, or an
out of memory error or a stack overflow error that leaves the virtual
machine in an unstable and unpredictable state. This commit removes
catch throwable from the codebase and removes the temptation to use it
by modifying listener APIs to receive instances of Exception instead of
the top-level Throwable.

Relates #19231
2016-07-04 08:41:06 -04:00
Boaz Leskes 86d2e88362 re-introduce: Inline reroute with process of node join/master election (#18938) 2016-07-04 13:06:56 +02:00
Jim Ferenczi afe99fcdcd Restore reverted change now that alpha4 is out:
Rename `fields` to `stored_fields` and add `docvalue_fields`

`stored_fields` parameter will no longer try to retrieve fields from the _source but will only return stored fields.
`fields` will throw an exception if the user uses it.
Add `docvalue_fields` as an adjunct to `fielddata_fields` which is deprecated. `docvalue_fields` will try to load the value from the docvalue and fallback to fielddata cache if docvalues are not enabled on that field.

Closes #18943
2016-07-04 10:39:49 +02:00
Daniel Mitterdorfer 25881c265b Improve error checking and reporting in CBSIT 2016-07-04 10:24:15 +02:00
Daniel Mitterdorfer 4131476db1 Propagate canTripCircuitBreaker for all broadcasted actions
With this commit we also propagate the `canTripCircuitBreaker`
setting for the main action in TransportBroadcastByNodeAction.

Previously, we set it only on the additional action added by
this handler.
2016-07-04 10:24:15 +02:00
Ryan Ernst 5cf7583bde Internal: Remove unused methods and classes
This change removes a handful of classes and methods that were simply
unused. Some of the classes were intermediate abstract classes that
added nothing to the base class they extended.
2016-07-02 11:33:16 -07:00
Yannick Welsch 3e199b1dff Make testAllOperationsInvoked properly clean up after itself 2016-07-02 11:54:52 +02:00
Yannick Welsch 3221c9d970 Match exception message more exactly in test assertion 2016-07-02 10:07:57 +02:00
Yannick Welsch b4064ce43f Make primary relocation handoff non-blocking (#19013)
Primary relocation and indexing concurrently can currently lead to a deadlock situation as indexing operations are blocked on a (bounded) thread pool during the hand-off phase between old and new primary. This change replaces blocking of indexing operations by putting operations that cannot be executed during relocation hand-off in a queue to be executed once relocation completes.

Closes #18553.
2016-07-02 09:35:54 +02:00
Yannick Welsch 50b97ba5f5 Fix test assertion matching exception message
Newer versions of the URL class in JDK 9 use a different exception message when throwing a MalformedURLException due to an
unknown protocol.
2016-07-02 09:27:07 +02:00
Ryan Ernst 5a66c08ae9 Merge branch 'master' into ingest_plugin_api 2016-07-01 16:27:52 -07:00
Ryan Ernst c7b9489be8 Merge pull request #19225 from rjernst/we_dont_need_generics
Internal: Remove generics from LifecycleComponent
2016-07-01 16:25:34 -07:00
Ryan Ernst 822c995367 Internal: Remove generics from LifecycleComponent
The only reason for LifecycleComponent taking a generic type was so that
it could return that type on its start and stop methods. However, this
chaining has no practical necessity. Instead, start and stop can be
void, and a whole bunch of confusing generics disappear.
2016-07-01 16:17:42 -07:00
Ali Beyad 05998224d8 Adding repository index generational files
Before, a repository would maintain an index file (named 'index') per
repository, that contained the current snapshots in the repository.
This file was not atomically written, so repositories had to depend on
listing the blobs in the repository to determine what the current
snapshots are, and only rely on the index file if the repository does
not support the listBlobs operation.  This could cause an incorrect view
of the current snapshots in the repository if any prior snapshot delete
operations failed to delete snapshot metadata files.

This commit introduces the atomic writing of the index file, and because
atomic writes are not guaranteed if the file already exists, we write to
a generational index file (index-N, where N is the current generation).
We also maintain an index-latest file that contains the current
generation, for those repositories that cannot list blobs.

Closes #19002
Relates #18156
2016-07-01 17:52:57 -04:00
Ryan Ernst 40e1fe7d9e Remove unused ClusterService member of NodeService 2016-07-01 13:17:12 -07:00
Ryan Ernst f9eed91f33 Fix test oops 2016-07-01 13:00:33 -07:00
Ryan Ernst 2f12d1cb45 Merge branch 'master' into ingest_plugin_api 2016-07-01 12:58:11 -07:00
Ryan Ernst 62397c0c3d Fix node service to still work when http is disabled 2016-07-01 12:55:41 -07:00