Commit Graph

4714 Commits

Author SHA1 Message Date
Christoph Büscher d1703decee Adapting to changes in master 2017-04-22 22:06:06 +02:00
Christoph Büscher 5254731039 Merge branch 'master' into feature/rank-eval 2017-04-22 21:47:32 +02:00
Ryan Ernst 473e98981b Scripts: Remove unnecessary executable shortcut ()
ScriptService has two executable methods, one which takes a
CompiledScript, which is similar to search, and one that takes a raw
Script and both compiles and returns an ExecutableScript for it. The
latter is not needed, and the call sites which used one or the other
were mixed. This commit removes the extra executable method in favor of
callers first calling compile, then executable.
2017-04-21 17:53:03 -07:00
Ryan Ernst aadc33d260 Scripts: Remove unwrap method from executable scripts ()
The unwrap method was leftover from support javascript and python. Since
those languages are removed in 6.0, this commit removes the unwrap
feature from scripts.
2017-04-21 17:50:22 -07:00
Nik Everett caf376c8af Start building analysis-common module ()
Start moving built in analysis components into the new analysis-common
module. The goal of this project is:
1. Remove core's dependency on lucene-analyzers-common.jar which should
shrink the dependencies for transport client and high level rest client.
2. Prove that analysis plugins can do all the "built in" things by moving all
"built in" behavior to a plugin.
3. Force tests not to depend on any oddball analyzer behavior. If tests
need anything more than the standard analyzer they can use the mock
analyzer provided by Lucene's test infrastructure.
2017-04-19 18:51:34 -04:00
Nik Everett db0a5e4263 Painless: more testing for script_stack ()
`script_stack` is super useful when debugging Painless scripts
because it skips all the "weird" stuff involved that obfuscates
where the actual error is. It skips Painless's internals and
call site bootstrapping.

It works fine, but it didn't have many tests. This converts a
test that we had for line numbers into a test for the
`script_stack`. The line numbers test was an indirect test
for `script_stack`.
2017-04-18 22:52:59 -04:00
Ryan Ernst 212f24aa27 Tests: Clean up rest test file handling ()
This change simplifies how the rest test runner finds test files and
removes all leniency.  Previously multiple prefixes and suffixes would
be tried, and tests could exist inside or outside of the classpath,
although outside of the classpath never quite worked. Now only classpath
tests are supported, and only one resource prefix is supported,
`/rest-api-spec/tests`.

closes 
2017-04-18 15:07:08 -07:00
Nik Everett 0b15fde27a Start on custom whitelists for Painless ()
We'd like to be able to support context-sensitive whitelists in
Painless but we can't now because the whitelist is a static thing.
This begins to de-static the whitelist, in particular removing
the static keyword from most of the methods on `Definition` and
plumbing the static instance into the appropriate spots as though
it weren't static. Once we de-static all the methods we should be
able to fairly simply build context-sensitive whitelists.

The only "fun" bit of this is that I added another layer in the
chain of methods that bootstraps `def` calls. Instead of running
`invokedynamic` directly on `DefBootstrap` we now `invokedynamic`
`$bootstrapDef` on the script itself loads the `Definition` that
the script was compiled against and then calls `DefBootstrap`.

I chose to put `Definition` into `Locals` so I didn't have to
change the signature of all the `analyze` methods. I could have
do it another way, but that seems ok for now.
2017-04-18 10:39:42 -04:00
Adrien Grand 4632661bc7 Upgrade to a Lucene 7 snapshot ()
We want to upgrade to Lucene 7 ahead of time in order to be able to check whether it causes any trouble to Elasticsearch before Lucene 7.0 gets released. From a user perspective, the main benefit of this upgrade is the enhanced support for sparse fields, whose resource consumption is now function of the number of docs that have a value rather than the total number of docs in the index.

Some notes about the change:
 - it includes the deprecation of the `disable_coord` parameter of the `bool` and `common_terms` queries: Lucene has removed support for coord factors
 - it includes the deprecation of the `index.similarity.base` expert setting, since it was only useful to configure coords and query norms, which have both been removed
 - two tests have been marked with `@AwaitsFix` because of , which we intend to address after the merge
2017-04-18 15:17:21 +02:00
Nik Everett 25119a7e78 Harden painless test against "fun" caching ()
The JVM caches `Integer` objects. This is known. A test in Painless
was relying on the JVM not caching the particular integer `1000`.
It turns out that when you provide `-XX:+AggressiveOpts` the JVM
*does* cache `1000`, causing the test to fail when that is
specified.

This replaces `1000` with a randomly selected integer that we test
to make sure *isn't* cached by the JVM. *Hopefully* this test is
good enough. It relies on the caching not changing in between when
we check that the value isn't cached and when we run the painless
code. The cache now is a simple array but there is nothing
preventing it from changing. If it does change in a way that thwarts
this test then the test fail fail again. At least when that happens
the next person can see the comment about how it is important
that the integer isn't cached and can follow that line of inquiry.

Closes 
2017-04-17 13:44:05 -04:00
Jason Tedor 972bdc09ee Reject empty IDs
When indexing a document via the bulk API where IDs can be explicitly
specified, we currently accept an empty ID. This is problematic because
such a document can not be obtained via the get API. Instead, we should
rejected these requets as accepting them could be a dangerous form of
leniency. Additionally, we already have a way of specifying
auto-generated IDs and that is to not explicitly specify an ID so we do
not need a second way. This commit rejects the individual requests where
ID is specified but empty.

Relates 
2017-04-15 10:36:03 -04:00
Jay Modi 30ab8739a6 Closing a ReleasableBytesStreamOutput closes the underlying BigArray ()
This commit makes closing a ReleasableBytesStreamOutput release the underlying BigArray so
that we can use try-with-resources with these streams and avoid leaking memory by not returning
the BigArray. As part of this change, the ReleasableBytesStreamOutput adds protection to only
release the BigArray once.

In order to make some of the changes cleaner, the ReleasableBytesStream interface has been
removed. The BytesStream interface is changed to a abstract class so that we can use it as a
useable return type for a new method, Streams#flushOnCloseStream. This new method wraps a
given stream and overrides the close method so that the stream is simply flushed and not closed.
This behavior is used in the TcpTransport when compression is used with a
ReleasableBytesStreamOutput as we need to close the compressed stream to ensure all of the data
is written from this stream. Closing the compressed stream will try to close the underlying stream
but we only want to flush so that all of the written bytes are available.

Additionally, an error message method added in the BytesRestResponse did not use a builder
provided by the channel and instead created its own JSON builder. This changes that method to use
the channel builder and in turn the bytes stream output that is managed by the channel.

Note, this commit differs from 6bfecdf921 in that it updates
ReleasableBytesStreamOutput to handle the case of the BigArray decreasing in size, which changes
the reference to the BigArray. When the reference is changed, the releasable needs to be updated
otherwise there could be a leak of bytes and corruption of data in unrelated streams.

This reverts commit afd45c1432, which reverted .
2017-04-14 10:50:31 -04:00
Tim Brooks ffaac5a08a Simplify BulkProcessor handling and retry logic ()
This commit collapses the SyncBulkRequestHandler and
AsyncBulkRequestHandler into a single BulkRequestHandler. The new
handler executes a bulk request and awaits for the completion if the
BulkProcessor was configured with a concurrentRequests setting of 0.
Otherwise the execution happens asynchronously.

As part of this change the Retry class has been refactored.
withSyncBackoff and withAsyncBackoff have been replaced with two
versions of withBackoff. One method takes a listener that will be
called on completion. The other method returns a future that will been
complete on request completion.
2017-04-13 14:48:52 -05:00
Nik Everett e99f90fb46 Add more debugging information to rethrottles
I'm still trying to track down failures like:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+dockeralpine-periodic/1180/console

It looks like a task is hanging but I'm not sure why. So this
adds more logging for next time.
2017-04-12 08:37:31 -04:00
Jason Tedor 653619079c Skip two Painless branch tests on Windows
This commit skips the two Painless tests
EqualsTests#testBranchEqualsDefAndPrimitive and
EqualsTests#testBranchNotEqualsDefAndPrimitive on Windows as the tests
are repeatedly failing there.
2017-04-11 06:19:42 -04:00
Colin Goodheart-Smithe 0114f0061c Removes version 2.x constants from Version ()
* Removes version 2.x constants from Version

Closes 

* Addresses review comments
2017-04-11 08:31:22 +01:00
Luca Cavanna 2c545c064d Move getProperty method out of MultiBucketsAggregation.Bucket interface ()
The getProperty method is an internal method needed to run pipeline aggregations and retrieve info by path from the aggs tree. It is not needed in the MultiBucketsAggregation.Bucket interface, which is returned to users running aggregations from the transport client. The method is moved to the InternalMultiBucketAggregation class as that's where it belongs.
2017-04-10 13:35:01 +02:00
Nik Everett de6837b7ac Fix throttled reindex_from_remote ()
reindex_from_remote was using `TimeValue#toString` to generate the
scroll timeout which is bad because that generates fractional
time values that are useful for people but bad for Elasticsearch
which doesn't like to parse them. This switches it to using
`TimeValue#getStringRep` which spits out whole time values.

Closes to 

Makes  even more desirable
2017-04-07 15:56:52 -04:00
Martijn van Groningen 3d9671a668
[PERCOLATOR] Allowing range queries with now ranges inside percolator queries.
Before now ranges where forbidden, because the percolator query itself could get cached and then the percolator queries with now ranges that should no longer match, incorrectly will continue to match.
By disabling caching when the `percolator` is being used, the percolator can now correctly support range queries with now based ranges.

 I think this is the right tradeoff. The percolator query is likely to not be the same between search requests and disabling range queries with now ranges really disabled people using the percolator for their use cases.

 Also fixed an issue that existed in the percolator fieldmapper, it was unable to find forbidden queries inside `dismax` queries.

 Closes 
2017-04-07 08:44:43 +02:00
Tim Brooks 5b1fbe5e6c Decouple BulkProcessor from client implementation ()
This commit modifies the BulkProcessor to be decoupled from the
client implementation. Instead it just takes a
BiConsumer<BulkRequest, ActionListener<BulkResponse>> that executes
the BulkRequest.
2017-04-05 12:12:43 -05:00
Jason Tedor afd45c1432 Revert "Closing a ReleasableBytesStreamOutput closes the underlying BigArray ()"
This reverts commit 6bfecdf921.
2017-04-04 20:33:51 -04:00
Christoph Büscher 6cfbef73a0 Follow renaming of randomAsciiOfLength() to randomAlphaOfLength() 2017-04-04 18:31:00 +02:00
Christoph Büscher 024ed1b6ca Merge branch 'master' into feature/rank-eval 2017-04-04 18:23:41 +02:00
Jay Modi 6bfecdf921 Closing a ReleasableBytesStreamOutput closes the underlying BigArray ()
This commit makes closing a ReleasableBytesStreamOutput release the underlying BigArray so
that we can use try-with-resources with these streams and avoid leaking memory by not returning
the BigArray. As part of this change, the ReleasableBytesStreamOutput adds protection to only release the BigArray once.

In order to make some of the changes cleaner, the ReleasableBytesStream interface has been
removed. The BytesStream interface is changed to a abstract class so that we can use it as a
useable return type for a new method, Streams#flushOnCloseStream. This new method wraps a
given stream and overrides the close method so that the stream is simply flushed and not closed.
This behavior is used in the TcpTransport when compression is used with a
ReleasableBytesStreamOutput as we need to close the compressed stream to ensure all of the data
is written from this stream. Closing the compressed stream will try to close the underlying stream
but we only want to flush so that all of the written bytes are available.

Additionally, an error message method added in the BytesRestResponse did not use a builder
provided by the channel and instead created its own JSON builder. This changes that method to use the channel builder and in turn the bytes stream output that is managed by the channel.
2017-04-04 17:01:30 +01:00
Jason Tedor 3136ed1490 Rename random ASCII helper methods
This commit renames the random ASCII helper methods in ESTestCase. This
is because this method ultimately uses the random ASCII methods from
randomized runner, but these methods actually only produce random
strings generated from [a-zA-Z].

Relates 
2017-04-04 11:04:18 -04:00
Nik Everett ebd74f09cf Add extra debugging to reindex cancel tests
Adds more diagnostics when reindex's cancel tests fail. It fails
every once in a while and didn't have useful failure messages:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+5.3+multijob-unix-compatibility/os=amazon/86/consoleFull
2017-03-31 11:17:06 -04:00
Jim Ferenczi f3a925fdbe Fix reindex with a remote source on a version before 2.0.0 ()
Send the scroll id in the body as plain text when the remote version is before 2.0.0
2017-03-31 09:07:43 +02:00
Tim Brooks 5fa80a6521 Pass exception from sendMessage to listener ()
This commit changes the listener passed to sendMessage from a Runnable
to a ActionListener.

This change also removes IOException from the sendMessage signature.
That signature is misleading as it allows implementers to assume an
exception will be thrown in case of failure. That does not happen due
to Netty's async nature.
2017-03-30 15:08:23 -05:00
Dimitris Athanasiou 34f116eae3 Require explicit query in _delete_by_query API ()
As the query of a search request defaults to match_all,
calling _delete_by_query without an explicit query may
result in deleting all data.

In order to protect users against falling into that
pitfall, this commit adds a check to require the explicit
setting of a query.

Closes 
2017-03-28 15:44:57 +01:00
Jim Ferenczi 0e95c90e9f Upgrade to Lucene 6.5.0 () 2017-03-27 15:57:54 +02:00
Ryan Ernst 4cb8a0100c Build: Rewrite antlr regeneration in gradle ()
This change ports the regeneration of antlr parser/lexer into gradle
(but does still take advantage of ant calls where appropriate).
2017-03-24 09:44:53 -07:00
Ryan Ernst 8c53555b28 Tests: Use local clone build of 5.x with bwc tests ()
The current rest backcompat tests, which run against a mixed cluster of
5.x and 6.0 nodes, depend on snapshot builds of 5.x. However, this has
the potential for inconsistency that results in CI failures, and happens
quite often, whenever some backcompat logic is added to 5.x, but the bwc
test on master fails because the 5.x code has not yet been published as
a snapshot.

This change creates a git clone of the 5.x branch,
builds the zip distribution, and ties that into gradle substitutions for
the 5.x version.
2017-03-23 22:32:13 -07:00
Christoph Büscher 4a75ede208 Reformatting source to fit 100 character line length restriction 2017-03-23 20:20:22 +01:00
Christoph Büscher 96fc3aaf6f Merge branch 'master' into feature/rank-eval 2017-03-23 19:55:47 +01:00
AdityaJNair 63757efe9c Remove DocumentMapper#parse(String index, String type, String id, BytesReference source) ()
Removed `parse(String index, String type, String id, BytesReference source)` in DocumentMapper.java and replaced all of its use in Test files with `parse(SourceToParse source)`.

`parse(String index, String type, String id, BytesReference source)` was only used in test files and never in the main code so it was removed. All of the test files that used it was then modified to use `parse(SourceToParse source)` method that existing in DocumentMapper.java
2017-03-23 11:01:09 -04:00
Nik Everett 257a7d77ed Painless: Fix regex lexer and error messages ()
Without this change, if write a script with multiple regexes
*sometimes* the lexer will decide to look at them like one
big regex and then some trailing garbage. Like this discuss post:
https://discuss.elastic.co/t/error-with-the-split-function-in-painless-script/79021

```
def val = /\\\\/.split(ctx._source.event_data.param17);
if (val[2] =~ /\\./) {
  def val2 = /\\./.split(val[2]);
  ctx._source['user_crash'] = val2[0]
} else {
  ctx._source['user_crash'] = val[2]
}
```

The error message you get from the lexer is `lexer_no_viable_alt_exception`
right after the *second* regex.

With this change each regex is just a single regex like it ought to be.

As a bonus, while looking into this issue I found that the error
reporting for regexes wasn't very nice. If you specify an invalid
pattern then you get an error marker on the start of the pattern
with the JVM's regex error message which attempts to point you to the
location in the regex but is totally unreadable in the JSON response.

This change fixes the location to point to the appropriate spot
inside the pattern and removes the portion of the JVM's error message
that doesn't render well. It is no longer needed now that we point
users to the appropriate spot in the pattern.
2017-03-22 15:56:17 -04:00
Nik Everett bc65be2a65 Reindex: wait for cleanup before responding ()
Changes reindex and friends to wait until the entire request has
been "cleaned up" before responding. "Clean up" in this context
is clearing the scroll and (for reindex-from-remote) shutting
down the client. Failures to clean up are still only logged, not
returned to the user.

Closes 
2017-03-21 15:33:39 -04:00
Christoph Büscher f5388e5799 Adapting rank_eval integration tests 2017-03-14 12:21:28 -07:00
Christoph Büscher cf35545e2d Merge branch 'master' into feature/rank-eval 2017-03-13 17:36:13 -07:00
Jason Tedor 8dfb68cf1c Upgrade to Netty 4.1.9
This commit upgrades the Netty dependencies from version 4.1.8 to
version 4.1.9. This commit picks up a few bug fixes that impacted us:
 - Netty was incorrectly ignoring interfaces with self-assigned MAC
   addresses (e.g., instances running in Docker containers or on EC2)
 - incorrect handling of the Expect: 100-continue header

Relates 
2017-03-11 18:28:31 -08:00
Daniel Mitterdorfer 6f7cd71e1f Adjust default Netty receive predictor size to 64k ()
With this commit we change the default receive predictor size for Netty
from 32kB to 64kB as our testing has shown that this leads to less
allocations on smaller heaps like the default out of the box
configuration and this value also works reasonably well for larger
heaps.

Closes 
2017-03-11 17:32:35 -08:00
Jason Tedor 8e09eca9a6 Mute Painless lambda tests on JDK 9
This commit mutes a ton of Painless lambda tests on JDK 9. This commit
did not attempt to discover exactly which tests are failing, but instead
just blanket muted all tests in LambdaTests, FunctionRefTests, and
AugmentationTests.

Relates 
2017-03-02 22:36:26 -05:00
Jay Modi 01502893eb HTTP transport stashes the ThreadContext instead of the RestController ()
Previously, the RestController would stash the context prior to copying headers. However, there could be deprecation
log messages logged and in turn warning headers being added to the context prior to the stashing of the context. These
headers in the context would then be removed from the request and also leaked back into the calling thread's context.

This change moves the stashing of the context to the HttpTransport so that the network threads' context isn't
accidentally populated with warning headers and to ensure the headers added early on in the RestController are not
excluded from the response.
2017-03-02 14:44:01 -05:00
Luca Cavanna cc65a94fd4 [TEST] improve yaml test sections parsing ()
Throw error when skip or do sections are malformed, such as they don't start with the proper token (START_OBJECT). That signals bad indentation, which would be ignored otherwise. Thanks (or due to) our pull parsing code, we were still able to properly parse the sections, yet other runners weren't able to.

Closes 

* [TEST] fix indentation in matrix_stats yaml tests

* [TEST] fix indentation in painless yaml test

* [TEST] fix indentation in analysis yaml tests

* [TEST] fix indentation in generated docs yaml tests

* [TEST] fix indentation in multi_cluster_search yaml tests
2017-03-02 12:43:20 +01:00
Nik Everett 2dcdaa1c9d Mustache: don't extend AbstractComponent ()
Don't extend `AbstractComponent` in `MustacheScriptEngine` because
it doesn't buy anything.
2017-03-01 14:54:27 -05:00
Ryan Ernst 019263d664 Revert "Internal: Change version constant names for already released versions ()"
This reverts commit dc0e93ed62.
2017-02-28 14:45:13 -08:00
Ryan Ernst dc0e93ed62 Internal: Change version constant names for already released versions ()
We have many version constants in master that have already been
released, but are still marked (by naming convention) as unreleased.
This commit renames those version constants.
2017-02-28 13:05:44 -08:00
Tanguy Leroux 33eb6a13bf Tests: Fix RemoteScrollableHitSourceTests
With  , the expected exception is wrapped two times into a RuntimeException instead of being thrown directly.
2017-02-28 11:30:33 +01:00
Jim Ferenczi 5c84640126 Upgrade to lucene-6.5.0-snapshot-d00c5ca ()
Lucene upgrade
2017-02-27 18:39:04 +01:00
javanna 9a2dba3036 [TEST] add support for binary responses to REST tests infra 2017-02-27 12:27:03 +01:00
javanna dad025a6ad [TEST] move test for binary field to specific test file that sets Content-Type header explicitly 2017-02-27 12:27:03 +01:00
Christoph Büscher ddae32705c Adapting build.gradle to changes on master 2017-02-27 11:42:46 +01:00
Christoph Büscher 1f4c4d99b9 Merge branch 'master' into feature/rank-eval 2017-02-27 11:25:17 +01:00
Ryan Ernst 9df95def90 Build: Remove extra copies of netty license ()
The dependencyLicenses check has the ability to map multiple jar files
to the same license file. However, netty was not taking advantage of
this, and had duplicate copies of its license/notice files for each jar.
This commit reduces the copies to one and uses the mapping feature.
2017-02-24 14:40:07 -08:00
Jason Tedor f85a7aed37 Keep the pipeline handler queue small initially
This commit sets the intial size of the pipeline handler queue small to
prevent waste if pipelined requests are never sent. Since the queue will
grow quickly if pipeline requests are indeed set, this should not be
problematic.

Relates 
2017-02-23 14:17:46 -05:00
sabi0 09b3c7f270 Do not create String instances in 'Strings' methods accepting StringBuilder () 2017-02-23 10:57:34 -08:00
Christoph Büscher 8b1b152e91 Remove abstract InternalMetricsAggregation class ()
This class doesn't seem to do much other than to group together
certain types of aggregations.
2017-02-23 18:03:40 +01:00
Jason Tedor 3e69c38dbd Respect promises on pipelined responses
When pipelined responses are sent to the pipeline handler for writing,
they are not necessarily written immediately. They must be held in a
priority queue until all responses preceding the given response are
written. This means that when write is invoked on the handler, the
promise that is attached to the write invocation will not necessarily be
the promise associated with the responses that are written while the
queue is drained. To address this, the promise associated with a
pipelined response must be held with the response and then used when the
channel context is actually written to. This was introduced when
ensuring that the releasing promise is always chained through on write
calls lest the releasing promise never be invoked. This leads to many
failing test cases, so no new test cases are needed here.

Relates 
2017-02-23 09:32:43 -05:00
Jason Tedor 6ca90a61a6 Relocate a comment in HttpPipeliningHandler
This commit moves a comment in HttpPipeliningHandler as it makes more
sense for this comment to be where the field that it is explaining is
declared.
2017-02-22 20:51:18 -05:00
Jason Tedor 30f723d2b0 Add comments to HttpPipeliningHandler
This commit adds some comments explaining the design of
HttpPipeliningHandler.
2017-02-22 20:47:34 -05:00
Ryan Ernst 175bda64a0 Build: Rework integ test setup and shutdown to ensure stop runs when desired ()
Gradle's finalizedBy on tasks only ensures one task runs after another,
but not immediately after. This is problematic for our integration tests
since it allows multiple project's integ test clusters to be
simultaneously. While this has not been a problem thus far (gradle 2.13
happened to keep the finalizedBy tasks close enough that no clusters
were running in parallel), with gradle 3.3 the task graph generation has
changed, and numerous clusters may be running simultaneously, causing
memory pressure, and thus generally slower tests, or even failure if the
system has a limited amount of memory (eg in a vagrant host).

This commit reworks how integ tests are configured. It adds an
`integTestCluster` extension to gradle which is equivalent to the current
`integTest.cluster` and moves the rest test runner task to
`integTestRunner`.  The `integTest` task is then just a dummy task,
which depends on the cluster runner task, as well as the cluster stop
task. This means running `integTest` in one project will both run the
rest tests, and shut down the cluster, before running `integTest` in
another project.
2017-02-22 12:43:15 -08:00
Jason Tedor 708d11f54a Ensure that releasing listener is called
When sending a response to a client, we attach a releasing listener to
the channel promise. If the client disappears before the response is
sent, the releasing listener was never notified. The reason the
listeners were never notified was due to a mistaken invocation of write
and flush on the channel which has two overrides: one that takes an
existing promise, and one that does not and instead creates a new
promise. When the client disappears, it is this latter promise that is
notified, which does not contain the releasing listener. This commit
addreses this issue by invoking the override that passes our channel
promise through.

Relates 
2017-02-22 13:54:17 -05:00
javanna 594f00c582 Remove content type auto-detection from search templates
Now that search templates always get converted to json, we don't need to try and auto-detect their content-type, which anyways didn't work as expected before given that only json was really working.
2017-02-22 16:20:53 +01:00
javanna f2acf466aa Convert script/template objects to json format
Elasticsearch accepts multiple content-type formats, hence scripts can be stored/provided in json, yaml, cbor or smile. Yet the format that should be used internally is json. This is a problem mainly around search templates, as they only support json out of the four content-types, so instead of maintaining the content-type of the request we should rather convert the scripts/templates to json.

 Binary formats were not previously supported. If you stored a template in yaml format, you'd get back an error "No encoder found for MIME type [application/yaml]" when trying to execute it. With this commit the request content-type is independent from the template, which always gets converted to json internally. That is transparent to users and doesn't affect the content type of the response obtained when executing the template.
2017-02-22 16:20:53 +01:00
javanna 9391c6ffa9 Replace CustomMustacheFactory constant with same constant from Script (CONTENT_TYPE_OPTION) 2017-02-22 16:20:53 +01:00
Nik Everett 38d25a0369 Fix Painless's implementation of interfaces returning primitives ()
Fixes Painless to properly implement scripts that return primitives
and void. Adds some simple tests that we emit sane opcodes and some
other tests that we implement primitives as expected.

Mostly this is just a fix following up from  but there is one
thing I did really worth talking about, I think. So, before this script
Painless scripts could only ever return Object and they did would always
return null for paths that didn't return any values. Now that they
can return primitives the question is "what should Painless return
from paths that don't return any values?" And I answered that with
"whatever the JLS default value is". So 0/0L/0f/0d/false.
2017-02-21 17:10:55 -05:00
Martijn van Groningen 81d53470e7 percolator: add support for term extraction for MultiPhraseQuery 2017-02-21 21:10:55 +01:00
Nik Everett 9105672969 Allow painless to implement more interfaces ()
Generalizes three previously hard coded things in painless into
generic concepts:

1. The "main method" is no longer hardcoded to:
```
public abstract Object execute(Map<String, Object> params,
        Scorer scorer, LeafDocLookup doc, Object value);
```
Instead Painless's compiler takes an interface and implements it. It looks like:
```
public interface SomeScript {
    // Argument names we expose to Painless scripts
    String[] ARGUMENTS = new String[] {"a", "b"};
    // Method implemented by Painless script. Must be named execute but can have any parameters or return any value.
    Object execute(String a, int b);
    // Is the "a" argument used by the script?
    boolean uses$a();
}
SomeScript script = scriptEngine.compile(SomeScript.class, null, "the_script_here", emptyMap());
Object result = script.execute("a", 1);
```

`PainlessScriptEngine` now compiles all scripts to the new
`GenericElasticsearchScript` interface by default for compatibility
with the rest of Elasticsearch until it is able to use this new
ability.

2. `_score` and `ctx` are no longer hardcoded to be extracted from
`#score` and `params` respectively. Instead Painless's default
implementation of Elasticsearch scripts uses the `uses$_score` and
`uses$ctx` methods to determine if it is used and gives them
dummy values if they are not used.

3. Throwing the `ScriptException` is now handled by the Painless
script itself. That way Painless doesn't have to leak the metadata
that is required to build the fancy stack trace. And all painless scripts
get the fancy stack trace.
2017-02-21 14:08:57 -05:00
Jack Conradson fac2d954e3 Fix certain bad casts in Painless due to boxing/unboxing. () 2017-02-21 10:23:27 -08:00
Daniel Mitterdorfer 0744a00001 Set network receive predictor size to 32kb ()
Previously we calculated Netty' receive predictor size for HTTP and transport
traffic based on available memory and worker nodes. This resulted in a receive
predictor size between 64kb and 512kb. In our benchmarks this leads to increased
GC pressure.

With this commit we set Netty's receive predictor size to 32kb. This value is in
a sweet spot between heap memory waste (-> GC pressure) and effect on request
metrics (achieved throughput and latency numbers).

Closes 
2017-02-21 14:45:33 +01:00
Jay Modi b234644035 Enforce Content-Type requirement on the rest layer and remove deprecated methods ()
This commit enforces the requirement of Content-Type for the REST layer and removes the deprecated methods in transport
requests and their usages.

While doing this, it turns out that there are many places where *Entity classes are used from the apache http client
libraries and many of these usages did not specify the content type. The methods that do not specify a content type
explicitly have been added to forbidden apis to prevent more of these from entering our code base.

Relates 
2017-02-17 14:45:41 -05:00
Christoph Büscher 6f6b2933b1 Fixing compile issues after merging in master 2017-02-16 11:02:02 +01:00
Christoph Büscher cfa52f8b9a Merge branch 'master' into feature/rank-eval 2017-02-16 10:39:07 +01:00
Jason Tedor 0a5917d182 Fix get HEAD requests
Get HEAD requests incorrectly return a content-length header of 0. This
commit addresses this by removing the special handling for get HEAD
requests, and just relying on the general mechanism that exists for
handling HEAD requests in the REST layer.

Relates 
2017-02-15 13:07:29 -05:00
Ryan Ernst 79a1629f74 Fix line length 2017-02-14 21:23:21 -08:00
Jason Tedor 9e80e290d6 Add failing tests for expect header violations
This commit adds unit tests for two cases where Elasticsearch violates
expect header handling. These tests are marked as awaits fix.

Relates 
2017-02-14 19:24:22 -05:00
Jason Tedor 673754b1d5 Fix get source HEAD requests
Get source HEAD requests incorrectly return a content-length header of
0. This commit addresses this by removing the special handling for get
source HEAD requests, and just relying on the general mechanism that
exists for handling HEAD requests in the REST layer.

Relates 
2017-02-14 16:37:22 -05:00
Martijn van Groningen cab43707dc [percolator] Removed old 2.x bwc logic. 2017-02-14 22:17:17 +01:00
Simon Willnauer aef0665ddb Detach SearchPhases from AbstractSearchAsyncAction ()
Today all search phases are inner classes of AbstractSearchAsyncAction or one of it's
subclasses. This makes unit testing of these classes practically impossible. This commit
Extracts `DfsQueryPhase` and `FetchSearchPhase` or of the code that composes the actual
query execution types and moves most of the fan-out and collect code into an `InitialSearchPhase`
class that can be used to build initial search phases (phases that retry on shards). This will
make modification to these classes simpler and allows to easily compose or add new search phases
down the road if additional roundtrips are required.
2017-02-14 12:34:25 +01:00
Jason Tedor 5343b87502 Handle bad HTTP requests
When Netty decodes a bad HTTP request, it marks the decoder result on
the HTTP request as a failure, and reroutes the request to GET
/bad-request. This either leads to puzzling responses when a bad request
is sent to Elasticsearch (if an index named "bad-request" does not exist
then it produces an index not found exception and otherwise responds
with the index settings for the index named "bad-request"). This commit
addresses this by inspecting the decoder result on the HTTP request and
dispatching the request to a bad request handler preserving the initial
cause of the bad request and providing an error message to the client.

Relates 
2017-02-13 17:39:25 -05:00
Jay Modi 61e383813d Make the version of the remote node accessible on a transport channel ()
This commit adds a new method to the TransportChannel that provides access to the version of the
remote node that the response is being sent on and that the request came from. This is helpful
for serialization of data attached as headers.
2017-02-13 15:15:57 -05:00
jaymode d8d03f45c2
Fix communication with 5.3.0 nodes
This commit fixes communication with 5.3.0 nodes to send XContentType to these nodes since  was backported to the
5.3 branch.
2017-02-13 13:15:51 -05:00
Jason Tedor 0f21ed5b70 Fix template HEAD requests
Template HEAD requests incorrectly return a content-length header of
0. This commit addresses this by removing the special handling for
template HEAD requests, and just relying on the general mechanism that
exists for handling HEAD requests in the REST layer.

Relates 
2017-02-11 18:30:16 -05:00
Jason Tedor a6158398dd Fix index HEAD requests
Index HEAD requests incorrectly return a content-length header of
0. This commit addresses this by removing the special handling for index
HEAD requests, and just relying on the general mechanism that exists for
handling HEAD requests in the REST layer.

Relates 
2017-02-10 09:44:01 -05:00
Jason Tedor 7ac44656df Fix alias HEAD requests
Alias HEAD requests incorrectly return a content-length header of
0. This commit addresses this by removing the special handling for alias
HEAD requests, and just relying on the general mechanism that exists for
handling HEAD requests in the REST layer.

Relates 
2017-02-10 09:19:35 -05:00
Adrien Grand 709cc9ba65 Upgrade to lucene-6.5.0-snapshot-f919485. () 2017-02-10 15:08:47 +01:00
Tanguy Leroux e2e5937455 Use `typed_keys` parameter to prefix suggester names by type in search responses ()
This pull request reuses the typed_keys parameter added in , but this time it applies it to suggesters. When set to true, the suggester names in the search response will be prefixed with a prefix that reflects their type.
2017-02-10 10:53:38 +01:00
Nik Everett 0250c7ab18 Fix reindex test after toString change
Weakens the assertion on wait_for_active_shards so that we don't
check the toString of the bulk request because it isn't important.

Relates to 
2017-02-09 16:48:40 -05:00
Tim Brooks a331405aff Isolated SocketPermissions to Netty ()
Netty 4.1.8 wraps connect and accept operations in doPrivileged blocks.
This means that we not need to give permissions to the entire transport
module. Additionally this commit deletes the privileged socket channel
and privileged server socket chanel.
2017-02-09 10:00:25 -06:00
Tanguy Leroux 3553522328 Add parameter to prefix aggs name with type in search responses ()
This pull request adds a new parameter to the REST Search API named `typed_keys`. When set to true, the aggregation names in the search response will be prefixed with a prefix that reflects the internal type of the aggregation.

Here is a simple example:
```
GET /_search?typed_keys
{
    "aggs": {
        "tweets_per_user": {
            "terms": {
                "field": "user"
            }
        }
    },
    "size": 0
}
```

And the response:

```
{
    "aggs": {
        "sterms:tweets_per_user": {
            ...
        }
    }
}
```

This parameter is intended to make life easier for REST clients that could parse back the prefix and could detect the type of the aggregation to parse. It could also be implemented for suggesters.
2017-02-09 11:19:04 +01:00
Tim Brooks 735e5b1983 Upgrade to Netty 4.1.8 ()
This commit upgrades the Netty dependency to version 4.1.8.Final.
2017-02-08 11:44:36 -06:00
Simon Willnauer ecb01c15b9 Fold InternalSearchHits and friends into their interfaces ()
We have a bunch of interfaces that have only a single implementation
for 6 years now. These interfaces are pretty useless from a SW development
perspective and only add unnecessary abstractions. They also require
lots of casting in many places where we expect that there is only one
concrete implementation. This change removes the interfaces, makes
all of the classes final and removes the duplicate `foo` `getFoo` accessors
in favor of `getFoo` from these classes.
2017-02-08 14:40:08 +01:00
Tim Brooks fcc568fd8d Add methods requiring connect to forbidden apis ()
This is related to . This commit adds calls that require
SocketPermission connect to forbidden APIs.

The following calls are now forbidden:

- java.net.URL#openStream()
- java.net.URLConnection#connect()
- java.net.URLConnection#getInputStream()
- java.net.Socket#connect(java.net.SocketAddress)
- java.net.Socket#connect(java.net.SocketAddress, int)
- java.nio.channels.SocketChannel#open(java.net.SocketAddress)
- java.nio.channels.SocketChannel#connect(java.net.SocketAddress)
2017-02-07 14:41:50 -06:00
Boaz Leskes ba06c14a97 TransportService.connectToNode should validate remote node ID ()
 gave us the ability to open low level temporary connections to remote node based on their address. With this use case out of the way, actual full blown connections should validate the node on the other side, making sure we speak to who we think we speak to. This helps in case where multiple nodes are started on the same host and a quick node restart causes them to swap addresses, which in turn can cause confusion down the road.
2017-02-07 22:11:32 +02:00
Tim Brooks 27b7d9bd8d Add FileSystemUtil method to read 'file:/' URLs ()
As part of  we are going to forbid usage of api
java.net.URL#openStream(). However in a number of places across the
we use this method to read files from the local filesystem. This commit
introduces a helper method openFileURLStream(URL url) to read files
from URLs. It does specific validation to only ensure that file:/
urls are read.

Additionlly, this commit removes unneeded method
FileSystemUtil.newBufferedReader(URL, Charset). This method used the
openStream () method which will soon be forbidden. Instead we use the
Files.newBufferedReader(Path, Charset).
2017-02-07 10:24:22 -06:00
Jay Modi c898e8ab83 Add support for newline delimited JSON Content-Type ()
This commit adds support for the newline delimited JSON Content-Type, which is how
the bulk, multi-search, and multi-search template APIs expect data to be formatted. The
`elasticsearch-js` client has also been using this content type for these types of requests.

Closes 
2017-02-07 09:20:06 -05:00
Nik Everett 0d6e622242 Make dates be ReadableDateTimes in scripts ()
Instead of longs. If you want millis since epoch you can call doc.date_field.value.millis.

Relates to 
2017-02-06 16:44:56 -05:00
Nicholas Knize 1c9fdfd1b3 Remove GeoPointFieldMapper abstraction
In order to support the evolving GeoPoint encodings in Lucene 5 and 6, ES 2.x and 5.x implements an abstraction layer to the GeoPointFieldMapper classes. As of 5.x the geo_point field mapper settled on using Lucene's more performant LatLonPoint field type and deprecated all other encodings. In 6.0 all encodings except LatLonPoint have been removed rendering this abstraction layer useless. This commit removes the abstraction layer and renames the LatLonPointFieldMapper back to GeoPointFieldMapper to mantain consistency with ES field naming.
2017-02-06 14:17:21 -06:00
Adrien Grand c8496fc4f4 Upgrade to Lucene 6.4.1. () 2017-02-06 09:28:43 +01:00
Nik Everett b0c9759441 Painless: Don't allow casting from void to def ()
Painless can cast anything into the magic type `def` but it
really shouldn't try to cast **nothing** into `def`. That causes
the byte code generation library to freak out a little.

Closes 
2017-02-03 16:38:47 -05:00
Christoph Büscher dde2a09ba5 Updating rank-eval module after major changes on master 2017-02-03 21:17:46 +01:00
Nik Everett 9ca871af7e Test: weaken assertion in fix sliced reindex test
This test was using initial count of slices instead of the count
of unfinished slices to pick the expected throttle. Unfortunely
due to race conditions the actual rethrottle count is between the
two. So we weaken the assertion from "the new throttle is exactly X"
to "the new throttle is between X and Y (inclusive)".
2017-02-03 13:00:49 -05:00
Christoph Büscher 4cb8d9d08c Merge branch 'master' into feature/rank-eval
Conflicts:
	core/src/main/java/org/elasticsearch/script/Script.java
        docs/reference/search.asciidoc
2017-02-03 17:27:20 +01:00
Tim Brooks f70188ac58 Remove connect SocketPermissions from core ()
This is related to . Core no longer needs `SocketPermission`
`connect`.

This permission is relegated to these modules/plugins:
- transport-netty4 module
- reindex module
- repository-url module
- discovery-azure-classic plugin
- discovery-ec2 plugin
- discovery-gce plugin
- repository-azure plugin
- repository-gcs plugin
- repository-hdfs plugin
- repository-s3 plugin

And for tests:
- mocksocket jar
- rest client
- httpcore-nio jar
- httpasyncclient jar
2017-02-03 09:39:56 -06:00
Christoph Büscher c33f894846 Fixing compilation problem in Eclipse () 2017-02-03 16:16:51 +01:00
Nik Everett 18eb0827e6 Reindex: do not log when can't clear old scroll ()
Versions of Elasticsearch prior to 2.0 would return a scroll id
even with the last scroll response. They'd then automatically
clear the scroll because it is empty. When terminating reindex
will attempt to clear the last scroll it received, regardless of
the remote version. This quiets the warning when the scroll cannot
be cleared for versions before 2.0.

Closes 
2017-02-03 10:08:27 -05:00
Jason Tedor 9a0b216c36 Upgrade checkstyle to version 7.5
This commit upgrades the checkstyle configuration from version 5.9 to
version 7.5, the latest version as of today. The main enhancement
obtained via this upgrade is better detection of redundant modifiers.

Relates 
2017-02-03 09:46:44 -05:00
Nik Everett ea4eb06b0a Test: Make update-by-query test more resilient
`UpdateByQueryWhileModifyingTests#testUpdateWhileReindexing`
runs update-by-query and concurrently updates, asserting that
the update-by-query never reverts any changes made by the update.
It is a smoke test for concurrent updates.

Now, it expects to hit a certain number of version conflicts
during the updates. This is normal as it is racing the
update-by-query. We have a maximum number of failures we
expect (10) and I'd never seen us come close until
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+5.x+multijob-unix-compatibility/os=sles/495/console

This bumps the max failures from 10 to 50 and improves
logging a bit. If we continue to see this failure then we have
some other issue.

Closes 
2017-02-03 09:18:26 -05:00
Jay Modi 7520a107be Optionally require a valid content type for all rest requests with content ()
This change adds a strict mode for xcontent parsing on the rest layer. The strict mode will be off by default for 5.x and in a separate commit will be enabled by default for 6.0. The strict mode, which can be enabled by setting `http.content_type.required: true` in 5.x, will require that all incoming rest requests have a valid and supported content type header before the request is dispatched. In the non-strict mode, the Content-Type header will be inspected and if it is not present or not valid, we will continue with auto detection of content like we have done previously.

The content type header is parsed to the matching XContentType value with the only exception being for plain text requests. This value is then passed on with the content bytes so that we can reduce the number of places where we need to auto-detect the content type.

As part of this, many transport requests and builders were updated to provide methods that
accepted the XContentType along with the bytes and the methods that would rely on auto-detection have been deprecated.

In the non-strict mode, deprecation warnings are issued whenever a request with body doesn't provide the Content-Type header.

See 
2017-02-02 14:07:13 -05:00
Nik Everett ce8e042b66 Reindex: fix reindex-from-remote from <2.0 ()
In 5.2 we stopped sending the source parameter if the user didn't
specify it. This was a mistake as versions before 2.0 look like
they don't always include the `_source`. This is because reindex
requests some metadata fields. Anyway, now we say `"_source": true`
if there isn't a `_source` configured in the reindex request.

Closes 
2017-02-02 11:46:24 -05:00
Nik Everett 73bf29072f Painless: Fix def invoked qualified method refs ()
We were incorrectly resolving qualified method references at run
time when invoked on `def`. This lead to errors like
`The struct with name [org] has not been defined.` when attempting

```
doc.date.dates.stream().map(
  org.joda.time.ReadableDateTime::centuryOfEra
).collect(Collectors.toList())
```
2017-02-02 10:15:03 -05:00
Nik Everett dacc150934 Expose multi-valued dates to scripts and document painless's date functions ()
Implemented by wrapping an array of reused `ModuleDateTime`s that
we grow when needed. The `ModuleDateTime`s are reused when we
move to the next document.

Also improves the error message returned when attempting to modify
the `ScriptdocValues`, removes a couple of allocations, and documents
that the date functions are available in Painless.

Relates to 
2017-02-01 21:57:07 -05:00
Jack Conradson 3d2626c4c6 Change Namespace for Stored Script to Only Use Id ()
Currently, stored scripts use a namespace of (lang, id) to be put, get, deleted, and executed. This is not necessary since the lang is stored with the stored script. A user should only have to specify an id to use a stored script. This change makes that possible while keeping backwards compatibility with the previous namespace of (lang, id). Anywhere the previous namespace is used will log deprecation warnings.

The new behavior is the following:

When a user specifies a stored script, that script will be stored under both the new namespace and old namespace.

Take for example script 'A' with lang 'L0' and data 'D0'. If we add script 'A' to the empty set, the scripts map will be ["A" -- D0, "A#L0" -- D0]. If a script 'A' with lang 'L1' and data 'D1' is then added, the scripts map will be ["A" -- D1, "A#L1" -- D1, "A#L0" -- D0].

When a user deletes a stored script, that script will be deleted from both the new namespace (if it exists) and the old namespace.

Take for example a scripts map with {"A" -- D1, "A#L1" -- D1, "A#L0" -- D0}. If a script is removed specified by an id 'A' and lang null then the scripts map will be {"A#L0" -- D0}. To remove the final script, the deprecated namespace must be used, so an id 'A' and lang 'L0' would need to be specified.

When a user gets/executes a stored script, if the new namespace is used then the script will be retrieved/executed using only 'id', and if the old namespace is used then the script will be retrieved/executed using 'id' and 'lang'
2017-01-31 13:27:02 -08:00
Nik Everett 2e48fb8294 Move delete by query helpers into core ()
This moves the building blocks for delete by query into core. This
should enabled two thigns:
1. Plugins other than reindex to implement "bulk by scroll" style
operations.
2. Plugins to directly call delete by query. Those plugins should
be careful to make sure that task cancellation still works, but
this should be possible.

Notes:
1. I've mostly just moved classes and moved around tests methods.
2. I haven't been super careful about cohesion between these core
classes and reindex. They are quite interconnected because I wanted
to make the change as mechanical as possible.

Closes 
2017-01-27 16:09:18 -05:00
Nik Everett 8a2d424d68 Generate reference links for painless API ()
Adds "Appending B. Painless API Reference", a reference of all classes
and methods available from Painless. Removes links to java packages
because they contain methods that we don't expose and don't contain
methods that we do expose (the ones in Augmentation). Instead this
generates a list of every class and every exposed method using the same
type information available to the
interpreter/compiler/whatever-we-call-it. From there you can jump to
the relevant docs.

Right now you build all the asciidoc files by running
```
gradle generatePainlessApi
```

These files are expected to be committed because we build the docs
without running `gradle`.

Also changes the output of `Debug.explain` so that it is easy to
search for the class in the generated reference documentation.

You can also run it in an IDE safely if you pass the path to the
directory in which to generate the docs as the first parameter. It'll
blow away the entire directory an recreate it from scratch so be careful.

And then you can build the docs by running something like:
```
../docs/build_docs.pl --out ../built_docs/ --doc docs/reference/index.asciidoc --open
```

That is, if you have checked out https://github.com/elastic/docs in
`../docs`. Wait a minute or two and your browser will pop open in with
all of Elasticsearch's reference documentation. If you go to
`http://localhost:8000/painless-api-reference.html` you can see this
list. Or you can get there by following the links to `Modules` and
`Scripting` and `Painless` and then clicking the link in the paragraphs
below titled `Appendix B. Painless API Reference`.

I like having these in asciidoc because we can deep link to them from the
rest of the guide with constructs like
`<<painless-api-reference-Object-hashCode-0>>` and
`<<painless-api-reference->>` and we get link checking. Then the only
brittle link maintenance bit is the link generation for javadoc. Which
sucks. But I think it is important that we link to the methods directly
so they are easy to find.

Relates to 
2017-01-26 10:39:19 -05:00
Tim Brooks 719e75bb3f Add repository-url module and move URLRepository ()
This is related to . URLRepository requires SocketPermission
connect. This commit introduces a new module called "repository-url"
where URLRepository will reside. With the new module, permissions can
be removed from core.
2017-01-25 17:09:25 -06:00
Tal Levy e9a68b3287 fix date-processor to a new default year for every new pipeline execution. ()
Beforehand, the DateProcessor constructs its joda pattern formatter during processor
construction. This led to newly ingested documents being defaulted to
the year that the pipeline was constructed, not that of processing.

Fixes .
2017-01-25 15:09:07 -08:00
Chris Earle f0f75b187a Support Preemptive Authentication with RestClient ()
This adds the necessary `AuthCache` needed to support preemptive authorization. By adding every host to the cache, the automatically added `RequestAuthCache` interceptor will add credentials on the first pass rather than waiting to do it after _each_ anonymous request is rejected (thus always sending everything twice when basic auth is required).
2017-01-24 11:34:05 -05:00
Luca Cavanna 47c0e13a3b Stop returning "es." internal exception headers as http response headers ()
move "es." internal headers to separate metadata set in ElasticsearchException and stop returning them as response headers

Closes 

* [TEST] remove ESExceptionTests, move its methods to ElasticsearchExceptionTests or ExceptionSerializationTests
2017-01-24 16:12:45 +01:00
Nik Everett 28cfc533e2 Generate javadoc jar for painless's public API ()
The simplest way to do that is to move the public API into a
new package and generate javadoc for that package.
2017-01-23 17:16:20 -05:00
Jim Ferenczi e48bc2eed7 Add field collapsing for search request ()
* Add top hits collapsing to search request

The field collapsing is done with a custom top docs collector that "collapse" search hits with same field value.
The distributed aspect is resolve using the two passes that the regular search uses. The first pass "collapse" the top hits, then the coordinating node merge/collapse the top hits from each shard.

```
GET _search
{
   "collapse": {
      "field": "category",
   }
}
```

This change also adds an ExpandCollapseSearchResponseListener that intercepts the search response and expands collapsed hits using the CollapseBuilder#innerHit} options.
The retrieval of each inner_hits is done by sending a query to all shards filtered by the collapse key.

```
GET _search
{
   "collapse": {
      "field": "category",
      "inner_hits": {
	"size": 2
      }
   }
}
```
2017-01-23 16:33:51 +01:00
Tim Brooks a4ac29c005 Add single static instance of SpecialPermission ()
This commit adds a SpecialPermission constant and uses that constant
opposed to introducing new instances everywhere.

Additionally, this commit introduces a single static method to check that
the current code has permission. This avoids all the duplicated access
blocks that exist currently.
2017-01-21 12:03:52 -06:00
Jim Ferenczi 8028578305 Upgrade to Lucene 6.4.0 ()
* Upgrade to Lucene 6.4.0

`ValueSource`s are now converted to `DoubleValueSource`s using the Lucene adapter made for the migration to the new API in 6.4.0.
2017-01-21 04:48:01 +01:00
Nik Everett 6265ef1c1b Deguice rest handlers ()
There are presently 7 ctor args used in any rest handlers:
* `Settings`: Every handler uses it to initialize a logger and
  some other strange things.
* `RestController`: Every handler registers itself with it.
* `ClusterSettings`: Used by `RestClusterGetSettingsAction` to
  render the default values for cluster settings.
* `IndexScopedSettings`: Used by `RestGetSettingsAction` to get
  the default values for index settings.
* `SettingsFilter`: Used by a few handlers to filter returned
  settings so we don't expose stuff like passwords.
* `IndexNameExpressionResolver`: Used by `_cat/indices` to
  filter the list of indices.
* `Supplier<DiscoveryNodes>`: Used to fill enrich the response
  by handlers that list tasks.

We probably want to reduce these arguments over time but
switching construction away from guice gives us tighter
control over the list of available arguments.

These parameters are passed to plugins using
`ActionPlugin#initRestHandlers` which is expected to build and
return that handlers immediately. This felt simpler than
returning an reference to the ctors given all the different
possible args.

Breaks java plugins by moving rest handlers off of guice.
2017-01-20 11:48:51 -05:00
Tim Brooks bc16162d21 Remove accept SocketPermissions from core ()
This is related to . Core no longer needs SocketPermission 
accept. This permission is relegated to the transport-netty4 module 
and (for tests) to the mocksocket jar.
2017-01-20 09:27:45 -06:00
Nik Everett 22f1c9fa0f Remove @header we no longer need 2017-01-19 11:44:13 -05:00
Nik Everett bb83c283bb Make lexer abstract 2017-01-19 11:41:50 -05:00
Nik Everett dbb4a2ca6c Move lexer hacks to EnhancedPainlessLexer
This "feels" nicer. Less classes at least.
2017-01-19 11:23:16 -05:00
Nik Everett e2da6a8ee5 Improve painless's javadocs
Hopefully useful references.
2017-01-19 11:04:08 -05:00
Tim Brooks a10aa8aade Add TestWithDependenciesPlugin to build ()
This commit adds a MessyRestTestPlugin to the gradle build. It extends 
StandaloneRestTestPlugin. The main piece of functionality that it adds 
is to copy plugin-metadata from dependencies into the 
generated-resources for the current test source. This is necessary to 
ensure that permissions for dependencies are applied when running the 
tests.

A current limitation is that the permissions are applied differently 
than in the distribution sources. When permissions are granted to all 
depedencies for a module or plugin, the permissions are granted to all 
dependencies on the classpath for tests besides a few hardcoded 
exclusions:
- es core
- es test framework
- lucene test framework
- randomized runner
- junit library
2017-01-19 09:43:53 -06:00
Nik Everett 3ce41a0e15 Painless: Add augmentation to string for base 64 ()
We don't want to expose `String#getBytes` which is required for
`Base64.getEncoder.encode` to work because we're worried about
character sets. This adds `encodeBase64` and `decodeBase64`
methods to `String` in Painless that are duals of one another
such that:
`someString == someString.encodeBase64().decodeBase64()`.

Both methods work with the UTF-8 encoding of the string.

Closes 
2017-01-19 09:31:45 -05:00
Nik Everett ee5f8c4522 Consolidate some reindex utility classes ()
Everything that extended `AbstractAsyncBulkByScrollAction` also
extended `AbstractAsyncBulkIndexByScrollAction` so this removes
`AbstractAsyncBulkIndexByScrollAction`, merging it into
`AbstractAsyncBulkByScrollAction`.
2017-01-18 16:58:39 -05:00
Nik Everett 1fe74a6b4b Better error when can't auto create index ()
Changes the error message when `action.auto_create_index` or
`index.mapper.dynamic` forbids automatic creation of an index
from `no such index` to one of:
* `no such index and [action.auto_create_index] is [false]`
* `no such index and [index.mapper.dynamic] is [false]`
* `no such index and [action.auto_create_index] contains [-<pattern>] which forbids automatic creation of the index`
* `no such index and [action.auto_create_index] ([all patterns]) doesn't match`

This should make it more clear *why* there is `no such index`.

Closes 
2017-01-18 15:18:32 -05:00
Simon Willnauer 24e2847af2 Streamline foreign stored context restore and allow to perserve response headers ()
Today we do not preserve response headers if they are present on a transport protocol
response. While preserving these headers is not always desired, in the most cases we
should pass on these headers to have consistent results for depreciation headers etc.
yet, this hasn't been much of a problem since most of the deprecations are detected early
ie. on the coordinating node such that this bug wasn't uncovered until 

This commit allow to optionally preserve headers when a context is restored and also streamlines
the context restore since it leaked frequently into the callers thread context when the callers
context wasn't restored again.
2017-01-18 16:17:54 +01:00
Igor Motov 500548fcda Remove taskManager.registerChildTask
Instead of forcing each task to register all nodes where its children are running, this commit runs cancellation on all nodes. The task cancellation operation doesn't run too frequently, so this optimization doesn't seem to be worth additional complexity of the interface.
2017-01-17 18:07:31 -05:00
Ali Beyad e2977889b8 Allow comma delimited array settings to have a space after each entry ()
Previously, certain settings that could take multiple comma delimited
values would pick up incorrect values for all entries but the first if
each comma separated value was followed by a whitespace character.  For
example, the multi-value "A,B,C" would be correctly parsed as
["A", "B", "C"] but the multi-value "A, B, C" would be incorrectly parsed
as ["A", " B", " C"].

This commit allows a comma separated list to have whitespace characters
after each entry.  The specific settings that were affected by this are:

  cluster.routing.allocation.awareness.attributes
  index.routing.allocation.require.*
  index.routing.allocation.include.*
  index.routing.allocation.exclude.*
  cluster.routing.allocation.require.*
  cluster.routing.allocation.include.*
  cluster.routing.allocation.exclude.*
  http.cors.allow-methods
  http.cors.allow-headers

For the allocation filtering related settings, this commit also provides
validation of each specified entry if the filtering is done by _ip,
_host_ip, or _publish_ip, to ensure that each entry is a valid IP
address.

Closes 
2017-01-17 08:51:04 -06:00
Tanguy Leroux f5542ed47f Simplify ElasticsearchException rendering as a XContent ()
This commit tries to simplify the way ElasticsearchException are rendered to xcontent. It adds some documentation and renames and merges some methods. Current behavior is preserved, the goal is to be more readable and centralize everything in the ElasticsearchException class.
2017-01-17 15:44:49 +01:00
Tim Brooks 16a76d9bc0 Remove blocking TCP clients and servers ()
This commit removes the option to use the blocking variants of the TCP
transport server, TCP transport client, or http server.
2017-01-16 18:38:51 -06:00
Simon Willnauer f30b1f82ee Remove HttpServer and HttpServerAdapter in favor of a simple dispatch method ()
Today we have quite some abstractions that are essentially providing a simple
dispatch method to the plugins defining a `HttpServerTransport`. This commit
removes `HttpServer` and `HttpServerAdaptor` and introduces a simple `Dispatcher` functional
interface that delegate to `RestController` by default.

Relates to 
2017-01-16 21:06:08 +01:00
javanna a8a13bb46f replace custom functional interface with CheckedFunction in percolate module 2017-01-16 13:57:58 +01:00
Alexander Reelsen f6ee6e420b Indexing: Add shard id to indexing operation listener ()
The IndexingOperationListener interface did not provide any
information about the shard id when a document was indexed.

This commit adds the shard id as the first parameter to all methods
in the IndexingOperationListener.
2017-01-16 09:08:16 +01:00
Tim Brooks f4270f9914 Wrap netty accept/connect ops with doPrivileged ()
This is related to . netty channels require socket `connect` and
`accept` privileges. Netty does not currently wrap these operations
with `doPrivileged` blocks. These changes extend the netty channels
and wrap calls to the relevant super methods in doPrivileged blocks.
2017-01-13 14:27:09 -06:00
Zachary Tong 18fdc39b8c Increase visibility of doExecute so it can be used directly () 2017-01-13 09:42:02 -05:00
Nik Everett baed02bbe2 Whitelist some ScriptDocValues in painless ()
Without this whitelist painless can't use ip or binary doc values.

Closes 
2017-01-12 15:26:09 -05:00
Jason Tedor 126efea56c Upgrade to Netty 4.1.7
This commit upgrades the Netty dependency to version 4.1.7.Final,
picking up some important bug fixes.

Relates 
2017-01-12 10:58:21 -05:00
javanna 64c3212fdb Remove ParseFieldMatcher usages from IndexSettings 2017-01-12 14:43:35 +01:00
javanna 8072f168a3 Remove ParseFieldMatcher usages from QueryParseContext 2017-01-12 14:43:35 +01:00
Luca Cavanna 0f7d52df68 Remove some more ParseFieldMatcher usages () 2017-01-12 10:04:10 +01:00
Nik Everett 25a5f1869a Improve error message when reindex-from-remote gets bad json ()
Adds a message about how the remote is unlikely to be Elasticsearch.
This isn't as good as including the whole message from the remote but
we can't do that because we are stream parsing it and we don't want
to mark the whole request.

Closes 
2017-01-11 12:55:23 -05:00
Jack Conradson 0c694b3d19 Update loop counter to be higher (1000000) instead of (10000). 2017-01-11 09:22:24 -08:00
Nik Everett abb7d7841f Remove SearchRequestParsers ()
It is empty now that we've moved all the parsing into `namedObject`.
2017-01-11 10:28:14 -05:00
Luca Cavanna 0f391336f5 Clean up SearchShardTarget ()
* unify shard target setter

* Remove indexText member from SearchShardTarget

* Remove duplicated indexName getter from SearchShardTarget

* Remove duplicated shardId getter from SearchShardTarget

* Remove duplicated nodeIde getter from SearchShardTarget

* Rename SearchShardTarget#nodeIdText getter to getNodeIdText

* Remove unused InternalSearchHit#internalSourceRef unused method

* Remove unused InternalSearchHit#internalHighlightFields unused method

* Make SearchShardTarget members final
2017-01-11 10:08:31 +01:00
Nik Everett b71b8acf59 Remove ClusterService from ctors in reindex ()
Moves fetching the local node id into `NodeClient` which is a
fairly useful place to put it so you can generate task ids from
`NodeClient#executeLocally`.
2017-01-10 18:26:06 -05:00
Nik Everett d50f96e122 Remove InternalAggregation.Type ()
It is no longer needed. It used to contain a lot of strings
used by serialization but those have since been removed. Now
it is just another thing to pass around that we don't really
need.
2017-01-10 11:57:19 -05:00
Nik Everett 78bb56671e Fix reindex from remote clearing scroll ()
Reindex-from-remote had a race when it tried to clear the scroll. It
first starts the request to clear the scroll and then submits a task
to the generic threadpool to shutdown the client. These two things
race and, in my experience, closing the scroll generally loses. That
means that most of the time reindex-from-remote isn't clearing the
scrolls that it uses. This isn't the end of the world because we
flush old scroll contexts after a while but this isn't great.

Noticed while experimenting with .
2017-01-10 10:30:23 -05:00
Nik Everett 5ef78fd015 Fix source filtering in reindex-from-remote ()
Reindex-from-remote was accepting source filtering in the request
but ignoring it and setting `_source=true` on the search URI. This
fixes the filtering so it is piped through to the remote node and
adds tests for that.

Closes 
2017-01-10 09:00:12 -05:00
Martijn van Groningen cb2333dacd percolator: remove deprecated percolate and mpercolate apis 2017-01-10 11:18:27 +01:00
Nik Everett 3fb9254b95 Replace Suggesters with namedObject ()
Removes another parser registery type thing in favor of
`XContentParser#namedObject`.
2017-01-09 16:51:08 -05:00
Nik Everett 057194f9ab Fix test under windows
Silly `\r`.
2017-01-09 16:29:59 -05:00
Nik Everett e3f77b4795 Replace AggregatorParsers with namedObject ()
Removes `AggregatorParsers`, replacing all of its functionality with
`XContentParser#namedObject`.

This is the third bit of payoff from , one less thing to pass
around the entire application.
2017-01-09 13:59:38 -05:00
Nik Everett fc1f7c2147 Remove content-type detection from reindex-from-remote ()
If the remote doesn't return a content type then reindex
tried to guess the content-type. This didn't work most
of the time and produced a rather useless error message.
Given that Elasticsearch always returns the content-type
we are dropping content-type detection in favor of just
failing the request if the remote didn't return a content-type.

Closes 
2017-01-09 11:50:20 -05:00
Nik Everett f4884e0726 Replace SearchExtRegistry with namedObject ()
This is one of the last things in `SearchRequestParsers`.
2017-01-09 08:35:54 -05:00
javanna ded694fc83 Make StatusToXContent extend ToXContentObject and rename it to StatusToXContentObject
This also allows to make RestToXContentListener require ToXContentObject rather than ToXContent
2017-01-06 23:31:48 +01:00
javanna 4e49860f68 Make PercolateResponse a ToXContentObject 2017-01-06 23:31:48 +01:00
javanna d5510701a0 Make SearchResponse a ToXContentObject 2017-01-06 23:31:48 +01:00
javanna 45d4938fcc Migrate some more responses to ToXContentObject 2017-01-06 23:31:48 +01:00
Nik Everett f24ca5188a Fix some issues with painless's strings ()
1. Escape sequences we're working. For example `\\` is now correctly
interpreted as `\` instead of `\\`. Same with `\'` being `'` and
`\"` being `"`.
2. `'` delimited strings weren't allowed to contain `"`s but it looked
like they were intended to support it. Now they do.
3. Improves the error message when the script contains an invalid
escape sequence inside a string to include a list of the valid
escape sequences.

Closes 
2017-01-06 11:35:22 -05:00
javanna dea7d65439 remove ParseFieldMatcher usages from RestSearchTemplateAction 2017-01-05 19:33:04 +01:00
javanna 6102523033 remove ParseFieldMatcher usages from Script parsing code 2017-01-05 19:33:04 +01:00
javanna 9394792392 remove unused ParseFieldMatcher imports/arguments 2017-01-05 19:33:04 +01:00
Tim B be22a250b6 Replace Socket, ServerSocket, and HttpServer usages in tests with mocksocket versions ()
This integrates the mocksocket jar with elasticsearch tests. Mocksocket wraps actions requiring SocketPermissions in doPrivilege blocks. This will eventually allow SocketPermissions to be assigned to the mocksocket jar opposed to the entire elasticsearch codebase.
2017-01-04 14:38:51 -06:00
Adrien Grand f8998fece5 Upgrade to lucene-6.4.0-snapshot-084f7a0. () 2017-01-04 19:03:52 +01:00
Jason Tedor 96ba45e310 Fix stale comment in Netty4Utils
We previously named the thread using a frame from the stack trace, but
this was removed to simplify the code here. However, the comment
explaining this was left behind and this commit cleans that up.
2017-01-03 08:15:57 -05:00
Daniel Mitterdorfer 1ed64f0551 Eliminate unneccessary declaration of IOException
With this commit we remove the declaration of IOException from
assertWarnings and modify all call sites.

Checked with @javanna
2017-01-03 12:36:28 +01:00
javanna cd6b569286 Remove some usages of ParseFieldMatcher in favour of using ParseField directly
Relates to 
Relates to 
2016-12-31 09:24:44 +01:00
javanna df2acb3d9d Remove some more usages of ParseFieldMatcher in favour of using ParseField directly
Relates to 
Relates to 
2016-12-30 18:57:47 +01:00
javanna 6c54cbade4 Remove some more usages of ParseFieldMatcher in favour of using ParseField directly
Relates to 
Relates to 
2016-12-30 18:57:47 +01:00
javanna 45d010e874 Remove some usages of ParseFieldMatcher in favour of using ParseField directly
Relates to 
Relates to 
2016-12-30 18:57:47 +01:00
Martijn van Groningen 9ccdd3303d percolator: Fix NPE in percolator's 'now' range check for percolator queries with range queries.
Closes 
2016-12-27 22:56:01 +01:00
Tal Levy e6fb3a5d95 fix index out of bounds error in KV Processor ()
- checks for index-out-of-bounds
- added unit tests for failed `field_split` and `value_split` scenarios

missed this test in .
2016-12-27 10:57:11 -08:00
Nik Everett f5f2149ff2 Remove much ceremony from parsing client yaml test suites ()
* Remove a checked exception, replacing it with `ParsingException`.
* Remove all Parser classes for the yaml sections, replacing them with static methods.
* Remove `ClientYamlTestFragmentParser`. Isn't used any more.
* Remove `ClientYamlTestSuiteParseContext`, replacing it with some static utility methods.

I did not rewrite the parsers using `ObjectParser` because I don't think it is worth it right now.
2016-12-22 11:00:34 -05:00
Jason Tedor 7946396fe6 Introduce translog no-op
As the translog evolves towards a full operations log as part of the
sequence numbers push, there is a need for the translog to be able to
represent operations for which a sequence number was assigned, but the
operation did not mutate the index. Examples of how this can arise are
operations that fail after the sequence number is assigned, and gaps in
this history that arise when an operation is assigned a sequence number
but the operation never completed (e.g., a node crash). It is important
that these operations appear in the history so that they can be
replicated and replayed during recovery as otherwise the history will be
incomplete and local checkpoints will not be able to advance. This
commit introduces a no-op to the translog to set the stage for these
efforts.

Relates 
2016-12-21 23:08:16 -05:00
Nik Everett 567c65b0d5 Replace IndicesQueriesRegistry ()
* Switch query parsing to namedObject
* Remove IndicesQueriesRegistry
2016-12-21 09:05:14 -05:00
Tal Levy c53b2ee9cd introduce KV Processor in Ingest Node ()
Now you can parse field values of the `key=value` variety and have
`key` be inserted as a field name in an ingest document.

Closes .
2016-12-20 13:26:17 -08:00
Nik Everett a04dcfb95b Introduce XContentParser#namedObject ()
Introduces `XContentParser#namedObject which works a little like
`StreamInput#readNamedWriteable`: on startup components register
parsers under names and a superclass. At runtime we look up the
parser and call it to parse the object.

Right now the parsers take a context object they use to help with
the parsing but I hope to be able to eliminate the need for this
context as most what it is used for at this point is to move
around parser registries which should be replaced by this method
eventually. I make no effort to do so in this PR because it is
big enough already. This is meant to the a start down a road that
allows us to remove classes like `QueryParseContext`,
`AggregatorParsers`, `IndicesQueriesRegistry`, and
`ParseFieldRegistry`.

The goal here is to reduce the amount of plumbing required to
allow parsing pluggable things. With this you don't have to pass
registries all over the place. Instead you must pass a super
registry to fewer places and use it to wrap the reader. This is
the same tradeoff that we use for NamedWriteable and it allows
much, much simpler binary serialization. We think we want that
same thing for xcontent serialization.

The only parsing actually converted to this method is parsing
`ScoreFunctions` inside of `FunctionScoreQuery`. I chose this
because it is relatively self contained.
2016-12-20 11:05:24 -05:00
Nik Everett 73320566c1 Reindex test: catch exception name instead of reason
It looks like the exception reason can differ in different default
locales, so the build would fail in any non-English locale. This
switches the catch to the name of the exception which shouldn't
vary.
2016-12-20 10:00:14 -05:00
Nik Everett 8de4be9e4d Reinex test: don't fail if iis is running on port 0 2016-12-19 16:44:08 -05:00
Grzegorz Gajos f6b6e4e376 Added ability to remove pipelines via wildcards () ()
This commit is adding an ability to remove pipelines with wildcards.
2016-12-19 10:59:59 -08:00
javanna 5dae10db11 [TEST] add warnings check to ESTestCase
We are currenlty checking that no deprecation warnings are emitted in our query tests. That can be moved to ESTestCase (disabled in ESIntegTestCase) as it allows us to easily catch where our tests use deprecated features and assert on the expected warnings.
2016-12-19 19:39:56 +01:00
javanna 6a27628f12 Remove support for strict parsing mode
We return deprecation warnings as response headers, besides logging them. Strict parsing mode stayed around, but was only used in query tests, though we also introduced checks for deprecation warnings there that don't need strict parsing anymore (see ).

 We can then safely remove support for strict parsing mode. The final goal is to remove the ParseFieldMatcher class, but there are many many users of it. This commit prepares the field for the removal, by deprecating ParseFieldMatcher and making it effectively not needed. Strict parsing is removed from ParseFieldMatcher, and strict parsing is replaced in tests where needed with deprecation warnings checks.

 Note that the setting to enable strict parsing was never ported to the new settings infra hance it cannot be set in production. It is really only used in our own tests.

 Relates to 
2016-12-19 19:39:56 +01:00
Nik Everett 2d71ced221 Properly fail reindex-from-remote if can't detect content type 2016-12-19 12:51:38 -05:00
Isabel Drost-Fromm 7d849fb861 RankEvaluation: Add mutation based testing to RankEvalSpec ()
RankEvaluation: Add mutation based testing to RankEvalSpec, also fix RatedRequestsTests that were failing intermittently.
2016-12-19 15:15:22 +01:00
Yannick Welsch 63af03a104 Atomic mapping updates across types ()
This commit makes mapping updates atomic when multiple types in an index are updated. Mappings for an index are now applied in a single atomic operation, which also allows to optimize some of the cross-type updates and checks.
2016-12-19 14:39:50 +01:00
Isabel Drost-Fromm 46c30e6bc3 Make maximum number of parallel search requests configurable. ()
Problem: So far all rank eval requests are being executed in parallel. If there
are more than the search thread pool can handle, or if there are other search
requests executed in parallel rank eval can fail.

Solution: Make number of max_concurrent_searches configurable.

Name of configuration parameter is analogous to msearch. Default
max_concurrent_searches set to 10: Rank_eval isn't particularly time critical so
trying to avoid being more clever than probably needed here. Can set this value
through the API to a higher value anytime.

Fixes 
2016-12-19 13:05:49 +01:00
Isabel Drost-Fromm bdc32be8b7 Support specifying multiple templates ()
Problem: We introduced the ability to shorten the rank eval request by using a
template in . When playing with the API it turned out that there might be
use cases where - e.g. due to various heuristics - folks might want to translate
the original user query into more than just one type of Elasticsearch query.

Solution: Give each template an id that can later be referenced in the
actual requests.

Closes 
2016-12-19 12:49:15 +01:00
Isabel Drost-Fromm b1e0d698ac Merge branch 'master' into feature/rank-eval 2016-12-19 10:16:16 +01:00
Simon Willnauer ccfeac8dd5 Remove `doHandshake` test-only settings from TcpTransport ()
In  we introduce a test-only setting to simulate transport
impls that don't support handshakes. This commit implements the same logic
without a setting.
2016-12-18 09:26:53 +01:00
Tal Levy bb37167946 Enables the ability to inject serialized json fields into root of document. ()
The JSON processor has an optional field called "target_field".
If you don't specify target_field then target_field becomes what you specified as "field".
There isn't anyway to add the fields to the root of a document. By
setting `add_to_root`, now serialized fields will be inserted into the
top-level fields of the ingest document.

Closes .
2016-12-16 10:17:27 -08:00
Jason Tedor df43c268da Eagerly initialize Netty 4
Today we initialize Netty in a static initializer. We trigger this
method via static initializers from Netty-related classes, but we can
trigger this method earlier than we do to ensure that Netty is
initialized how we want it to be.
2016-12-15 13:24:47 -05:00
Isabel Drost-Fromm 5618d6ca49 Merge branch 'master' into feature/rank-eval 2016-12-15 10:29:26 +01:00
Tal Levy eaf82a6e7e compile ScriptProcessor inline scripts when creating ingest pipelines ()
Inline scripts defined in Ingest Pipelines are now compiled at creation time to preemptively catch errors on initialization of the pipeline.

Fixes .
2016-12-14 17:26:51 -08:00
Simon Willnauer 80d6539e9c Handle connection close / reset events gracefully during handshake ()
Low level handshake code doesn't handle situations gracefully if the connection
is concurrently closed or reset by peer. This commit adds the relevant code to
fail the handshake if the connection is closed.
2016-12-14 23:04:14 +01:00
Nik Everett 749039ad4f Consolidate the last easy parser construction ()
Moves the last of the "easy" parser construction into
`RestRequest`, this time with a new method
`RestRequest#contentParser`. The rest of the production
code that builds `XContentParser` isn't "easy" because it is
exposed in the Transport Client API (a Builder) object.
2016-12-14 15:41:25 -05:00
Isabel Drost-Fromm ea9f11965d Merge branch 'master' into feature/rank-eval 2016-12-14 12:47:12 +01:00
Adrien Grand 149ef74b26 Fix `missing` on aggs on `boolean` fields. ()
The creation of the `ValuesSource` used to pass `DateTimeZone.UTC` as a time
zone all the time in case of empty fields in spite of the fact that all doc
value formats but the date one reject this parameter.

This commit centralizes the creation of the `ValuesSource` and adds unit tests
to it.

Closes 
2016-12-14 10:03:09 +01:00
Daniel Mitterdorfer 7e5058037b Enable strict duplicate checks for JSON content
With this commit we enable the Jackson feature 'STRICT_DUPLICATE_DETECTION'
by default. This ensures that JSON keys are always unique. While this has
a performance impact, benchmarking has indicated that the typical drop in
indexing throughput is around 1 - 2%.

As a last resort, we allow users to still disable strict duplicate checks
by setting `-Des.json.strict_duplicate_detection=false` which is
intentionally undocumented.

Closes 
2016-12-14 09:35:53 +01:00
Nik Everett 49bdd29f91 Consolidate more parser creation into ESTestCase
This will make it easier to add the forthcoming required argument,
`NamedXContentRegistry`.
2016-12-13 20:28:41 -05:00
Nik Everett 872984d21a Continue consolidating `XContentParser` construction in tests ()
Consolidate more parser creation in tests

Moves more parser creation in tests to the `createParser` methods
in `ESTestCase`.
2016-12-13 17:22:39 -05:00
Tal Levy f56097b57a Fixes GrokProcessor's ignorance of named-captures with same name. ()
Grok was originally ignoring potential matches to named-capture groups
larger than one. For example, If you had two patterns containing the
same named field, but only the second pattern matched, it would fail to
pick this up.

This PR fixes this by exploring all potential places where a
named-capture was used and chooses the first one that matched.

Fixes .
2016-12-13 13:19:55 -08:00
Simon Willnauer 7a9b667e98 Introduce a low level protocol handshake ()
Today we rely on the version that the API user passes in together with the DiscoveryNode. This commit introduces a low level handshake where nodes exchange their version to be used with the transport protocol that is executed every time a connection to a node is established. This, on the one hand allows to change the wire protocol based on the version we are talking to even without a full cluster restart. Today we would need to carry on a BWC layer across major versions but with a handshake we can rely on the fact that the latest version of the previous minor executes a handshake and uses the latest protocol version across all communication with the N+1 version nodes.

This change is yet fully backwards compatible, a followup PR will remove the BWC in 6.0 once this has been back-ported to the 5.x branch
2016-12-13 21:06:23 +01:00
Isabel Drost-Fromm 58342d4c9a Add checks to RankEvalSpec to safe guard against missing parameters. ()
Add checks to RankEvalSpec to safe guard against missing parameters.

Fail early in case no metric is supplied, no rated requests are supplied or the search source builder is missing but no template is supplied neither.

Add stricter checks around rank eval request parsing: Fail if in a rated request we see both, a verbatim request as well as request
template parameters.


Relates to 
2016-12-13 11:21:57 +01:00
Isabel Drost-Fromm 6d1a658106 Fix compile issues after merging in master. 2016-12-13 11:16:53 +01:00
Isabel Drost-Fromm b2e8455745 Merge branch 'master' into feature/rank-eval 2016-12-13 10:45:22 +01:00
Adrien Grand 049fd3991c Remove `AggregationContext`. ()
This class is just a wrapper around `SearchContext`, so let's use
`SearchContext` directly. The change is mechanical, except the
`ValuesSourceConfig` class, where I moved the logic to get a `ValuesSource`
given a config.
2016-12-13 09:09:40 +01:00
Luca Cavanna 6d987a9b69 Remove support for empty queries ()
Our query DSL supports empty queries (`{}`), which have a different meaning depending on the query that holds it, either ignored, match_all or match_none. We deprecated the support for empty queries in 5.0, where we log a deprecation warning wherever they are used.

The way we supported it once we moved query parsing to the coordinating node was having an Optional<QueryBuilder> return type in all of our parse methods (called fromXContent). See . The central place for this was QueryParseContext#parseInnerQueryBuilder. We can now remove all the optional return types and simply throw an exception whenever an empty query is found.
2016-12-12 12:37:12 +01:00
Simon Willnauer 01d67e09b9 Detach handshake from connect to node ()
Today we connect and publish the nodes connection before we execute a
handshake with the node we connect to. In the case of connecting to a node
that won't pass the handshake this connection is already `published` and other
code paths can use it. This commit detaches the connection and the publish of the
connection such that `TransportService` can do a handshake before actually connect
and publish the connection.
2016-12-10 10:03:26 +01:00
Nik Everett 3adefb7b4a Begin centralizing XContentParser creation into RestRequest ()
To get  in cleanly we need to centralize as much `XContentParser` creation as possible into `RestRequest`. That'll mean we have to plumb the `NamedXContentRegistry` into fewer places.

This removes `RestAction.hasBody`, `RestAction.guessBodyContentType`, and `RestActions.getRestContent`, moving callers over to `RestRequest.hasContentOrSourceParam`, `RestRequest.contentOrSourceParam`, and `RestRequest.contentOrSourceParamParser` and `RestRequest.withContentOrSourceParamParserOrNull`. The idea is to use `withContentOrSourceParamParserOrNull` if you need to handle requests without any sort of body content and to use `contentOrSourceParamParser` otherwise.

I believe the vast majority of this PR to be purely mechanical but I know I've made the following behavioral change (I'll add more if I think of more):
* If you make a request to an endpoint that requires a request body and has cut over to the new APIs instead of getting `Failed to derive xcontent` you'll get `Body required`.
* Template parsing is now non-strict by default. This is important because we need to be able to deprecate things without requests failing.
2016-12-09 20:23:02 -05:00
Nik Everett fc2060ba7e Don't close rest client from its callback ()
If you try to close the rest client inside one of its callbacks then
it blocks itself. The thread pool switches the status to one that
requests a shutdown and then waits for the pool to shutdown. When
another thread attempts to honor the shutdown request it waits
for all the threads in the pool to finish what they are working on.
Thus thread a is waiting on thread b while thread b is waiting
on thread a. It isn't quite that simple, but it is close.

Relates to 
2016-12-09 10:39:51 -05:00
Adrien Grand 36f598138a Start using `ObjectParser` for aggs. ()
This is an attempt to start moving aggs parsing to `ObjectParser`. There is
still A LOT to do, but ObjectParser is way better than the way aggregations
parsing works today. For instance in most cases, we reject numbers that are
provided as strings, which we are supposed to accept since some client languages
(looking at you Perl) cannot make sure to use the appropriate types.

Relates to 
2016-12-09 09:45:16 +01:00
Ryan Ernst b1cef5fdf8 Remove 2.0 prerelease version constants ()
* Remove 2.0 prerelease version constants

This is a start to addressing . This removes:
* pre 2.0 snapshot format support
* automatic units addition to cluster settings
* bwc check for delete by query in pre 2.0 indexes
2016-12-08 21:48:35 -08:00
Lee Hinman ef64d230e7 Merge remote-tracking branch 'dakrone/index-seq-id-and-primary-term' 2016-12-08 19:47:21 -07:00
Lee Hinman ee22a477df Add internal _primary_term doc values field, fix _seq_no indexing
This adds the `_primary_term` field internally to the mappings. This field is
populated with the current shard's primary term.

It is intended to be used for collision resolution when two document copies have
the same sequence id, therefore, doc_values for the field are stored but the
filed itself is not indexed.

This also fixes the `_seq_no` field so that doc_values are retrievable (they
were previously stored but irretrievable) and changes the `stats` implementation
to more efficiently use the points API to retrieve the min/max instead of
iterating on each doc_value value. Additionally, even though we intend to be
able to search on the field, it was previously not searchable. This commit makes
it searchable.

There is no user-visible `_primary_term` field. Instead, the fields are
updated by calling:

```java
index.parsedDoc().updateSeqID(seqNum, primaryTerm);
```

This includes example methods in `Versions` and `Engine` for retrieving the
sequence id values from the index (see `Engine.getSequenceID`) that are only
used in unit tests. These will be extended/replaced by actual implementations
once we make use of sequence numbers as a conflict resolution measure.

Relates to 
Supercedes 

P.S. As a side effect of this commit, `SlowCompositeReaderWrapper` cannot be
used for documents that contain `_seq_no` because it is a Point value and SCRW
cannot wrap documents with points, so the tests have been updated to loop
through the `LeafReaderContext`s now instead.
2016-12-08 19:47:03 -07:00
Isabel Drost-Fromm 48c784026a Merge branch 'master' into feature/rank-eval 2016-12-08 15:22:45 +01:00
Christoph Büscher 7454a9647b Add fromXContent to HighlightField
This adds a fromXContent method and unit test to the HighlightField class so we
can parse it as part of a serch response. This is part of the preparation for
parsing search responses on the client side.
2016-12-07 16:32:44 +01:00
Isabel Drost-Fromm 165cec2757 Serialisation and validation checks for rank evaluation request components ()
Adds tests around serialisation/validation checks for rank evaluation request components

* Add null/ empty string checks to RatedDocument constructor
* Add mutation test to RatedDocument serialization tests.
* Reorganise rank-eval RatedDocument tests and add serialisation test.
* Add roundtrip serialisation testing for RatedRequests
* Adds serialisation testing and equals/hashcode testing for RatedRequest.
* Fixes a bug in previous equals implementation of RatedRequest along the way.
* Add roundtrip tests for Precision and ReciprocalRank
* Also fixes a bug with serialising ReciprocalRank.
* Add roundtrip testing for DiscountedCumulativeGain
* Add serialisation test for DocumentKey and fix test init
* Add check that relevant doc threshold is always positive for precision.
* Check that relevant threshold is always positive for precision and reciprocal
rank

Closes 
2016-12-07 11:47:47 +01:00
Isabel Drost-Fromm 5c6cdb90ad Merge branch 'master' into feature/rank-eval 2016-12-07 10:59:05 +01:00
Nik Everett ef83dbfbe6 Reindex: Better error message for pipeline in wrong place ()
`_update_by_query` supports specifying the `pipeline` to process the
documents as a url parameter but `_reindex` doesn't. It doesn't because
everything about the `_reindex` request that has to do with writing
the documents is grouped under the `dest` object in the request body.
This changes the response parameter from
`request [_reindex] contains unrecognized parameter: [pipeline]` to
`_reindex doesn't support [pipeline] as a query parmaeter. Specify it in the [dest] object instead.`
2016-12-06 14:55:46 -05:00
Isabel Drost-Fromm aa20ffcac7 Merge branch 'master' into feature/rank-eval 2016-12-06 10:33:24 +01:00
Isabel Drost-Fromm e0b15eafb0 Move rank-eval template compilation down to TransportRankEvalAction ()
Move rank-eval template compilation down to TransportRankEvalAction

Closes  and 

Search templates for rank_eval endpoint so far only worked when sent through
REST end point

However we also allow templates to be set through a Java API call to
"setTemplate" on that same spec. This doesn't go through template execution so
fails further down the line.

To make this work, moved template execution further down, probably to
TransportRankEvalAction.

* Add template IT test for Java API
* Move template compilation to TransportRankEvalAction
2016-12-06 10:30:25 +01:00
Ryan Ernst c8f241f284 Plugins: Remove response action filters ()
Action filters currently have the ability to filter both the request and
response. But the response side was not actually used. This change
removes support for filtering responses with action filters.
2016-12-05 16:14:04 -08:00
Nik Everett 2087234d74 Timeout improvements for rest client and reindex ()
Changes the default socket and connection timeouts for the rest
client from 10 seconds to the more generous 30 seconds.

Defaults reindex-from-remote to those timeouts and make the
timeouts configurable like so:
```
POST _reindex
{
  "source": {
    "remote": {
      "host": "http://otherhost:9200",
      "socket_timeout": "1m",
      "connect_timeout": "10s"
    },
    "index": "source",
    "query": {
      "match": {
        "test": "data"
      }
    }
  },
  "dest": {
    "index": "dest"
  }
}
```

Closes 
2016-12-05 10:54:51 -05:00
Isabel Drost-Fromm 5601049014 Merge branch 'master' into feature/rank-eval 2016-12-05 11:22:39 +01:00
Igor Motov c391b3fff6 Add proper descriptions to reindex, update-by-query and delete-by-query tasks.
Related to 
2016-12-02 21:46:38 -05:00
Jack Conradson 0ecdef026d Test fix for def equals test in Painless. ()
Closes 
2016-12-02 14:41:13 -08:00
Nik Everett 0c724b1878 Keep context during reindex's retries ()
* Keep context during reindex's retries

This fixes reindex and friend's retries to keep the context.

* Docs
2016-12-02 13:48:51 -05:00
Simon Willnauer 842e00c689 [TEST] Add back skip of external clusters 2016-12-02 11:53:33 +01:00
Simon Willnauer 572b4c3e72 Port assert from 5.x to master
I added an assertion to Netty4/Netty3Transport in 5.x that is not in
master yet. This commit port the assert to ensure we consumed all connection
in `connectToChannels`
2016-12-02 10:34:33 +01:00
Simon Willnauer adf9bd90a4 Remove legacy BWC test infrastructure and tests ()
We don't use the test infra nor do we run the tests. They might all be
entirely out of date. We also have a different BWC test infra in-place.
This change removes all of the legacy infra.
2016-12-02 08:06:20 +01:00
Simon Willnauer 155de53fe3 Add a connect timeout to the ConnectionProfile to allow per node connect timeouts ()
Timeouts are global today across all connections this commit allows to specify
a connection timeout per node such that depending on the context connections can
be established with different timeouts.

Relates to 
2016-12-01 15:39:49 +01:00
Boaz Leskes fe01c0f83b fix TemplateQueryBuilderTests & Murmur3FieldMapperTests 2016-12-01 14:21:57 +01:00
Simon Willnauer dd5256c324 Reduce number of connections per node depending on the nodes role ()
We currently treat every node equally when we establish connections to a node.
Yet, if we are not master eligible or can't hold any data there is no point in creating
a dedicated connection for sending the cluster state or running remote recoveries respectively.
The usage of STATE and RECOVERY connections on non-master and/or non-data nodes will result in an IllegalStateException.
2016-12-01 08:00:48 +01:00
Jason Tedor 6c45695d52 Add version 5.1.1
This commit removes the version constant for 5.1.0 (due to an
inadvertent release) and adds the version constant for 5.1.1.

Relates 
2016-11-30 11:14:17 -05:00
Luca Cavanna 5b8bdba12e Remove subrequests method from CompositeIndicesRequest () 2016-11-30 15:03:58 +01:00
Adrien Grand 6231009a8f Remove 2.x backward compatibility of mappings. ()
For the record, I also had to remove the geo-hash cell and geo-distance range
queries to make the code compile. These queries already throw an exception in
all cases with 5.x indices, so that does not hurt any more.

I also had to rename all 2.x bwc indices from `index-${version}` to
`unsupported-${version}` to make `OldIndexBackwardCompatibilityIT`
happy.
2016-11-30 13:34:46 +01:00
Isabel Drost-Fromm 2fecd8e394 Merge branch 'master' into feature/rank-eval 2016-11-30 11:17:56 +01:00
Luca Cavanna 6eaff9432d SearchTemplateRequest to implement CompositeIndicesRequest ()
SearchTemplateRequest to implement CompositeIndicesRequest

Given that SearchTemplateRequest effectively delegates to search when a search is being executed, it should implement the CompositeIndicesRequest interface. The subrequests method should return a single search request. When a search is not going to be executed, because we are in simulate mode, there are no inner requests, and there are no corresponding indices to that request either.

Closes 
2016-11-29 20:52:43 +01:00
Jim Ferenczi d791ddf704 Upgrade to lucene-6.4.0-snapshot-ec38570 ()
Set lucene version to 6.4.0-snapshot-ec38570 and update all the sha1s/license
Fix invalid combo after upgrade in query_string query. split_on_whitespace=false is disallowed if auto_generate_phrase_queries=true
Adapt the expectations of some tests to the new format of the Lucene explain output
2016-11-29 18:40:31 +01:00
Nicholas Knize af1ab68b64 Add RangeFieldMapper for numeric and date range types
Lucene 6.2 added index and query support for numeric ranges. This commit adds a new RangeFieldMapper for indexing numeric (int, long, float, double) and date ranges and creating appropriate range and term queries. The design is similar to NumericFieldMapper in that it uses a RangeType enumerator for implementing the logic specific to each type. The following range types are supported by this field mapper: int_range, float_range, long_range, double_range, date_range.

Lucene does not provide a DocValue field specific to RangeField types so the RangeFieldMapper implements a CustomRangeDocValuesField for handling doc value support.

When executing a Range query over a Range field, the RangeQueryBuilder has been enhanced to accept a new relation parameter for defining the type of query as one of: WITHIN, CONTAINS, INTERSECTS. This provides support for finding all ranges that are related to a specific range in a desired way. As with other spatial queries, DISJOINT can be achieved as a MUST_NOT of an INTERSECTS query.
2016-11-29 10:10:14 -06:00
Isabel Drost-Fromm 3e0aedd7da Merge branch 'master' into feature/rank-eval 2016-11-29 11:09:22 +01:00
Simon Willnauer f5ff69fabe Remove connectToNodeLight and replace it with a connection profile ()
The Transport#connectToNodeLight concepts is confusing and not very flexible.
neither really testable on a unittest level. This commit cleans up the code used
to connect to nodes and simplifies transport implementations to share more code.
This also allows to connect to nodes with custom profiles if needed, for instance
future improvements can be added to connect to/from nodes that are non-data nodes without
dedicated bulks and recovery connections.
2016-11-29 09:35:07 +01:00
Jason Tedor a6082eb563 Grant Netty permission to read system somaxconn
When Netty listens on a socket, it specifies the established connection
backlog for the socket. On Linux, Netty tries to read the system-wide
configuration for this from /proc/sys/net/core/somaxconn and falls back
to a default value when it can not read this value. This commit grants
Netty permission to read this file so that it can honor the system-wide
configuration for the connection backlog for sockets that it is
listening on. This also removes an obnoxious stack trace that appears
when Netty logging is set to debug logging.

Relates 
2016-11-28 18:47:32 -05:00
Luca Cavanna 360b74eda8 [TEST] Don't reinitialize YamlTestClient and RestClient before each single test ()
In the past we ran yaml tests against an internal cluster, which would get restarted after each test failure, hence the client objects needed to eventually be refreshed before each test. That is why we had the initClient method to re-initialize the YamlTestClient in the execution context. We ended up though re-initializing the client unconditionally, which is not needed.

Also, ESRestTestCase recreates the RestClient against the external cluster before each test, which is not needed given that nothing changes in the external cluster.

This commit removes the initClient method from the yaml tests execution context. The YamlTestClient can be eagerly created before the first yaml test runs and then re-used in subsequent tests. Also api calls to check for nodes versions etc. are moved out of YamlTestClient to ESClientYamlSuiteTestCase. Also the RestClient is now initialized in ESRestTestCase before the first test runs, and kept around afterwards as a static member.

Basically each subclass of EsRestTestCase will have its own RestClient instance, but the client will be shared across the different tests within the same class. The yaml test suite is just a special suite, composed of 600+ tests that are loaded from files, which will share the same client instance.

This change should speed tests up as well, as we don't recreate the RestClient before each single test, and we don't call _cat/nodes either before each single test.
2016-11-28 18:43:27 +01:00
Isabel Drost-Fromm 2b24091361 Merge branch 'master' into feature/rank-eval 2016-11-28 11:45:29 +01:00
Jason Tedor 6f95261632 Remove unused imports from Netty4Utils
This commit removes two unused imports from Netty4Utils that were
leftover from a previous change.
2016-11-27 13:18:50 -05:00
Jason Tedor 5e73282bbc Simplify handling of fatal network layer errors
This commit simplifies the handling of fatal errors on the network
layer. The simplification here is to remove the use of a
StringWriter/PrintWriter pair to format the stack trace, removing the
need for the method to declare that it throws a checked IOException.
2016-11-27 13:14:24 -05:00
Tanguy Leroux 28dc02f01a [Test] Mute EqualsTests..testBranch(Not)EqualsDefAndPrimitive
It fails regurlarly and it is tracked by https://github.com/elastic/elasticsearch/issues/21801
2016-11-25 17:21:59 +01:00
Christoph Büscher 8ba771e913 Rank Eval: Handle exceptions on search requests and add them to response
Currently we fail the whole ranking evaluation request when we receive an
exception for any of the search requests. We should collect those errors and
report them back to the user in the rest response. This change adds collecting
the errors and propagating them back via the RankEvalResponse.

Closes 
2016-11-24 13:52:10 +01:00
Christoph Büscher e1fe0dc462 Merge branch 'master' into feature/rank-eval 2016-11-24 08:57:26 +01:00
Ryan Ernst c3ec8e22b8 Wrap VerifyError in ScriptException ()
If a bug occurs in painless compilation (not from a user, but from the
painless infrastructure), a VerifyError may be thrown when compiling the
broken generated class. This commit wraps VerifyErrors in
ScriptException so that useful information is returned to the user,
which can be passed on to the ES team for analysis.
2016-11-23 14:45:21 -08:00
Jack Conradson ba2d772668 Fix a VerifyError bug in Painless ()
This bug would cause a VerifyError when scripts using the === operator
were comparing a def type against a primitive type since the primitive
type wasn't being appropriately boxed.
2016-11-23 13:57:14 -08:00
Jason Tedor 8416b16dfd Improve handling of unreleased versions
Today when handling unreleased versions for backwards compatilibity
support, we scatted version constants across the code base and add some
asserts to support removing these constants when the version in question
is actually released. This commit improves this situation, enabling us
to just add a single unreleased version constant that can be renamed
when the version is actually released. This should make maintenance of
these versions simpler.

Relates 
2016-11-23 15:49:05 -05:00
Nik Everett 434fa4bd26 Docs and tests for painless lack of boxing for ?: and ?. ()
NOTE: The result of `?.` and `?:` can't be assigned to primitives. So
`int[] someArray = null; int l = someArray?.length` and
`int s = params.size ?: 100` don't work. Do
`def someArray = null; def l = someArray?.length` and
`def s = params.size ?: 100` instead.

Relates to 
2016-11-23 14:33:32 -05:00
Ryan Ernst 6940b2b8c7 Remove groovy scripting language ()
* Scripting: Remove groovy scripting language

Groovy was deprecated in 5.0. This change removes it, along with the
legacy default language infrastructure in scripting.
2016-11-22 19:24:12 -08:00
Nik Everett dbdcf9e95c Move painless yaml tests into painless dir
They were in a directory named "plan_a", the old name for painless.
2016-11-22 20:27:14 -05:00
Nik Everett 457c2d8fb0 Add Debug.explain to painless
You can use `Debug.explain(someObject)` in painless to throw an
`Error` that can't be caught by painless code and contains an
object's class. This is useful because painless's sandbox doesn't
allow you to call `someObject.getClass()`.

Closes 
2016-11-22 12:46:02 -05:00
Christoph Büscher 1d3e58ab9f Merge branch 'master' into feature/rank-eval 2016-11-22 11:31:46 +01:00
Jason Tedor 446037ccb8 Die with dignity on the network layer
When a fatal error is thrown on the network layer, such an error never
makes its way to the uncaught exception handler. This prevents the node
from being torn down if an out of memory error or other fatal error is
thrown while handling HTTP or transport traffic. This commit adds logic
to ensure that such errors bubble their way up to the uncaught exception
handler, even though Netty tries really hard to swallow everything.

Relates 
2016-11-21 22:14:30 -05:00
Nik Everett f5c8c746e6 Implement toString in painless's AST
This should make debugging painless' analysis and code generation a
little easier.

The `toString` implementations mirror the AST somewhat, and look like
`(SSource (SReturn (ENumeric 1)))`.
2016-11-21 16:24:10 -05:00
Simon Willnauer cb5c25ab4f Add a StreamInput#readArraySize method that ensures sane array sizes ()
Today we read a vint from the stream to allocate the size of an array up-front
before we start reading the values. This can be dangerous if for instance we read
from a corrupted stream or if some manipulated bytes are send for instance from
an attacker or a fuzzer. In most of the cases we can apply some best effort and
validate the array size to be _sane_ by ensuring we can at read at least N bytes
where N is the expected size of the array.
2016-11-21 21:39:21 +01:00
Jason Tedor 655c4fe172 Wrap GroovyBugErrors in ScriptExceptions
When Groovy detects a bug in its runtime because an internal assertion
was violated, it throws an GroovyBugError. This descends from
AssertionError and if it goes uncaught will land in the uncaught
exception handler and will not deliver any useful information to the
user. This commit wraps GroovyBugErrors in ScriptExceptions so that
useful information is returned to the user.
2016-11-19 07:11:13 -05:00
Nik Everett ae468441dc Implement the ?: operator in painless ()
Implements a null coalescing operator in painless that looks like `?:`. This form was chosen to emulate Groovy's `?:` operator. It is different in that it only coalesces null values, instead of Groovy's `?:` operator which coalesces all falsy values. I believe that makes it the same as Kotlin's `?:` operator. In other languages this operator looks like `??` (C#) and `COALESCE` (SQL) and `:-` (bash).

This operator is lazy, meaning the right hand side is only evaluated at all if the left hand side is null.
2016-11-18 13:54:26 -05:00
Jack Conradson ced433e9a8 Fix reserved variable availability in lambdas in Painless 2016-11-17 13:39:08 -08:00
Christoph Büscher 1964d45453 Merge branch 'master' into feature/rank-eval 2016-11-17 17:37:39 +01:00
Jason Tedor b08a2e1f31 Expose executor service interface from thread pool
This commit exposes the executor service interface from thread
pool. This will enable some high-level concurrency primitives that will
make some code cleaner and simpler.

Relates 
2016-11-17 09:18:49 -05:00
Christoph Büscher 9c58578dc6 Renaming RankEvalRequestTests to RankEvalRequestIT 2016-11-17 15:18:08 +01:00
Simon Willnauer de04aad994 Remove `modules/transport_netty_3` in favor of `netty_4` ()
We kept `netty_3` as a fallback in the 5.x series but now that master
is 6.0 we don't need this or in other words all issues coming up with
netty 4 will be blockers for 6.0.
2016-11-17 12:44:42 +01:00
Jason Tedor d06a8903fd Merge branch 'master' into feature/seq_no
* master: (22 commits)
  Add proper toString() method to UpdateTask ()
  Fix `InternalEngine#isThrottled` to not always return `false`. ()
  add `ignore_missing` option to SplitProcessor ()
  fix trace_match behavior for when there is only one grok pattern ()
  Remove dead code from GetResponse.java
  Fixes date range query using epoch with timezone ()
  Do not cache term queries. ()
  Updated dynamic mapper section
  Docs: Clarify date_histogram bucket sizes for DST time zones
  Handle release of 5.0.1
  Fix skip reason for stats API parameters test
  Reduce skip version for stats API parameter tests
  Strict level parsing for indices stats
  Remove cluster update task when task times out ()
  [DOCS] Mention "all-fields" mode doesn't search across nested documents
  InternalTestCluster: when restarting a node we should validate the cluster is formed via the node we just restarted
  Fixed bad asciidoc in boolean mapping docs
  Fixed bad asciidoc ID in node stats
  Be strict when parsing values searching for booleans ()
  Fix time zone rounding edge case for DST overlaps
  ...
2016-11-16 09:10:35 -05:00
Tal Levy 6796464f16 add `ignore_missing` option to SplitProcessor ()
Closes .
2016-11-16 15:46:09 +02:00
Tal Levy 04b712bdc5 fix trace_match behavior for when there is only one grok pattern ()
There is an issue in the Grok Processor, where trace_match: true does not inject the _ingest._grok_match_index into the ingest-document when there is just one pattern provided. This is due to an optimization in the regex construction. This commit adds a check for when this is the case, and injects a static index value of "0", since there is only one pattern matched (at the first index into the patterns).

To make this clearer, more documentation was added to the grok-processor docs.

Fixes .
2016-11-16 15:41:54 +02:00
Christoph Büscher 6c6ddf0245 Merge branch 'master' into feature/rank-eval 2016-11-16 11:57:22 +01:00
Christoph Büscher 6cc7fb5600 Fix potential NPE in RankEvalSpec roundtrip test 2016-11-15 20:53:51 +01:00
Boaz Leskes 2c0338fa87 Merge remote-tracking branch 'upstream/master' into feature/seq_no 2016-11-15 17:09:08 +00:00
Christoph Büscher 3d4804f9ed Fix test after merging in master branch 2016-11-15 15:55:09 +01:00
Adrien Grand df4482fdc8 Do not cache the QueryShardContext in PercolatorFieldMapper: it is cheap to create. 2016-11-15 15:45:18 +01:00
Christoph Büscher d6738fa650 Merge branch 'master' into feature/rank-eval
Conflicts:
	core/src/main/java/org/elasticsearch/script/Script.java
2016-11-15 14:56:36 +01:00
Adrien Grand 54809065a6 Make PercolatorFieldMapper get a QueryShardContext lazily. 2016-11-15 12:02:40 +01:00
Boaz Leskes c9f49039d3 Merge remote-tracking branch 'upstream/master' into feature/seq_no 2016-11-15 10:14:47 +00:00
Ryan Ernst d14c470b89 Remove generics from ActionRequest
closes 
2016-11-14 15:32:01 -08:00
Christoph Büscher 57ea1abb55 Fixing compile error in SmokeTestRankEvalWithMustacheYAMLTestSuiteIT 2016-11-14 13:42:14 +01:00
Adrien Grand 1fd5c47e7f Upgrade to lucene-6.3.0. () 2016-11-14 09:36:45 +01:00
Jason Tedor c7a1b3eb50 Merge branch 'master' into feature/seq_no
* master:
  Hack around cluster service and logging race
  Do not prematurely shutdown Log4j
  Support decimal constants with trailing [dD] in painless ()
  In painless suggest a long constant if int won't do ()
  Account for different paths for sysctl utilities
  [TEST] testRebalancePossible() may not have an assigned node id
  Tests: Disable merge in SearchCancellationTests
  Tests: clean search scroll at the end of SearchCancellationIT
2016-11-13 20:01:44 -05:00
Nik Everett 2a328034ef Support decimal constants with trailing [dD] in painless ()
This adds support to painless for decimal constants with trailing `d` or
`D` to make it compatible with Java. It already supported integer
constants with a trailing `d` or `D` but this adds tests for it.

Closes 
2016-11-12 11:08:39 -05:00
Nik Everett a26b5a113c In painless suggest a long constant if int won't do ()
In painless we prefer explicit types over implicit ones whereas
groovy is the other way around. Take this groovy code:

```
> 86400000.class
java.lang.Integer
> 864000000000.class
java.lang.Long
```

Painless accepts `86400000` just fine because that is a valid `int`
in the jvm. It rejects `864000000000` as an invlid `int` constant
because, in painless as in java, `long` constants always end in `L`
or `l`.

To ease the transition from groovy to painless, this changes the
compilation error returned from these invalid constants from:

```
Invalid int constant [864000000000].
```

to

```
Invalid int constant [864000000000]. If you want a long constant then change it to [864000000000L].
```

Inspired by 
2016-11-12 11:08:18 -05:00
Jason Tedor d3417fb022 Merge branch 'master' into feature/seq_no
* master: (516 commits)
  Avoid angering Log4j in TransportNodesActionTests
  Add trace logging when aquiring and releasing operation locks for replication requests
  Fix handler name on message not fully read
  Remove accidental import.
  Improve log message in TransportNodesAction
  Clean up of Script.
  Update Joda Time to version 2.9.5 ()
  Remove unused ClusterService dependency from SearchPhaseController ()
  Remove max_local_storage_nodes from elasticsearch.yml ()
  Wait for all reindex subtasks before rethrottling
  Correcting a typo-Maan to Man-in README.textile ()
  Fix InternalSearchHit#hasSource to return the proper boolean value ()
  Replace all index date-math examples with the URI encoded form
  Fix typos ()
  Adapt ES_JVM_OPTIONS packaging test to ubuntu-1204
  Add null check in InternalSearchHit#sourceRef to prevent NPE ()
  Add VirtualBox version check ()
  Export ES_JVM_OPTIONS for SysV init
  Skip reindex rethrottle tests with workers
  Make forbidden APIs be quieter about classpath warnings ()
  ...
2016-11-10 23:40:33 -05:00
Jack Conradson aeb97ff412 Clean up of Script.
Closes 
2016-11-10 09:59:13 -08:00
Nik Everett 4db21db0aa Wait for all reindex subtasks before rethrottling
In the test for reindex and friend's rethrottling feature we were waiting
only for a single reindex sub task to start before rethrottling. This
mostly worked because starting tasks is fast. But it didn't *always work
and CI found that for us. This fixes the test to wait for all subtasks
to start before rethrottling.

I reproduced this locally semi-consistently with some fairly creative
`Thread.sleep` calls and this test fix fixes the issue even with the
sleeps so I'm fairly sure this will work consistently.

Closes 
2016-11-10 10:49:25 -05:00
Luca Cavanna bd23921a3a Fix InternalSearchHit#hasSource to return the proper boolean value ()
The method used to be called `isSourceEmpty`, and was renamed to `hasSource`, but the return value never changed. Updated tests and users accordingly.

Closes 
2016-11-10 13:13:38 +01:00
Nik Everett b0f5ea3f59 Skip reindex rethrottle tests with workers
They are flakey and spuriously fail the build. I'll hunt down
the cause soon and reenabled but for now they should stop.

Relates 
2016-11-09 17:50:09 -05:00
Christoph Büscher 4e5c868709 RankEval: Add optional parameter to ignore unlabeled documents in Precision metric
Our current default behaviour to ignore unrated documents when calculating the
precision seems a bit counter intuitive. Instead we should treat those documents
as "irrelevant" by default and provide an optional parameter to ignore those
documents if that is the behaviour the user wants.
2016-11-09 15:53:16 +01:00
Nik Everett d03b8e4abb Implement reading from null safe dereferences
Null safe dereferences make handling null or missing values shorter.
Compare without:
```
if (ctx._source.missing != null && ctx._source.missing.foo != null) {
  ctx._source.foo_length = ctx.source.missing.foo.length()
}
```

To with:
```
Integer length = ctx._source.missing?.foo?.length();
if (length != null) {
  ctx._source.foo_length = length
}
```

Combining this with the as of yet unimplemented elvis operator allows
for very concise defaults for nulls:
```
ctx._source.foo_length = ctx._source.missing?.foo?.length() ?: 0;
```

Since you have to start somewhere, we started with null safe dereferenes.

Anyway, this is a feature borrowed from groovy. Groovy allows writing to
null values like:
```
def v = null
v?.field = 'cat'
```
And the writes are simply ignored. Painless doesn't support this at this
point because it'd be complex to implement and maybe not all that useful.

There is no runtime cost for this feature if it is not used. When it is
used we implement it fairly efficiently, adding a jump rather than a
temporary variable.

This should also work fairly well with doc values.
2016-11-09 07:20:11 -05:00
Nik Everett a3bd6d1ad9 Switch reindex with slices error to IAE
If you try to reindex with multiple slices against a node that
doesn't support it we throw an `IllegalArgumentException` so
`assertVersionSerializable` is ok with it and so if this happens
in REST it comes back as a 400 error.
2016-11-08 11:42:07 -05:00
Luca Cavanna 293a3cab01 Rest client: don't reuse that same HttpAsyncResponseConsumer across multiple retries ()
* Rest client: don't reuse that same HttpAsyncResponseConsumer across multiple retries

Turns out that AbstractAsyncResponseConsumer from apache async http client is stateful and cannot be reused across multiple requests. The failover mechanism was mistakenly reusing that same instance, which can be provided by users, across retries in case nodes are down or return 5xx errors. The downside is that we have to change the signature of two public methods, as HttpAsyncResponseConsumer cannot be provided directly anymore, rather its factory needs to be provided which is going to be used to create one instance of the consumer per request attempt.

Up until now we tested our RestClient against multiple nodes only in a mock environment, where we don't really send http requests. In that scenario we can verify that retries etc. work properly but the interaction with the http client library in a real scenario is different and can catch other problems. With this commit we also add an integration test that sends requests to multiple hosts, and some of them may also get stopped meanwhile. The specific test for pathPrefix was also removed as pathPrefix is now randomly applied by default, hence implicitly tested. Moved also a small test method that checked the validity of the path argument to the unit test RestClientSingleHostTests.

Also increase default buffer limit to 100MB and make it required in default consumer

The default buffer limit used to be 10MB but that proved not to be high enough for scroll requests (see reindex from remote). With this commit we increase the limit to 100MB and make it a bit more visibile in the consumer factory.
2016-11-08 16:42:42 +01:00
Christoph Büscher 4718f000df Remove test that uses max_acceptable_rank parameter 2016-11-08 15:51:37 +01:00
Christoph Büscher adb24333ca Remove maxAcceptableRank parameter from ReciprocalRank 2016-11-08 12:35:06 +01:00
Christoph Büscher c8d9d063ca Removing the 'size' parameter from the dcg metric 2016-11-08 12:35:06 +01:00
Christoph Büscher 3855d7f721 Removing the 'size' parameter from precision metric 2016-11-08 12:35:06 +01:00
Christoph Büscher d013a2c8f1 Adapting to change in ESClientYamlSuiteTestCase api 2016-11-08 11:30:36 +01:00
Christoph Büscher dbef96c69e Merge branch 'master' of github.com:elastic/elasticsearch into feature/rank-eval 2016-11-08 11:10:10 +01:00
Ryan Ernst 7a2c984bcc Test: Remove multi process support from rest test runner ()
At one point in the past when moving out the rest tests from core to
their own subproject, we had multiple test classes which evenly split up
the tests to run. However, we simplified this and went back to a single
test runner to have better reproduceability in tests. This change
removes the remnants of that multiplexing support.
2016-11-07 15:07:34 -08:00
Jason Tedor 23a271f092 Address race condition in HTTP pipeline tests
This commit adapts a previous fix to the HTTP pipeline tests for Netty 4
to Netty 3.

Relates 
2016-11-07 13:20:22 -05:00
Christoph Büscher 1702304ecd Merge branch 'master' into feature/rank-eval 2016-11-07 11:02:22 +01:00
Nik Everett a13a050271 Add automatic parallelization support to reindex and friends ()
Adds support for `?slices=N` to reindex which automatically
parallelizes the process using parallel scrolls on `_uid`. Performance
testing sees a 3x performance improvement for simple docs
on decent hardware, maybe 30% performance improvement
for more complex docs. Still compelling, especially because
clusters should be able to get closer to the 3x than the 30%
number.

Closes 
2016-11-04 20:59:15 -04:00
Christoph Büscher 619e4c1a44 Merge branch 'master' into feature/rank-eval 2016-11-04 12:48:42 +01:00
Adrien Grand 2a70f6e7b1 Upgrade to lucene-6.3.0-snapshot-a66a445. ()
This addresses a bug that was introduced with https://issues.apache.org/jira/browse/LUCENE-7501.
2016-11-04 10:34:04 +01:00
Christoph Büscher 2dad72e68c Rank Eval: Handle precion@ edge case
There's a currently unhandled edge case for the precion@ metric. When none of
the search hits in the result are rated, we have neither true nor false
positives which currently leads to division by zero. We should return a precion
of 0.0 in this case.
2016-11-03 12:59:36 +01:00
Nik Everett 24d5f31a54 Make painless's assertion about out of bound less brittle
Instead of asserting that the message is shaped a certain way we
cause the exception and catch it and assert that the messages are
the same. This is the way to go because the exception message from
the jvm is both local and jvm dependent.

This is the CI failure that found this:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+5.x+java9-periodic/515/consoleFull
2016-11-02 12:38:51 -04:00
Christoph Büscher b3370de715 Tests: Add warning header checks to QueryBuilder tests and QueryParseContextTests
This adds checks for expected warning headers to the query builder test
infrastructure. Tests that are adding deprecation warnings to the response
headers need to check those, otherwise the abstract base class for the test
class will complain at teardown.
2016-11-02 15:45:33 +01:00
Adrien Grand aa6cd93e0f Require arguments for QueryShardContext creation. ()
The `IndexService#newQueryShardContext()` method creates a QueryShardContext on
shard `0`, with a `null` reader and that uses `System.currentTimeMillis()` to
resolve `now`. This may hide bugs, since the shard id is sometimes used for
query parsing (it is used to salt random score generation in `function_score`),
passing a `null` reader disables query rewriting and for some use-cases, it is
simply not ok to rely on the current timestamp (eg. percolation). So this pull
request removes this method and instead requires that all call sites provide
these parameters explicitly.
2016-11-02 09:48:49 +01:00
Nik Everett a612e5988e Bump reindex-from-remote's buffer to 200mb
It was 10mb and that was causing trouble when folks reindex-from-remoted
with large documents.

We also improve the error reporting so it tells folks to use a smaller
batch size if they hit a buffer size exception. Finally, adds some docs
to reindex-from-remote mentioning the buffer and giving an example of
lowering the size.

Closes 
2016-11-01 13:19:28 -04:00
Christoph Büscher 25565b9baa RankEval: Check for duplicate keys in rated documents
When multiple ratings for the same document (identified by _index, _type,
_id) are specified in the request we should throw an error. This change adds a
check for this in the RatedRequest setter (and ctor that uses that setter).

Closes 
2016-11-01 14:54:05 +01:00
Christoph Büscher 51102ee91c Fixing compile issue with ScriptType after merge with master 2016-11-01 14:42:24 +01:00
Isabel Drost-Fromm 0b8a2e40cb First step towards supporting templating in rank eval requests. ()
This adds support for templating in rank eval requests.

Relates to 

Problem: In it's current state the rank-eval request API forces the user to repeat complete queries for each test request. In most use cases the structure of the query to test will be stable with only parameters changing across requests, so this looks like lots of boilerplate json for something that could be expressed in a more concise way.

Uses templating/ ScriptServices to enable users to submit only one test request template and let them only specify template parameters on a per test request basis.
2016-11-01 11:36:22 +01:00
Jason Tedor 38663351dc Fix logger names for Netty
Previously Elasticsearch would only use the package name for logging
levels, truncating the package prefix and the class name. This meant
that logger names for Netty were just prefixed by netty3 and netty. We
changed this for Elasticsearch so that it's the fully-qualified class
name now, but never corrected this for Netty. This commit fixes the
logger names for the Netty modules so that their levels are controlled
by the fully-qualified class name.

Relates 
2016-10-31 17:23:21 -04:00
Jack Conradson 185dff7346 Cleanup ScriptType ()
Refactored ScriptType to clean up some of the variable and method names. Added more documentation. Deprecated the 'in' ParseField in favor of 'stored' to match the indexed scripts being replaced by stored scripts.
2016-10-31 13:48:51 -07:00
Nik Everett 1bbd3c5400 Fix painless's out of bounds assertions in java 9
Java 9's exception message when lists have an out of bounds index
is much better than java 8 but the painless code asserted on the
java 8 message. Now it'll accept either.

I'm tempted to weaken the assertion but I like asserting that the
message is readable.
2016-10-29 22:21:57 -04:00
Nik Everett 3a7a218e8f Support negative array ofsets in painless
Adds support for indexing into lists and arrays with negative
indexes meaning "counting from the back". So for if
`x = ["cat", "dog", "chicken"]` then `x[-1] == "chicken"`.

This adds an extra branch to every array and list access but
some performance testing makes it look like the branch predictor
successfully predicts the branch every time so there isn't a
in execution time for this feature when the index is positive.
When the index is negative performance testing showed the runtime
is the same as writing `x[x.length - 1]`, again, presumably thanks
to the branch predictor.

Those performance metrics were calculated for lists and arrays but
`def`s get roughly the same treatment though instead of inlining
the test they need to make a invoke dynamic so we don't screw up
maps.

Closes 
2016-10-29 16:12:40 -04:00
Adrien Grand b3cc54cf0d Upgrade to lucene-6.3.0-snapshot-ed102d6 ()
Lucene 6.3 is expected to be released in the next weeks so it'd be good to give
it some integration testing. I had to upgrade randomized-testing too so that
both Lucene and Elasticsearch are on the same version.
2016-10-28 14:47:15 +02:00
Christoph Büscher 51a2e3bf1e Merge branch 'master' into feature/rank-eval 2016-10-27 11:11:37 +02:00
Jack Conradson 512a77a633 Refactor ScriptType to be a top-level class. 2016-10-26 10:21:22 -07:00
Jason Tedor 9c3e4d6e22 Add correct Content-Length on HEAD requests
This commit fixes responses to HEAD requests so that the value of the
Content-Length is correct per the HTTP spec. Namely, the value of this
header should be equal to the Content-Length if the request were not a
HEAD request.

This commit also fixes a memory leak on HEAD requests to the main action
that arose from the bytes on a builder not being released due to them
being dropped on the floor to ensure that the response to the main
action did not have a body.

Relates 
2016-10-25 23:08:19 -04:00
Christoph Büscher 67e2de6702 Merge branch 'master' into feature/rank-eval 2016-10-25 11:05:31 +02:00
Nik Everett 18393a06f3 Fix reindex-from-remote for parent/child from <2.0
Versions before 2.0 needed to be told to return interesting fields
like `_parent`, `_routing`, `_ttl`, and `_timestamp`. And they come
back inside a `fields` block which we need to parse.

Closes 
2016-10-21 13:14:33 -04:00
Jason Tedor f51bf8ee47 Upgrade to Netty 4.1.6
This commit upgrades the transport-netty4 module dependency from Netty
version 4.1.5 to version 4.1.6. This is a bug fix release of Netty.

Relates 
2016-10-20 20:13:29 -04:00
Jack Conradson ceaae47d38 Remove more equivalents of the now method from the Painless whitelist. 2016-10-20 10:35:26 -07:00
Nik Everett b5da42905f Remove publishAddress from reindex whitelist
Removes the `publishAddress` parameter from the reindex-from-remote
whitelist checking because it isn't in use after .
2016-10-20 12:51:10 -04:00
Fanfan 043a45746c some misspelled words in code ()
as the title mentioned, misspelling as follows, "construct" to "constrcut", "cumulation" to "cumalation", "initialize" to "intialize".
2016-10-19 11:42:38 -04:00
Christoph Büscher e8e65c3a1e Merge branch 'master' into feature/rank-eval 2016-10-19 11:37:29 +02:00
Nik Everett acf7c7430b Add "simple match" support for reindex-from-remote whitelist
This allows you to whitelist `localhost:*` or `127.0.10.*:9200`.
It explicitly checks for patterns like `*` in the whitelist and
refuses to start if the whitelist would match everything. Beyond
that the user is on their own designing a secure whitelist.
2016-10-18 21:47:21 -04:00
Tal Levy 38c650f376 make painless the default scripting language for ScriptProcessor ()
- fixes a bug in the docs that mentions `lang` as optional
- now `lang` defaults to "painless"
2016-10-18 16:22:01 -07:00
Ryan Ernst dca614aa3b Build: Change `gradle run` to use zip distribution ()
When running `gradle run`, a developer usually intends to get a running
instance as if they had run elasticsearch from the command line. This is
different than the isolated environment we use for integration testing
plugins. This change switches the run task to use the zip distribution,
so that all modules included in the normal distribution are included.
2016-10-18 11:48:58 -07:00
Ryan Ernst 3d3dd7185d Add support for booleans in scripts ()
* Scripting: Add support for booleans in scripts

Since 2.0, booleans have been represented as numeric fields (longs).
However, in scripts, this is odd, since you expect doing a comparison
against a boolean to work. While languages like groovy will auto convert
between booleans and longs, painless does not.

This changes the doc values accessor for boolean fields in scripts to
return Boolean objects instead of Long objects.

closes 

* Make Booleans final and remove wrapping of `this` for getValues()
2016-10-17 11:11:42 -07:00
Christoph Büscher f927a235b3 Merge branch 'master' into feature/rank-eval 2016-10-17 14:50:10 +02:00
Jason Tedor c1bdaaf80f Fix connection keep-alive header handling
This commit fixes an issue with the handling of the value "keep-alive"
on the Connection header in the Netty 4 HTTP implementation while
handling an HTTP 1.0 request. The issue was using the wrong equals
method to compare an AsciiString instance and a String instance (they
could never be equal). This commit fixes this to use the correct equals
method to compare for content equality.
2016-10-16 19:51:00 -04:00
Jason Tedor cd5777593a Fix connection close header handling
This commit fixes an issue with the handling of the value "close" on the
Connection header in the Netty 4 HTTP implementation. The issue was
using the wrong equals method to compare an AsciiString instance and a
String instance (they could never be equal). This commit fixes this to
use the correct equals method to compare for content equality.

Relates 
2016-10-16 13:18:09 -04:00
Christoph Büscher dfc6d1f369 Remove unknown docs from EvalQueryQuality
The unknown document section in the response for each query can be rendered
using the rated hits that are now also part of the response by just filtering
the documents without a rating.
2016-10-14 17:14:05 +02:00
Christoph Büscher 9e394b0644 Pull common operations into RankedListQualityMetric interface
Currently each implementation of RankedListQualityMetric does some initial
joining operation that links the input search hits with a rated document rating,
if available. Also all metrics collect unknown docs and now also need to add the
list of rated search hits to the partial query evaluation. This change
centralizes this work in some new helper methods in RankedListQualityMetric.
2016-10-14 17:14:05 +02:00
Christoph Büscher ebe13100df Add `hits` section to response for each ranking evaluation query
This change adds a `hits` section to the response part for each ranking
evaluation query, containing a list of documents (index/type/id) and ratings (if
the document was rated in the request). This section can be used to better
understand the calculation of the ranking quality of this particular query, but
it can also be used to identify the "unknown" (that is unrated) documents that
were part of the seach hits, for example because a UI later wants to present
those documents to the user to get a rating for them.

If the user specifies a set of field names using a parameter called
`summary_fields` in the request, those fields are also included as part of the
response in addition to "_index", "_type", "_id".
2016-10-14 17:14:05 +02:00
Christoph Büscher cd9d07b91b Merge branch 'master' into feature/rank-eval 2016-10-14 17:03:30 +02:00
Jason Tedor 595ec8c948 Remove artificial default processors limit
Today Elasticsearch limits the number of processors used in computing
thread counts to 32. This was from a time when Elasticsearch created
more threads than it does now and users would run into out of memory
errors. It appears the real cause of these out of memory errors was not
well understood (it's often due to ulimit settings) and so users were
left hitting these out of memory errors on boxes with high core
counts. Today Elasticsearch creates less threads (but still a lot) and
we have a bootstrap check in place to ensure that the relevant ulimit is
not too low.

There are some caveats still to having too many concurrent indexing
threads as it can lead to too many little segments, and it's not a
magical go faster knob if indexing is already bottlenecked by disk, but
this limitation is artificial and surprising to users and so it should
be removed.

This commit also increases the lower bound of the max processes ulimit,
to prepare for a world where Elasticsearch instances might be running
with more the previous cap of 32 processors. With the current settings,
Elasticsearch wants to create roughly 576 + 25 * p / 2 threads, where p
is the number of processors. Add in roughly 7 * p / 8 threads for the GC
threads and a fudge factor, and 4096 should cover us pretty well up to
256 cores.

Relates 
2016-10-14 05:47:26 -04:00
Tanguy Leroux e71c30c71d Mustache: Add {{#url}}{{/url}} function to URL encode strings ()
This commit adds a new Mustache function (codename: url) and a new URLEncoder that can be used to URL encode strings in mustache templates.
2016-10-13 16:17:28 +02:00
Simon Willnauer 12392b5425 Ensure port range is readable in the exception message ()
Both netty3 and netty4 http implementation printed the default
toString representation of PortRange if ports couldn't be bound.
This commit adds a better default toString method to PortRange and
uses the string representation for the error message in the http
implementations.
2016-10-12 22:33:47 +02:00
Areek Zillur 133be6631d Merge branch 'master' into cleanup/transport_bulk 2016-10-12 13:09:29 -04:00
Isabel Drost-Fromm 20c1e25609 Merge branch 'master' into feature/rank-eval 2016-10-12 14:41:17 +02:00
Tanguy Leroux 44ac5d057a Remove empty javadoc ()
This commit removes as many as empty javadocs comments my regexp has found
2016-10-12 10:27:09 +02:00
Areek Zillur 481f7909ae Merge branch 'master' into cleanup/transport_bulk 2016-10-11 16:04:47 -04:00
Areek Zillur 0e8b6532ec rename DocumentRequest to DocWriteRequest 2016-10-11 16:00:10 -04:00
Tanguy Leroux e4c7d8183e XContentBuilder: Avoid building self-referencing objects ()
Some objects like maps, iterables or arrays of objects can self-reference themselves. This is mostly due to a bug in code but the XContentBuilder should be able to detect such situations and throws an IllegalArgumentException instead of building objects over and over until a stackoverflow occurs.

closes 
closes 
2016-10-11 11:41:54 +02:00
Simon Willnauer 37ca38df3d Expose `ctx._now` in update scripts ()
Update scripts might want to update the documents `_timestamp` but need a notion of `now()`.
Painless doesn't support any notion of now() since it would make scripts non-pure functions. Yet,
in the update case this is a valid value and we can pass it with the context together to allow the
script to record the timestamp the document was updated.

Relates to 
2016-10-10 21:14:14 +02:00
Jim Ferenczi c80a563a71 Replace org.elasticsearch.common.lucene.search.MatchNoDocsQuery with its Lucene version (org.apache.lucene.search.MatchNoDocsQuery) ()
* Replace org.elasticsearch.common.lucene.search.MatchNoDocsQuery with its Lucene version (org.apache.lucene.search.MatchNoDocsQuery)

This change removes the ES version of the match no docs query and replaces it with the Lucene version.

relates 

* Add missing change
2016-10-10 17:45:19 +02:00
Simon Willnauer 4fd1276542 Prevent AbstractArrays from release bytes more than once ()
Today we throw an assertion error if we release an AbstractArray more than once.
Yet, it's recommended to implement close methods such that they can be invoked
more than once. Guaranteed single release calls are hard to implement and some
situations might not be tested causing for instance `CircuitBreaker` to operate on
corrupted memory stats.
2016-10-10 17:30:37 +02:00
Christoph Büscher c3380863be Adapting RestRankEvalAction to changes in API on master 2016-10-10 12:57:18 +02:00
Christoph Büscher 0c25cfbd16 Merge branch 'master' into feature/rank-eval 2016-10-10 12:10:34 +02:00
Areek Zillur c747085b70 Merge branch 'master' into cleanup/transport_bulk 2016-10-07 17:52:53 -04:00
Areek Zillur 396f80c963 Revert "rename DocumentRequest to DocumentWriteRequest"
This reverts commit b5079ce009.
2016-10-07 17:50:07 -04:00
Nik Everett cf4038b668 DeGuice some of IndicesModule
UpdateHelper, MetaDataIndexUpgradeService, and some recovery
stuff.

Move ClusterSettings to nullable ctor parameter of TransportService
so it isn't forgotten.
2016-10-07 11:14:38 -04:00
Simon Willnauer 7452028e50 Simplify TransportAddress ()
since TransportAddress is now final we can simplify it's interface a bit
and remove methods that are only used in tests or are plain delegates.
2016-10-07 15:56:54 +02:00
Simon Willnauer 194a6b1df0 Remove LocalTransport in favor of MockTcpTransport ()
This change proposes the removal of all non-tcp transport implementations. The
mock transport can be used by default to run tests instead of local transport that has
roughly the same performance compared to TCP or at least not noticeably slower.

This is a master only change, deprecation notice in 5.x will be committed as a
separate change.
2016-10-07 11:27:47 +02:00
Simon Willnauer 9c9afe3f01 Remove SearchContext#current and all it's threadlocals ()
Today SearchContext expose the current context as a thread local which makes any kind of sane interface design very very hard. This PR removes the thread local entirely and instead passes the relevant context anywhere needed. This simplifies state management dramatically and will allow for a much leaner SearchContext interface down the road.
2016-10-06 19:51:54 +02:00
Areek Zillur b5079ce009 rename DocumentRequest to DocumentWriteRequest 2016-10-06 05:05:59 -04:00
Areek Zillur 9b691f0d93 Merge branch 'master' into cleanup/transport_bulk 2016-10-05 17:51:39 -04:00
Simon Willnauer ce21b607bb move test to a single node test 2016-10-05 21:55:50 +02:00
Simon Willnauer 50ca10378e Merge branch 'master' into dont_cache_scripts 2016-10-05 20:43:52 +02:00
Simon Willnauer 838c28eeb4 add percolate with script query test 2016-10-05 20:43:46 +02:00
Jack Conradson ba88d9af57 Remove all date 'now' methods from the Painless whitelist to ensure
Painless scripts are pure functions.
2016-10-05 09:47:20 -07:00
Simon Willnauer 57afbadf33 PercolateQuery is never cacheable 2016-10-05 16:38:47 +02:00
Colin Goodheart-Smithe 7bffe95025 Fix percolator queries to not be cacheable 2016-10-05 15:03:29 +01:00
Simon Willnauer 587bdcef38 add extra safety when accessing scripts or now and reqeusts are cached 2016-10-05 09:41:48 +02:00
Simon Willnauer 80bf44f709 Merge branch 'master' into dont_cache_scripts 2016-10-04 21:09:40 +02:00
Areek Zillur bd4a03a426 Merge branch 'master' into cleanup/transport_bulk 2016-10-04 14:06:17 -04:00
Jason Tedor 51d53791fe Remove lenient URL parameter parsing
Today when parsing a request, Elasticsearch silently ignores incorrect
(including parameters with typos) or unused parameters. This is bad as
it leads to requests having unintended behavior (e.g., if a user hits
the _analyze API and misspell the "tokenizer" then Elasticsearch will
just use the standard analyzer, completely against intentions).

This commit removes lenient URL parameter parsing. The strategy is
simple: when a request is handled and a parameter is touched, we mark it
as such. Before the request is actually executed, we check to ensure
that all parameters have been consumed. If there are remaining
parameters yet to be consumed, we fail the request with a list of the
unconsumed parameters. An exception has to be made for parameters that
format the response (as opposed to controlling the request); for this
case, handlers are able to provide a list of parameters that should be
excluded from tripping the unconsumed parameters check because those
parameters will be used in formatting the response.

Additionally, some inconsistencies between the parameters in the code
and in the docs are corrected.

Relates 
2016-10-04 12:45:29 -04:00
Simon Willnauer 94b7873b49 Add a #markAsNotCachable() method to context to mark requests as not cachable 2016-10-04 18:05:00 +02:00
Areek Zillur 248ac240ed Merge branch 'master' into cleanup/transport_bulk 2016-10-03 16:12:11 -04:00
Boaz Leskes 27eab74510 merge from master 2016-09-30 17:19:30 +02:00
Martijn van Groningen 55dce523c2 docs: marked `foreach` processor as experimental
Closes 
2016-09-30 12:23:42 +02:00
Tal Levy 33b9e2065b no null values in ingest configuration error messages ()
The invalid ingest configuration field name used to show itself,
even when it was null, in error messages. Sometimes this does not make
sense.

e.g.
```[null] Only one of [file], [id], or [inline] may be configure```
vs.
```Only one of [file], [id], or [inline] may be configure```

The above deals with three fields, therefore this no one property
responsible.
2016-09-29 11:34:52 +02:00
Tal Levy 1872d2ae46 [fix] JSON Processor was not properly added () 2016-09-29 03:04:43 +02:00
Jason Tedor 25fd9e26c4 Merge branch 'master' into feature/seq_no
* master: (1199 commits)
  [DOCS] Remove non-valid link to mapping migration document
  Revert "Default `include_in_all` for numeric-like types to false"
  test: add a test with ipv6 address
  docs: clearify that both ip4 and ip6 addresses are supported
  Include complex settings in settings requests
  Add production warning for pre-release builds
  Clean up confusing error message on unhandled endpoint
  [TEST] Increase logging level in testDelayShards()
  change health from string to enum ()
  Provide error message when plugin id is missing
  Document that sliced scroll works for reindex
  Make reindex-from-remote ignore unknown fields
  Remove NoopGatewayAllocator in favor of a more realistic mock ()
  Remove Marvel character reference from guide
  Fix documentation for setting Java I/O temp dir
  Update client benchmarks to log4j2
  Changes the API of GatewayAllocator#applyStartedShards and ()
  Removes FailedRerouteAllocation and StartedRerouteAllocation
  IndexRoutingTable.initializeEmpty shouldn't override supplied primary RecoverySource ()
  Smoke tester: Adjust to latest changes ()
  ...
2016-09-29 00:22:31 +02:00
Tal Levy 92ab44d35c [fix] JSON Processor was not properly added () 2016-09-28 23:04:22 +02:00
Christoph Büscher 92baf14c0a Merge branch 'master' into feature/rank-eval 2016-09-27 11:59:35 +02:00
Nik Everett 370afa371b Make reindex-from-remote ignore unknown fields
reindex-from-remote should ignore unknown fields so it is mostly
future compatible. This makes it ignore unknown fields by adding an
option to `ObjectParser` and `ConstructingObjectParser` that, if
enabled, causes them to ignore unknown fields.

Closes 
2016-09-26 00:55:46 +02:00
Simon Willnauer fe1803c957 Remove AnalysisService and reduce it to a simple name to analyzer mapping ()
Today we hold on to all possible tokenizers, tokenfilters etc. when we create
an index service on a node. This was mainly done to allow the `_analyze` API to
directly access all these primitive. We fixed this in  and can now get rid of
the AnalysisService entirely and replace it with a simple map like class. This
ensures we don't create a gazillion long living objects that are entirely useless since
they are never used in most of the indices. Also those objects might consume a considerable
amount of memory since they might load stopwords or synonyms etc.

Closes 
2016-09-23 08:53:50 +02:00
Christoph Büscher 1909877728 Merge branch 'master' into feature/rank-eval 2016-09-22 12:21:20 +02:00
Christoph Büscher 29402a28e0 RankEval: Adding details section to response ()
In order to understand how well particular queries in a joint ranking evaluation 
request work we want to break down the overall metric into its components, each
contributed by a particular query. The response structure now has a
`details` section under which we can summarize this information. Each
sub-section is keyed by the query-id and currently only contains the partial
metric and the unknown_docs section for each query.
2016-09-22 12:17:10 +02:00
Jack Conradson 5755dd256d Fix String concatentation bug. 2016-09-21 15:49:56 -07:00
Yannick Welsch bf5d425ab9 Fix wrong logger usages
These misusages were found by the logger usage checker that was re-enabled in the previous commit.
2016-09-21 14:45:28 +02:00
Christoph Büscher 6c1003b81a Normalize rated document key parameter
To be consitent with the output of the search API, we should use the same field
names for specifying the document ("_index", "_type", "_id") when providing the
rated documents in the `rank_eval` request.
2016-09-21 12:36:50 +02:00
Christoph Büscher 306ea1c613 Merge branch 'master' into feature/rank-eval 2016-09-21 10:58:36 +02:00
Tanguy Leroux 7645abaad9 Remove duplicate methods in ByteSizeValue ()
This commit removes `ByteSizeValue`'s methods that are duplicated (ex: `mbFrac()` and `getMbFrac()`) in order to only keep the `getN` form.
    
It also renames `mb()` -> `getMb()`, `kb()` -> `getKB()` in order to be more coherent with the `ByteSizeUnit` method names.
2016-09-20 14:07:23 +02:00
Christoph Büscher 8772725d46 Merge branch 'master' into feature/rank-eval 2016-09-20 11:23:24 +02:00
Ryan Ernst 85b8f29415 Build: Remove old maven deploy support ()
* Build: Remove old maven deploy support

This change removes the old maven deploy that we have in parallel to
maven-publish, and makes maven-publish fully work with publishing to
maven local. Using `gradle publishToMavenLocal` should be used to
publish to .m2.

Note that there is an unfortunate hack that means for
zip artifacts we must first create/publish a dummy pom file, and then
follow that with the real pom file. It would be nice to have the pom
file contains packaging=zip, but maven central then requires sources and
javadocs. But our zips are really just attached artifacts, so we already
set the packaging type to pom for our zip files. This change just works
around a limitation of the underlying maven publishing library which
silently skips attached artifacts when the packaging type is set to pom.

relates 
closes 

* Remove unnecessary extra spacing
2016-09-19 15:10:41 -07:00
Simon Willnauer ee8d14798f Unguice Transport and friends ()
This change removes all guice interaction from Transport, HttpServerTransport,
HttpServer and TransportService. All these classes as well as their subclasses
or extended version configured via plugins are now created by using plain old
bloody java constructors. YAY!
2016-09-19 22:10:47 +02:00
David Pilato ed4d0881b1 Add profile and explain parameters to template API
We can now run templates using `explain` and/or `profile` parameters.
Which is interesting when you have defined a complicated profile but want to debug it in an easier way than running the full query again.

You can use `explain` parameter when running a template:

```js
GET /_search/template
{
  "file": "my_template",
  "params": {
    "status": [ "pending", "published" ]
  },
  "explain": true
}
```

You can use `profile` parameter when running a template:

```js
GET /_search/template
{
  "file": "my_template",
  "params": {
    "status": [ "pending", "published" ]
  },
  "profile": true
}
```
2016-09-19 17:52:13 +02:00
Christoph Büscher 10d465f946 Merge pull request from cbuescher/rankEval-removeTopLevelId
RankEval: Remove top level `spec_id`
2016-09-19 12:34:14 +02:00
Christoph Büscher 8f88917c4e Merge branch 'master' into feature/rank-eval 2016-09-19 11:50:53 +02:00
Simon Willnauer f5daa165f1 Remove ability to plug-in TransportService ()
TransportService is such a central part of the core server, replacing
it's implementation is risky and can cause serious issues. This change removes the ability to
plug in TransportService but allows registering a TransportInterceptor that enables
plugins to intercept requests on both the sender and the receiver ends. This is a commonly used
and overwritten functionality but encapsulates the custom code in a contained manner.
2016-09-16 09:47:53 +02:00
Luca Cavanna 37489c3274 Add clusterUUID to RestMainAction output ()
Add clusterUUID to RestMainAction output

GET / now returns the clusterUUID as well as part of its output for monitoring purposes
2016-09-15 16:25:17 +02:00
Christoph Büscher f91ee9a897 Merge branch 'master' into feature/rank-eval 2016-09-15 11:38:40 +02:00
Jim Ferenczi 1764ec56b3 Fixed naming inconsistency for fields/stored_fields in the APIs ()
This change replaces the fields parameter with stored_fields when it makes sense.
This is dictated by the renaming we made in  for the search API.

The following list of endpoint has been changed to use `stored_fields` instead of `fields`:
* get
* mget
* explain

The documentation and the rest API spec has been updated to cope with the changes for the following APIs:
* delete_by_query
* get
* mget
* explain

The `fields` parameter has been deprecated for the following APIs (it is replaced by _source filtering):
* update: the fields are extracted from the _source directly.
* bulk: the fields parameter is used but fields are extracted from the source directly so it is allowed to have non-stored fields.

Some APIs still have the `fields` parameter for various reasons:
* cat.fielddata: the fields paramaters relates to the fielddata fields that should be printed.
* indices.clear_cache: used to indicate which fielddata fields should be cleared.
* indices.get_field_mapping: used to filter fields in the mapping.
* indices.stats: get stats on fields (stored or not stored).
* termvectors: fields are retrieved from the stored fields if possible and extracted from the _source otherwise.
* mtermvectors:
* nodes.stats: the fields parameter is used to concatenate completion_fields and fielddata_fields so it's not related to stored_fields at all.

Fixes 
2016-09-13 20:54:41 +02:00
Christoph Büscher 5398425f4f Remove top level spec_id
Currently the top level spec_id serves as a human-readable description of the
ranking evaluation API call. Since there is only one id possible, it can be
dropped to simplify the request.

Closes 
2016-09-13 13:10:47 +02:00
Christoph Büscher 9e2ec53458 Merge branch 'master' into feature/rank-eval 2016-09-13 12:47:42 +02:00
Christoph Büscher 7dc1c3349d Remove nested `key` field for rated documents
Every rated document needs an index/type/id parameter, so adding a "key" object
like we currently do only leads to an additional unneeded level of nesting in
the rest request.

Closes 
2016-09-13 12:12:39 +02:00
Lee Hinman 94625d74e4 No longer allow cluster name in data path
In 5.x we allowed this with a deprecation warning. This removes the code
added for that deprecation, requiring the cluster name to not be in the
data path.

Resolves 
2016-09-12 15:47:01 -06:00
Nik Everett 69bf08f6c6 Disable regexes by default in painless
Adds a new node level, non-dynamic setting, `script.painless.regex.enabled`
can be used to enable regexes.

Closes 
2016-09-12 14:09:43 -04:00
Christoph Büscher 63822c2745 Adapt to changes on master 2016-09-12 12:05:17 +02:00
Christoph Büscher 2fde4b2883 Merge branch 'master' into feature/rank-eval 2016-09-12 11:30:40 +02:00
Tal Levy 9f1f5fdedc introduce the JSON Processor ()
introduce the JSON Processor
2016-09-09 14:34:32 -07:00
Luca Cavanna 4b00cc37a1 Merge pull request from javanna/enhancement/cleanup_parse_elements
Cleanup sub fetch phase extension point
2016-09-09 22:47:15 +02:00
Tal Levy dda32545bb add ignore_missing option to relevant processors () 2016-09-09 12:20:18 -07:00
javanna 90ab460fcc move parsing of search ext sections to the coordinating node 2016-09-09 19:10:42 +02:00
Isabel Drost-Fromm 450b756152 We don't actually need this annotation anymore. 2016-09-07 13:54:27 +02:00
Isabel Drost-Fromm d34a117422 Use roundtrip helper for rank eval spec tests. 2016-09-07 13:03:18 +02:00
Isabel Drost-Fromm ba9956a468 Remove FromXContentBuilder from RatedDocumentKey 2016-09-07 12:52:30 +02:00
Isabel Drost-Fromm 9d8bb720dc Remove leftover FromXContentBuilder reference 2016-09-07 12:50:04 +02:00
Isabel Drost-Fromm 2af0218bdd Rename test class to match renamed implementation
Used to be QuerySpec, is now RatedRequest; changing name
of test accordingly.
2016-09-07 12:12:06 +02:00
Isabel Drost-Fromm 0d25fc4925 Re-use roundtrip helper for QuerySpecTests 2016-09-07 12:11:07 +02:00
Isabel Drost-Fromm b3a4a89151 Replace magic number in test with random number. 2016-09-07 12:06:39 +02:00
Isabel Drost-Fromm c8a3b3c32f Move xcontent roundtrip method to helper class. 2016-09-07 12:02:00 +02:00
Isabel Drost-Fromm efbef20361 Remove call to getClass from hashCode implementations. 2016-09-07 11:53:39 +02:00
Isabel Drost-Fromm 333b769871 Remove usage of FromXContentBuilder 2016-09-07 11:48:50 +02:00
Isabel Drost-Fromm 6bc646ac84 Make constructor enforce non-optional position argument. 2016-09-07 11:21:50 +02:00
Isabel Drost-Fromm b8652b1224 Test wouldn't compile w/o these annotations. 2016-09-07 11:21:29 +02:00
Isabel Drost-Fromm c4cfb9d7b5 Merge branch 'feature/rank-eval' into feature/rank-eval-roundtrip-testing 2016-09-07 11:05:56 +02:00
Christoph Büscher dab771a0b4 Merge branch 'master' into feature/rank-eval 2016-09-07 09:57:43 +02:00
Martijn van Groningen 245882cde3 * Removed `script.default_lang` setting and made `painless` the hardcoded default script language.
** The default script language is now maintained in `Script` class.
* Added `script.legacy.default_lang` setting that controls the default language for scripts that are stored inside documents (for example percolator queries).  This defaults to groovy.
** Added `QueryParseContext#getDefaultScriptLanguage()` that manages the default scripting language. Returns always `painless`, unless loading query/search request in legacy mode then the returns what is configured in `script.legacy.default_lang` setting.
** In the aggregation parsing code added `ParserContext` that also holds the default scripting language like `QueryParseContext`. Most parser don't have access to `QueryParseContext`. This is for scripts in aggregations.
* The `lang` script field is always serialized (toXContent).

Closes 
2016-09-06 18:44:48 +02:00
Alexander Reelsen b6bf20c2da Dependencies: Updates to mustache 0.9.3 () 2016-09-06 13:31:59 +02:00
Christoph Büscher 0b92d524a7 Add threshold for document ratings for PrecisionAtN and ReciprocalRank
PrecisionAtN and ReciprocalRank are binary evaluation metrics by default that
only distiguish between relevant/irrelevant search results. So far we assumed
that relevant documents are labaled with 1 (irrelevant docs with 0) in the
evaluation request, but this is cumbersome if the ratings are provided on a
larger integer scale and would need to get mapped to a 0/1 value.

This change introduces a threshold parameter on the PrecisionAtN and
ReciprocalRank metric than can be used to set the threshold from which on a
document is considered "relevant". It defaults to 1, so in case of 0/1 ratings
the threshold doesn't have to be set and only ratings with value 0 are
considered to be irrelevant.
2016-09-05 15:34:47 +02:00
Christoph Büscher 8910864a18 Merge branch 'master' into feature/rank-eval 2016-09-05 15:22:04 +02:00
Martijn van Groningen 6f6d17dc9c ingest: Add `dot_expander` processor that can turn fields with dots in the field name into object fields. 2016-09-05 07:28:38 +02:00
Christoph Büscher 8daec109fc Adding additional json object level to PrecisionAtN rendering 2016-09-02 18:03:58 +02:00
Christoph Büscher 11116db337 Merge branch 'master' into feature/rank-eval 2016-09-02 10:49:33 +02:00
javanna 536d13ff11 ProcessInfo to implement Writeable rather than Streamable 2016-09-02 10:23:05 +02:00
Jack Conradson d0f9ab06f6 Removed extraneous import. 2016-09-01 16:29:08 -07:00
Jack Conradson 71d8ee5eac Merge branch 'master' into deprecate 2016-09-01 08:51:29 -07:00
Christoph Büscher 1c02e48a8f Merge branch 'master' into feature/rank-eval
Conflicts:
	test/framework/src/main/java/org/elasticsearch/test/TestCluster.java
2016-09-01 13:06:21 +02:00
Martijn van Groningen a110498ad8 settings: Make `action.auto_create_index` setting a dynamic cluster setting.
Closes 
2016-09-01 12:33:30 +02:00
Jason Tedor d9064f454e Fix additional exception logging calls
This commit modifies a pair of exception logging calls to use
parameterized messages from Log4j.
2016-08-31 23:14:13 -04:00
Jack Conradson 1cdfc3ccfe Merge branch 'master' into deprecate 2016-08-31 15:58:14 -07:00
Jack Conradson 3b3baa6e6c Made deprecation of Groovy, Javascript, and Python more explicit. 2016-08-31 15:56:31 -07:00
Jason Tedor 76ab02e002 Merge branch 'master' into log4j2
* master:
  Avoid NPE in LoggingListener
  Randomly use Netty 3 plugin in some tests
  Skip smoke test client on JDK 9
  Revert "Don't allow XContentBuilder#writeValue(TimeValue)"
  [docs] Remove coming in 2.0.0
  Don't allow XContentBuilder#writeValue(TimeValue)
  [doc] Remove leftover from CONSOLE conversion
  Parameter improvements to Cluster Health API wait for shards ()
  Add 2.4.0 to packaging tests list
  Docs: clarify scale is applied at origin+offest ()
2016-08-31 16:37:55 -04:00
Jason Tedor 54083f7d6e Randomly use Netty 3 plugin in some tests
When Netty 4 was introduced, it was not the default network
implementation. Some tests were constructed to randomly use Netty 4
instead of the default network implementation. When Netty 4 was made the
default implementation, these tests were not updated. Thus, these tests
are randomly choosing between the default network implementation (Netty
4) and Netty 4. This commit updates these tests to reverse the role of
Netty 3 and Netty 4 so that the randomization is choosing between Netty
3 and the default (again, now Netty 4).

Relates 
2016-08-31 15:41:39 -04:00
Jason Tedor e166459bbe Merge branch 'master' into log4j2
* master:
  Increase visibility of deprecation logger
  Skip transport client plugin installed on JDK 9
  Explicitly disable Netty key set replacement
  percolator: Fail indexing percolator queries containing either a has_child or has_parent query.
  Make it possible for Ingest Processors to access AnalysisRegistry
  Allow RestClient to send array-based headers
  Silence rest util tests until the bogusness can be simplified
  Remove unknown HttpContext-based test as it fails unpredictably on different JVMs
  Tests: Improve rest suite names and generated test names for docs tests
  Add support for a RestClient base path
2016-08-31 10:59:27 -04:00
Martijn van Groningen 3fcb95b814 percolator: Fail indexing percolator queries containing either a has_child or has_parent query.
Closes 
2016-08-31 07:46:17 +02:00
Jason Tedor abf8a1a3f0 Avoid allocating log parameterized messages
This commit modifies the call sites that allocate a parameterized
message to use a supplier so that allocations are avoided unless the log
level is fine enough to emit the corresponding log message.
2016-08-30 18:17:09 -04:00
Jason Tedor 7da0cdec42 Introduce Log4j 2
This commit introduces Log4j 2 to the stack.
2016-08-30 13:31:24 -04:00
Jack Conradson c8f57df5cf Merge branch 'master' into deprecate 2016-08-30 09:11:18 -07:00
Jack Conradson 7930233527 Deprecate Groovy, Python, and Javascript scripts. 2016-08-30 09:06:18 -07:00
Isabel Drost-Fromm 8f71c29c16 Merge branch 'master' into feature/rank-eval 2016-08-30 14:20:43 +02:00
Jason Tedor 0df92a8da8 Upgrade to Netty 4.1.5
This commit upgrades the Netty dependencies from version 4.1.4 to
version 4.1.5. This upgrade brings several bug fixes including the
removal of a obnoxious and scary-looking log message when unsafe is
explicitly disabled.

Relates 
2016-08-30 05:46:40 -04:00
Martijn van Groningen 1925813e09 ingest: Fix rename processor change rename leaf fields into branch fields
Instead of get, set and remove we do get, remove and then set to avoid type conflicts in IngestDocument.
If the set still fails we try to restore the original field in ingest document.

Closes 
2016-08-30 07:38:01 +02:00
Tanguy Leroux 9727f123b9 Rename Netty TCP transports thread factories from http_* to transport_*
Netty3/4 TcpTransport implementations are creating thread factories with a "http_server" thread prefix whereas it should start with "transport_server" and let the "http_server" prefix for the HttpServerTransport implementations.
2016-08-29 13:49:52 +02:00
Jun Ohtani 2a00c9dc46 Merge pull request from johtani/fix/validate_empty_field_name
Validate blank field name
2016-08-29 11:52:18 +09:00
Martijn van Groningen 48926b4d66 ingest: don't render template twice for append processor 2016-08-26 18:07:32 +02:00
Chris Earle bd0b06440e Add "Async" to the end of each Async RestClient method
This makes it much harder to accidentally miss the Response.
2016-08-26 10:51:33 -04:00
Jun Ohtani 450f47d5b5 Validate blank field name
add validation and validate only 5.0+
Add tests before 5.0

Closes 
2016-08-26 20:10:33 +09:00
Jason Tedor bc136a90d5 Add network types to cluster stats
The network types in use on a cluster can be useful information to have,
so this commit adds aggregate metrics for the network types in use in a
cluster to the cluster stats.

Relates 
2016-08-25 21:08:05 -04:00
Jason Tedor 5a48ad661d Address race condition in HTTP pipeline tests
The Netty 4 HTTP server pipeline tests contains two different test
cases. The general idea behind these tests is to submit some requests to
a Netty 4 HTTP server, one test with pipelining enabled and another test
with pipelining disabled. These requests are submitted to two endpoints,
one with a path like /{id} and another with a path like /slow with a
query string parameter sleep. This parameter tells the request handler
how long to sleep for before replying. The idea is that in the case of
the pipelining enabled tests, the requests should come back exactly in
the order submitted, even with some of the requests hitting the slow
endpoint with random sleep durations; this is the guarantee that
pipelining provides. And in the case of the pipelining disabled tests,
requests were randombly submitted to /{id} and /slow with sleep
parameters starting at 600ms and increasing by 100ms for each slow
request constructed. We would expect the requests to come back with the
all the responses to the /{id} requests first because these requests
will execute instantaneously, and then the responses to the /slow
requests. Further, it was expected that the slow requests would come
back ordered by the length of the sleep, the thinking being that 100ms
should be enough of a difference between each request that we would
avoid any race conditions. Sadly, this is not the case, the threads do
sometimes hit race conditions.

This commit modifies the HTTP server pipelining tests to address this
race condition. The modification is that the query string parameter on
the /slow endpoint is removed in favor of just submitting requests to
the path /slow/{id}, where id just used a marker to distinguish each
request. The server chooses a random sleep of at least 500ms for each
request on the slow path. The assertion here then is that the /{id}
responses arrive first, then then /slow responses. We can not make an
assertion on the order of the responses, but we can assert that we did
see every expected response.

Relates 
2016-08-25 14:34:11 -04:00
Jack Conradson 0fdadf4737 Merge branch 'master' into break 2016-08-25 09:26:04 -07:00
Isabel Drost-Fromm 1d69a943be Merge branch 'feature/rank-eval' into feature/rank-eval-roundtrip-testing 2016-08-25 12:14:34 +02:00
Isabel Drost-Fromm 518e09d028 Merge branch 'master' into feature/rank-eval 2016-08-25 12:13:59 +02:00
Isabel Drost-Fromm f18e23eb51 Fix test errors, add roundtrip testing to RankEvalSpec
This adds roundtrip testing to RankEvalSpec, fixes issues
introduced with the previous roundtrip tests, splits
xcontent generation/parsing from actually checking the resulting
objects to deal with e.g. all evaluation metrics needing
some extra treatment.

Renames QuerySpec to RatedRequest, renames newly introduced xcontent
generation helper to conform with naming conventions.

Fixes several lines that were too long, adds missing types where
needed.
2016-08-25 12:09:55 +02:00
Michael McCandless 1fe3e36934 Merge pull request from mikemccand/lucene_620_upgrade
Upgrade to Lucene 6.2.0
2016-08-25 06:03:34 -04:00
Jack Conradson 3deea3dbde Made for/each break tests more robust in Painless. 2016-08-24 15:17:18 -07:00
Mike McCandless 0ccfe69789 Upgrade to Lucene 6.2.0 2016-08-24 17:26:28 -04:00
Jack Conradson c60885b5d4 Fix break bug in for/foreach loops. 2016-08-24 14:25:54 -07:00
Igor Motov b36fbc4452 Add support for parameters to the script ingest processor
The script processor should support `params` to be consistent with all other script consumers.
2016-08-24 16:49:48 -04:00
Jim Ferenczi 4682fc34ae Add the ability to disable the retrieval of the stored fields entirely
This change adds a special field named _none_ that allows to disable the retrieval of the stored fields in a search request or in a TopHitsAggregation.

To completely disable stored fields retrieval (including disabling metadata fields retrieval such as _id or _type) use _none_ like this:

````
POST _search
{
   "stored_fields": "_none_"
}
````
2016-08-24 16:40:08 +02:00
Isabel Drost-Fromm 87367be4e7 Add roundtrip testing to RatedDocumentKey 2016-08-24 15:32:44 +02:00
Isabel Drost-Fromm 94497871b5 Add roundtrip testing to QuerySpec 2016-08-24 15:17:58 +02:00
Isabel Drost-Fromm 5979802415 Add comment wrt to changed xcontent generation 2016-08-24 15:17:42 +02:00
Clinton Gormley abc025e18b Fixed the reindex_rethrottle REST tests
The API was renamed from reindex.rethrottle to reindex_rethrottle
2016-08-24 14:55:02 +02:00
Isabel Drost-Fromm cebb0ba0d8 Add roundtripping to ReciprocalRank 2016-08-24 14:41:58 +02:00
Isabel Drost-Fromm 5c9cc1d453 Add roundtripping to PrecisionAtN 2016-08-24 14:39:12 +02:00
Isabel Drost-Fromm a2a92b9629 Add roundtrip xcontent test to DiscountedCumulativeGainAt
This factors the roundtripping out of RatedDocumentTests. Makes
RankedListQualityMetric and RatedDocument implement FromXContenBuilder
to be able to do the aforementioned refactoring in a generic way. Adds
a roundtrip test to DiscountedCumulativeGainAt.

Open questions:

DiscountedCumulativeGain didn't have a constructor that accepted all possible
parameters as arguments. Added one. I guess we still want to keep the one
that only requires the position argument?

To make roundtripping work I had to change the NAME parameter when generating
XContent for DiscountedCumulativeGainAt - all remaining unit tests seem to be
passing (haven't checked the REST tests yet) - need to figure out why that was
there to begin with.
2016-08-24 14:21:24 +02:00
Isabel Drost-Fromm e5fd0cb3a2 Merge branch 'master' into feature/rank-eval 2016-08-24 12:44:37 +02:00
Areek Zillur 80ca78479f Make bulk item-level requests implement DocumentRequest interface
Currently, bulk item requests can be any ActionRequest, this commit
restricts bulk item requests to DocumentRequest. This simplifies
handling failures during bulk requests. Additionally, a new enum
is added to DocumentRequest to represent the intended operation
to be performed by a document request. Now, index operation type
also uses the new enum to specify whether the request should
create or index a document.
2016-08-23 10:33:37 -04:00
Jack Conradson 131e370a16 Make Painless the default scripting language.
Closes 
2016-08-22 17:38:02 -07:00
Christoph Büscher 2f506bfe04 Add toXContent method to classes used in ranking request 2016-08-18 18:14:21 +02:00
Christoph Büscher e58e25f338 Merge branch 'master' into feature/rank-eval 2016-08-18 10:43:48 +02:00
Nik Everett 312a7d45ba Wait for task to start in reindex test
`RethrottleTests#testReindex` fail in CI:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-intake/1274/console

I was unable to reproduce it locally but it *looks* like a race to start
the task. So I've added a wait for it to start just in case.
2016-08-17 12:08:55 -04:00
Nik Everett 39d8f5f123 Reindex tests should expect the right failure
Reindex intentionally tries to fail the search operation to make sure
that the exception flows back. The exception message changed so we
should catch the appropriate exception.
2016-08-17 10:25:38 -04:00
Christoph Büscher 2e892185f0 Adapting to introduction of SearchRequestParers on master 2016-08-17 11:54:07 +02:00
Christoph Büscher ff17b9b38f Merge branch 'master' into feature/rank-eval 2016-08-17 11:06:32 +02:00
Nik Everett 34bbd27f84 Fix _update_by_query's ingest pipeline support
It wasn't being serialized so it wasn't working with the transport
client.
2016-08-16 16:03:18 -04:00
Nik Everett 862843ec90 Suppress failing test
This test was failing in the presence of transport clients. This turns
off transport clients while I fix the test so it doesn't fail for
everyone in the mean time.
2016-08-16 15:12:40 -04:00
Ryan Ernst 743d9fd008 Merge branch 'master' into search_parser 2016-08-16 11:28:59 -07:00
Nik Everett fdd50612ae Fix reindex under the transport client
The big change here is cleaning up the `TaskListResponse` so it doesn't
have a breaky `toString` implementation. That was causing the reindex
tests to break.

Also removed `NetworkModule#registerTaskStatus` which is part of the
Plugin API. Use `Plugin#getNamedWriteables` instead.
2016-08-16 12:15:15 -04:00
Nik Everett 46bf8baf2e Switch aggregation registration for push to pull
Adds `getAggregations` to `SearchPlugin` which can be used to register
aggregations.

Fixup MockNode which wasn't createing MockBigArrays.
2016-08-16 09:08:36 -04:00
Christoph Büscher 1eedb4c033 Resolve missing imports due to changes in master 2016-08-16 11:48:28 +02:00
Ryan Ernst 7fde410586 Internal: Consolidate search parser registries
Parsing a search request is currently split up among a number of
classes, using multiple public static methods, which take multiple
regstries of elements that may appear in the search request like query
parsers and aggregations. This change begins consolidating all this code
by collapsing the registries normally used for parsing search requests
into a single SearchRequestParsers class. It is also made available to
plugin services to enable templating of search requests.  Eventually all
of the actual parsing logic should move to the class, and the registries
should be hidden, but for now they are at least co-located to reduce the
number of objects that must be passed around.
2016-08-16 01:59:24 -07:00
Christoph Büscher fe2bfc3a27 Merge branch 'master' into feature/rank-eval 2016-08-16 10:42:55 +02:00
Nik Everett 1452ab4b9f Squash the rest of o.e.rest.action
Squashes all the subpackages of `org.elasticsearch.rest.action` down to
the following:
* `o.e.rest.action.admin` - Administrative actions
* `o.e.rest.action.cat` - Actions that make tables for `grep`ing
* `o.e.rest.action.document` - Actions that act on documents
* `o.e.rest.action.ingest` - Actions that act on ingest pipelines
* `o.e.rest.action.search` - Actions that search

I'm tempted to merge `search` into `document` but the `document`
package feels fairly complete as is and `Suggest` isn't actually always
about documents either....

I'm also tempted to merge `ingest` into `admin.cluster` because the
latter contains the actions for dealing with stored scripts.

I've moved the `o.e.rest.action.support` into `o.e.rest.action`.

I've also added `package-info.java`s to all packges in `o.e.rest`. I
figure if the package is too small to deserve a `package-info.java` file
then it is too small to deserve to be a package....

Also fixes checkstyle in all moved classes.
2016-08-15 21:06:32 -04:00
Tal Levy 84bf24b1e9 remove ability to set field value in script-processor configuration () 2016-08-15 10:57:39 -07:00
Igor Motov 10a766704e Rename Task Persistence into Storing Task Results
The term persisted task was used to indicate that a task should store its results upon its completion. We would like to use this term to indicate that a task can survive restart of nodes instead. This commit removes usages of the term "persist" when it means store results.
2016-08-15 10:02:43 -04:00
Christoph Büscher a7e56d7fa5 Merge branch 'master' into feature/rank-eval 2016-08-15 11:26:34 +02:00
Nik Everett cf6e1a4362 Move all FetchSubPhases to `o.e.search.fetch.subphase`
As the most complicated `FetchSubPhase` highlighting gets its own package
(`o.e.seach.fetch.subphase.highlight`. No other `FetchSubPhase`s get their
own package. Instead they all reside together in `o.e.search.fetch.subphase`.

Add package descriptions to `o.e.search.fetch` and subpackages.
2016-08-12 18:21:15 -04:00
Lee Hinman f832eaa6d2 Merge remote-tracking branch 'dakrone/handle-groovy-errors-better' 2016-08-12 13:07:13 -06:00
Lee Hinman aab8c1f032 Catch AssertionError and NoClassDefFoundError in groovy scripts
Previously, we only caught subclasses of Exception, however, there are
some cases when Errors are thrown instead of Exceptions. These two cases
are `assert` and when a class cannot be found.

Without this change, the exception would bubble up to the
`uncaughtExceptionHandler`, which would in turn, exit the JVM
(related: ).

A note of difference between regular Java asserts and Groovy asserts,
from http://docs.groovy-lang.org/docs/latest/html/documentation/core-testing-guide.html

"Another important difference from Java is that in Groovy assertions are
enabled by default. It has been a language design decision to remove the
possibility to deactivate assertions."

In the event that a user uses an assert such as:

```groovy
def bar=false; assert bar, "message";
```

The GroovyScriptEngineService throws a NoClassDefFoundError being unable
to find the `java.lang.StringBuffer` class. It is *highly* recommended
that any Groovy scripting user switch to using regular exceptions rather
than unconfiguration Groovy assertions.

Resolves 
2016-08-12 13:06:49 -06:00
Christoph Büscher b7af7c21d1 Adapt to changes in master 2016-08-12 11:03:55 +02:00
Christoph Büscher 064def76eb Merge branch 'master' into feature/rank-eval 2016-08-12 10:40:14 +02:00
Nik Everett 9f8f2ea54b Remove ESIntegTestCase#pluginList
It was a useful method in 1.7 when javac's type inference wasn't as
good, but now we can just replace it with `Arrays.asList`.
2016-08-11 15:44:02 -04:00
Luca Cavanna a80a35ebc4 Merge pull request from javanna/fix/reindex_repleaceable
update and delete by query requests to implement IndicesRequest.Replaceable
2016-08-11 21:10:58 +02:00
javanna 4424d2263f UpdateByQueryRequest to implement IndicesRequest.Replaceable rather than CompositeIndicesRequest
Update by query is a shortcut to search + index. UpdateByQueryRequest gets serialized on the transport layer only when the transport client is used. Given that the request supports wildcards and allows to set its indices, it should implement IndicesRequest.Repleaceable. implementing CompositeIndicesRequest makes little sense as the indices that the request works against depend entirely on the inner search request.
2016-08-11 18:11:26 +02:00
javanna 11d770dde3 DeleteByQueryRequest to implement IndicesRequest.Replaceable
Delete by query is a shortcut to search + delete. DeleteByQueryRequest gets serialized on the transport layer only when the transport client is used. Given that the request supports wildcards and allows to set its indices, it should implement IndicesRequest.Repleaceable
2016-08-11 18:11:26 +02:00
Nik Everett e07e5d66fa Make reindex and lang-javascript compatible
Fixes two issues:
1. lang-javascript doesn't support `executable` with a `null` `vars`
parameters. The parameter is quite nullable.
2. reindex didn't support script engines who's `unwrap` method wasn't
a noop. This didn't come up for lang-groovy or lang-painless because
both of those `unwrap`s were noops. lang-javascript copys all maps that
it `unwrap`s.

This adds fairly low level unit tests for these fixes but dosen't add
an integration test that makes sure that reindex and lang-javascript
play well together. That'd make backporting this difficult and would
add a fairly significant amount of time to the build for a fairly rare
interaction. Hopefully the unit tests will be enough.
2016-08-11 09:54:03 -04:00
Christoph Büscher cac4961ef4 Fix test failure because of broken RankEvalResponse serialization
The introduction of RatedDocumentKey accidentally broke the response
serialization because it cannot be written using writeGenericValue().
2016-08-11 15:42:18 +02:00
Christoph Büscher ae721ef79b Merge branch 'master' into feature/rank-eval 2016-08-11 11:39:05 +02:00
Christoph Büscher a2e6dc2750 Adressing review comments and adding test for failed request 2016-08-11 10:08:40 +02:00
Christoph Büscher 8856565b89 Make RankEvalRequestTests work with transport client 2016-08-11 10:07:09 +02:00
Christoph Büscher 795017ddfa Adding listeners for search requests that collect results 2016-08-11 10:06:23 +02:00
Christoph Büscher e71e29b3a0 Using client instead of TransportSearchAction 2016-08-11 10:05:28 +02:00
Jack Conradson f38d64039b Catch OutOfMemory and StackOverflow errors in Painless since it's safe
to do so.
2016-08-10 18:09:45 -07:00
Ali Beyad 04e8272607 Fixes netty4 module's CORS config to use defaults ()
Fixes netty4 CORS config to use defaults for `http.cors.allow-methods`
and `http.cors.allow-headers` when none are specified.
2016-08-10 16:47:21 -04:00
Adrien Grand 0d6ac57acf Collapse o.e.index.mapper packages.
I also reduced the visibility of a couple classes and renamed/consolidated some
test classes for consistency, eg. removing the `Simple` prefix or using the
`<Type>FieldMapperTests` convention for testing field mappers.
2016-08-10 17:51:11 +02:00
Tanguy Leroux cefc346d37 [Test] Add test for Netty4 HTTP support of header 100-continue ()
* [Test] Add test for Netty4 HTTP support of header 100-continue

Related to 

* Update after David comments
2016-08-10 07:22:18 -04:00
Jason Tedor f025c83a17 Fix expect 100 continue header handling
Due to a misordering of the HTTP handlers, the Netty 4 HTTP server
mishandles Expect: 100-continue headers from clients. This commit fixes
this issue by ordering the handlers correctly.

Relates 
2016-08-10 07:21:22 -04:00
javanna 2c44278ce8 [TEST] use ParseField instead of plain strings in query tests 2016-08-10 12:21:25 +02:00
javanna 0a98b5e56e [TEST] make AbstractQueryTestCase#testUnknownObjectException more accurate
testUnknownObjectException used to generate malformed json objects in some cases, due to the existence of arrays as it was not closing the injected object correctly. That is why the test was catching JsonParseException among the exception that are expected to be thrown. That is fixed by tracking where the new object is placed and placing its end object marker to the right level rather than always at the end.

Also introduced a mechanism to explicitly declare objects that won't cause any exception when they get additional objects injected, so that there is no need to override the method anymore as that caused copy pasting of the whole test method. This also makes sure that changes are reflected in tests, as those inner objects are not skipped but we actually check that what is declared is true (no exceptions get thrown when an additional object is added within them.
2016-08-10 11:48:51 +02:00
David Pilato 90dbce9682 Merge branch 'fix/19772-toString' 2016-08-09 20:37:27 +02:00
Lee Hinman 5849c488b5 Merge remote-tracking branch 'dakrone/compliation-breaker' 2016-08-09 11:57:26 -06:00
Lee Hinman 2be52eff09 Circuit break the number of inline scripts compiled per minute
When compiling many dynamically changing scripts, parameterized
scripts (<https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-scripting-using.html#prefer-params>)
should be preferred. This enforces a limit to the number of scripts that
can be compiled within a minute. A new dynamic setting is added -
`script.max_compilations_per_minute`, which defaults to 15.

If more dynamic scripts are sent, a user will get the following
exception:

```json
{
  "error" : {
    "root_cause" : [
      {
        "type" : "circuit_breaking_exception",
        "reason" : "[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead",
        "bytes_wanted" : 0,
        "bytes_limit" : 0
      }
    ],
    "type" : "search_phase_execution_exception",
    "reason" : "all shards failed",
    "phase" : "query",
    "grouped" : true,
    "failed_shards" : [
      {
        "shard" : 0,
        "index" : "i",
        "node" : "a5V1eXcZRYiIk8lecjZ4Jw",
        "reason" : {
          "type" : "general_script_exception",
          "reason" : "Failed to compile inline script [\"aaaaaaaaaaaaaaaa\"] using lang [painless]",
          "caused_by" : {
            "type" : "circuit_breaking_exception",
            "reason" : "[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead",
            "bytes_wanted" : 0,
            "bytes_limit" : 0
          }
        }
      }
    ],
    "caused_by" : {
      "type" : "general_script_exception",
      "reason" : "Failed to compile inline script [\"aaaaaaaaaaaaaaaa\"] using lang [painless]",
      "caused_by" : {
        "type" : "circuit_breaking_exception",
        "reason" : "[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead",
        "bytes_wanted" : 0,
        "bytes_limit" : 0
      }
    }
  },
  "status" : 500
}
```

This also fixes a bug in `ScriptService` where requests being executed
concurrently on a single node could cause a script to be compiled
multiple times (many in the case of a powerful node with many shards)
due to no synchronization between checking the cache and compiling the
script. There is now synchronization so that a script being compiled
will only be compiled once regardless of the number of concurrent
searches on a node.

Relates to 
2016-08-09 10:26:27 -06:00
Isabel Drost-Fromm 0be997232a Add ConstructingObjectParser to RatedDocumentKey 2016-08-09 12:49:07 +02:00
Isabel Drost-Fromm c2bd58e13d Switch to ConstructingObjectParser 2016-08-09 12:22:44 +02:00
David Pilato 4d272cc9b2 Merge branch 'master' into fix/19772-toString
# Conflicts:
#	core/src/test/java/org/elasticsearch/action/admin/cluster/node/tasks/TransportTasksActionTests.java
2016-08-09 11:53:29 +02:00
Isabel Drost-Fromm 978d5366d2 Merge branch 'feature/rank-eval' into feature/rank-eval_index_type_to_id 2016-08-09 10:54:14 +02:00
Christoph Büscher fa459f88dd Add normalization option
When switched on, compute the normalized ndcg variant.
2016-08-08 16:37:15 +02:00
Christoph Büscher 87e13ca8bb Add Discounted Cumulative Gain metric 2016-08-08 16:37:15 +02:00
Isabel Drost-Fromm b2fa7c4d96 Fix merge errors. 2016-08-08 14:41:58 +02:00
Isabel Drost-Fromm a6caabb730 Merge branch 'feature/rank-eval' into feature/rank-eval_index_type_to_id 2016-08-08 14:21:19 +02:00
Isabel Drost-Fromm cfaa62723d Actually use index/type in addition to docid for comparing hits. 2016-08-08 14:09:27 +02:00
Christoph Büscher 0578a96483 Merge RankEvalResult with Response
The current response object only serves as a wrapper around the result object.
This change merges the two classes into one.
2016-08-08 13:29:44 +02:00
Christoph Büscher 4385b29f80 Merge branch 'master' into feature/rank-eval 2016-08-08 10:10:58 +02:00
Nicholas Knize 2d590af593 Deprecate GeoDistance enumerators and remove geo distance script helpers
GeoDistance is implemented using a crazy enum that causes issues with the scripting modules. This commit moves all distance calculations to arcDistance and planeDistance static methods in GeoUtils. It also removes unnecessary distance helper methods from ScriptDocValues.GeoPoints.
2016-08-05 18:42:06 -05:00
javanna 2437226802 [TEST] restore tests repeatability in AbstractQueryTestCase
Some random operations were conditionally performed in the before test, which made tests not repeatable. For instance take the seed chain to repeat a specific iteration and try to reproduce it, this conditional code would get executed in both cases when trying to isolate the failure, but not among the different iterations (as only the first method/iteration executes it), hence the failure will not reproduce.

Moved the random operations to beforeClass and left the non random part in the before method, which is needed as it depends on some method that can be overridden by subclasses.
2016-08-05 22:38:31 +02:00
Jason Tedor a62740bbd2 Avoid early initializing Netty
Today when we load the Netty plugins, we indirectly cause several Netty
classes to initialize. This is because we attempt to load some classes
by name, and loading these classes is done in a way that triggers a long
chain of class initializers within Netty. We should not do this, this
can lead to log messages before the logger is loader, and it leads to
initialization in cases when the classes would never be needed (for
example, Netty 3 class initialization is never needed if Netty 4 is
used, and vice versa). This commit avoids this early initialization of
these classes by removing the need for the early loading.

Relates 
2016-08-05 14:58:33 -04:00
Tanguy Leroux 841d5a210e Update to Jackson 2.8.1
This commit updates Jackson to the 2.8.1 version, which is more strict when it comes to build objects. It also adds the snakeyaml dependency that was previously shaded in jackson libs.

It also closes 
2016-08-05 12:26:06 +02:00
Christoph Büscher 3895aa470a Merge branch 'master' into feature/rank-eval 2016-08-05 12:09:07 +02:00
Nik Everett 1e587406d8 Fail yaml tests and docs snippets that get unexpected warnings
Adds `warnings` syntax to the yaml test that allows you to expect
a `Warning` header that looks like:
```
    - do:
        warnings:
            - '[index] is deprecated'
            - quotes are not required because yaml
            - but this argument is always a list, never a single string
            - no matter how many warnings you expect
        get:
            index:    test
            type:    test
            id:        1
```

These are accessible from the docs with:
```
// TEST[warning:some warning]
```

This should help to force you to update the docs if you deprecate
something. You *must* add the warnings marker to the docs or the build
will fail. While you are there you *should* update the docs to add
deprecation warnings visible in the rendered results.
2016-08-04 15:23:05 -04:00
Christoph Büscher 438893c0c7 Adapt to changes to how named writables are registered 2016-08-04 13:40:46 +02:00
Christoph Büscher 1b4b4a6298 Merge branch 'master' into feature/rank-eval 2016-08-04 12:52:23 +02:00
Christoph Büscher d71dc205fa Moving averaging of partial evaluation results to RankedListQualityMetric
For the two current metrics Prec@ and reciprocal rank we currently average the
partial results in the transport action. If other metric later need a different
behaviour or want to parametrize this, this operation should be part of the
metric itself, so this change moves it there. Also removing on of the two test
packages, main code is also in one package only.
2016-08-04 10:34:46 +02:00
Jason Tedor 7d750d2811 Increase Netty 3 REST test suite timeout
This commit increases the Netty 3 REST test suite timeout to thirty
minutes. This is to address these tests running slowly after increasing
the number of nodes in the tests to two. This has surfaced that the
tests are heavily impacted by excessive fsyncs from most tests using the
default number of shards of five.
2016-08-03 21:31:48 -04:00
David Pilato 54603903f3 Remove ListTasksResponse#setDiscoveryNodes 2016-08-04 02:02:51 +02:00
Jason Tedor e74d02138f Merge branch 'master' into netty4
* master:
  Fix REST test documentation
  [Test] move methods from bwc test to test package for use in plugins ()
  package-info.java should be in src/main only.
  Split regular histograms from date histograms. 
  Tighten up concurrent store metadata listing and engine writes ()
  Plugins: Make NamedWriteableRegistry immutable and add extenion point for named writeables
  Add documentation for the 'elasticsearch-translog' tool
  [TEST] Increase time waiting for all shards to move off/on to a node
  Fixes the active shard count check in the case of ()
  Fixes cat tasks operation in detailed mode
  ignore some docker craziness in scccomp environment checks
2016-08-03 09:16:18 -04:00
Isabel Drost-Fromm ac3f2421b1 Merge branch 'origin/feature/rank-eval' into feature/rank-eval_index_type_to_id 2016-08-03 13:50:13 +02:00
Ryan Ernst 18f242b069 Merge pull request from rjernst/writeable_registry
Make NamedWriteableRegistry immutable and add extension point for named writeables
2016-08-03 01:36:38 -07:00
Adrien Grand a0818d3b87 Split regular histograms from date histograms.
Currently both aggregations really share the same implementation. This commit
splits the implementations so that regular histograms can support decimal
intervals/offsets and compute correct buckets for negative decimal values.

However the response API is still the same. So for intance both regular
histograms and date histograms will produce an
`org.elasticsearch.search.aggregations.bucket.histogram.Histogram`
aggregation.

The optimization to compute an identifier of the rounded value and the
rounded value itself has been removed since it was only used by regular
histograms, which now do the rounding themselves instead of relying on the
Rounding abstraction.

Closes 
Closes 
2016-08-03 08:39:48 +02:00
Ryan Ernst df8dc64e9b Plugins: Make NamedWriteableRegistry immutable and add extenion point for named writeables
Currently any code that wants to added NamedWriteables to the
NamedWriteableRegistry can do so via guice injection of the registry,
and registering at construction time. However, this makes the registry
complex: it has both get and register methods synchronized, and there is
likely contention on the read side from multiple threads.  The
registration has mostly already been contained to guice modules at node
construction time.

This change makes the registry immutable, taking all of the
NamedWriteable readers at construction time. It also allows plugins to
added arbitrary named writables that it may use in its own transport
actions.
2016-08-02 15:56:25 -07:00
Jason Tedor 0461e12663 Simplify Netty 4 transport implementations
The Netty 4 transport implementations have an unnecessary dependency on
SocketChannels, and can instead just use plain Channels.
2016-08-02 17:07:30 -04:00
jaymode 6def10c5d9 make netty4 http request public 2016-08-02 14:46:44 -04:00
Jason Tedor 669daccfbb Allow for Netty 4 HTTP extensions
This commit enables the Netty 4 HTTP server implementation to allow for
extensions.
2016-08-02 14:10:40 -04:00
jaymode 00ca6c417e add javadocs 2016-08-02 14:06:29 -04:00
jaymode 8498470781 update transport to allow for extensions 2016-08-02 13:45:41 -04:00
Jason Tedor 227463c356 Default transport and HTTP to Netty 4
This commit sets the default transport and HTTP implementation to use
transport-netty4.

Relates 
2016-08-02 12:19:19 -04:00
Ali Beyad 3d2a105825 Merge pull request from abeyad/remove-write-consistency-level
Removes write consistency level across replication action APIs in favor of wait_for_active_shards
2016-08-02 09:01:11 -04:00
Isabel Drost-Fromm 0fb7dd9054 Merge pull request from MaineC/feature/rank-eval
Add parsing of type information
2016-08-02 13:50:16 +02:00
Christoph Büscher 0188363a40 Merge branch 'master' into feature/rank-eval 2016-08-02 12:05:44 +02:00
Nicholas Knize 63f33e0f1e Serialize doc counts in Matrix-Stats module
This fixes a bug in the RunningStats class for the matrix stats aggregation module. doc counts were not being searlized which means they were only computed the first time the aggregation was computed. This was causing incorrect results when the aggregation was pulled from cache.
2016-08-01 18:05:10 -05:00
Ali Beyad 6a7d005081 Makes the index.write.wait_for_active_shards setting index-level and
dynamically updatable for both index creation and write operations.
2016-08-01 13:37:05 -04:00
Ali Beyad 25d8eca62d Removes the notion of write consistency level across all APIs in
favor of waiting for active shard copy count (wait_for_active_shards).
2016-08-01 13:35:29 -04:00
Tanguy Leroux 386902903e [TEST] Kill remaining lang-groovy messy tests
After  many tests that used Groovy scripts (for good or bad reason) in their tests have been moved in the lang-groovy module and the issue  has been created to track these messy tests in order to clean them up.

The work started with ,  and  and this PR moves the remaining messy tests back in core, removes the dependency on Groovy, changes the scripts in order to use the mocked script engine, and change the tests to integration tests.

It also moves IndexLookupIT test back (even if it has good chance to be removed soon) and fixes its tests.

It also changes AbstractQueryTestCase to use custom script plugins in tests.

closes 
2016-08-01 16:59:47 +02:00
Alexander Lin 9ac6389e43 Rename operation to result and reworking responses
* Rename operation to result and reworking responses
* Rename DocWriteResponse.Operation enum to DocWriteResponse.Result

These are just easier to interpret names.

Closes 
2016-08-01 10:42:58 -04:00
Christoph Büscher 8efccd1165 Merge branch 'master' into feature/rank-eval 2016-08-01 11:50:14 +02:00
Nik Everett 303c9faca5 Squash o.e.rest.action.admin.cluster
In an effort to reduce the number of tiny packages we have in the
code base this moves all the files that were in subdirectories of
`org.elasticsearch.rest.action.admin.cluster` into
`org.elasticsearch.rest.action.admin.cluster`.

Also fixes line length in these packages.
2016-07-29 20:31:24 -04:00
Nik Everett 6f24866902 Reindex: Only ask for _version we need it
`_reindex` only needs the `_version` if the `dest` has
`"version_type": "external"`. So it shouldn't ask for it unless it does.

`_update_by_query` and `_delete_by_query` always need the `_version`.

Closes 
2016-07-29 17:13:27 -04:00
Alexander Lin 119026b4fb Remove isCreated and isFound from the Java API
This is cleanup work from , where @nik9000 suggested trying to nuke the isCreated and isFound methods. I've combined nuking the two methods with removing UpdateHelper.Operation in favor of DocWriteResponse.Operation here.

Closes .
2016-07-29 14:21:43 -04:00
Nik Everett c9790a1257 Use fewer threads when reindexing-from-remote
Reindex from remote uses the Elasticsearch client which uses apache
httpasyncclient which spins up 5 thread by default, 1 as a dispatcher
and 4 more to handle IO. This changes Reindex's usage so it only spins
up two thread - 1 dispatcher and one to handle io. It also renames the
threads to "es-client-$taskid-$thread_number". That way if we see any
thread sticking around we can trace it back to the task.
2016-07-29 14:13:10 -04:00
Martijn van Groningen a91bb29585 ingest: Made the response format of the get pipeline api match with the response format of the index template api
Closes 
2016-07-29 17:58:30 +02:00
Alexander Reelsen 30b497622b Dependencies: Upgrade to netty 4.1.4 ()
Removes our internal snapshot repo we used until 4.1.4 was released.
2016-07-29 17:53:12 +02:00
Martijn van Groningen 7b36a72ccb fixed compiler warnining and removed unused imports 2016-07-29 15:06:49 +02:00
Christoph Büscher f31fecc5a4 Merge branch 'master' into feature/rank-eval 2016-07-29 12:13:15 +02:00
David Pilato 0d2ccf0989 Merge branch 'pr/15724-gce-network-host-master' 2016-07-28 16:59:18 +02:00
Isabel Drost-Fromm 869e471f9f Add missing file 2016-07-28 11:42:05 +02:00
Isabel Drost-Fromm 34cbc10128 Add index and type information to rated doc
Also add roundtrip testing of the xcontent serialisation of RatedDoc
2016-07-28 11:41:54 +02:00
Jack Conradson 9d87314105 Refactored Painless link nodes into expression nodes to simplify
load/store operations of variable/method chains.

Closes 
2016-07-27 14:01:08 -07:00
Nik Everett fb45f6a8a8 Add authentication to reindex-from-remote
The tests for authentication extend ESIntegTestCase and use a mock
authentication plugin. This way the clients don't have to worry about
running it. Sadly, that means we don't really have good coverage on the
REST portion of the authentication.

This also adds ElasticsearchStatusException, and exception on which you
can set an explicit status. The nice thing about it is that you can
set the RestStatus that it returns to whatever arbitrary status you like
based on the status that comes back from the remote system.
reindex-from-remote then uses it to wrap all remote failures, preserving
the status from the remote Elasticsearch or whatever proxy is between us
and the remove Elasticsearch.
2016-07-27 14:17:41 -04:00
Christoph Büscher 71c0d59e95 Add option for maximally allowed rank to Reciprocal Rank metric 2016-07-27 16:24:41 +02:00
Isabel Drost-Fromm ad9f060dc7 Use type information in request
Adds parsing of type and actually using it in TransportRankEvalAction.

Missing: A good idea how to actually test this in isolation...
2016-07-27 15:49:37 +02:00
Isabel Drost-Fromm 2bb5cb83a7 Merge branch 'feature/rank-eval' of github.com:elasticsearch/elasticsearch into feature/rank-eval 2016-07-27 15:46:44 +02:00
Christoph Büscher ad87bacf91 Add Reciprocal Rank query evaluation metric
This adds a second query evaluation metric alongside precision_at. Reciprocal
Rank is defined as 1/rank, where rank is the position of the first relevant
document in the search result. The results are averaged across all queries
across the sample of queries, according to
https://en.wikipedia.org/wiki/Mean_reciprocal_rank
2016-07-27 15:23:05 +02:00
Martijn van Groningen 7e3d5b21bb test: fix generic type 2016-07-27 14:56:17 +02:00
Christoph Büscher 4162582ee8 Adapt to renaming of RestTestCandidate 2016-07-27 11:39:59 +02:00
David Pilato e9339a1960 Merge branch 'master' into pr/15724-gce-network-host-master 2016-07-27 11:24:53 +02:00
David Pilato 1c03d0bb2a Fix after merge 2016-07-27 11:23:18 +02:00
Christoph Büscher 3ac35d1f4a Merge branch 'master' into feature/rank-eval 2016-07-27 11:14:30 +02:00
Martijn van Groningen d1b7a335cb removed useless comment and make sure all public constructors delegate to the package protected constructor. 2016-07-27 10:01:36 +02:00
Martijn van Groningen 2fdf79d8d4 Deprecate template query.
Closes 
2016-07-27 09:50:44 +02:00
Martijn van Groningen 24d7fa6d54 ingest: Change the `foreach` processor to use the `_ingest._value` ingest metadata attribute to store the current array element being processed.
Closes 
2016-07-27 09:35:09 +02:00
Nik Everett 9270e8b22b Rename client yaml test infrastructure
This makes it obvious that these tests are for running the client yaml
suites. Now that there are other ways of running tests using the REST
client against a running cluster we can't go on calling the shared
client yaml tests "REST tests". They are rest tests, but they aren't
**the** rest tests.
2016-07-26 13:53:44 -04:00
Chris Earle 0553ba9151 [Ingest] Add REST _ingest/pipeline to get all pipelines
This adds an extra REST handler for "_ingest/pipeline" so that users do not need to supply "_ingest/pipeline/*" to get all of them.

- Also adds a teardown section to related REST-tests for ingest.
2016-07-26 13:48:15 -04:00
David Pilato 0d3edee928 Merge branch 'master' into pr/15724-gce-network-host-master 2016-07-26 18:51:01 +02:00
David Pilato fde15ae470 Move custom name resolvers to NetworkService CTOR
Instead of using NetworkModule we can directly inject them in NetworkService CTOR.

See https://github.com/elastic/elasticsearch/pull/15765#issuecomment-235307974
2016-07-26 18:26:30 +02:00
Alexander Lin 8f2882a442 Add _operation field to index, update, delete responses
Performing the bulk request shown in  now results in the following:
```
{"_index":"test","_type":"test","_id":"1","_version":1,"_operation":"create","forced_refresh":false,"_shards":{"total":2,"successful":1,"failed":0},"status":201}
{"_index":"test","_type":"test","_id":"1","_version":1,"_operation":"noop","forced_refresh":false,"_shards":{"total":2,"successful":1,"failed":0},"status":200}
```
2016-07-26 11:16:19 -04:00
Christoph Büscher ccf275af7b Adapt to introduction of ESClientYamlSuiteTestCase 2016-07-26 16:10:47 +02:00
Christoph Büscher 91f606988a Merge branch 'master' into feature/rank-eval 2016-07-26 15:49:07 +02:00
Daniel Mitterdorfer 94bc489275 Temporarily increase Netty4RestIT timeout
Recently, we experience timeouts on our Windows build slaves for
Netty4RestIT. Until we have figured out what's going on, we
increase this test suite's timeout temporarily to ensure this
timeout does not mask other problems.
2016-07-26 08:27:47 +02:00
Daniel Mitterdorfer d7d3b1a874 Remove duplicate dependency declaration for http client ()
We disable transitive dependencies in our build plugin
for all dependencies except for the group `org.elasticsearch`.
However, in the reindex plugin we depend on the REST client
and declare its dependencies again which is not necessary
(and led to problems with conflicting versions in ).

With this PR we remove the duplicate declaration.
2016-07-26 06:29:04 +02:00
Nik Everett a95d4f4ee7 Add Location header and improve REST testing
This adds a header that looks like `Location: /test/test/1` to the
response for the index/create/update API. The requirement for the header
comes from https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

https://tools.ietf.org/html/rfc7231#section-7.1.2 claims that relative
URIs are OK. So we use an absolute path which should resolve to the
appropriate location.

Closes 

This makes large changes to our rest test infrastructure, allowing us
to write junit tests that test a running cluster via the rest client.
It does this by splitting ESRestTestCase into two classes:
* ESRestTestCase is the superclass of all tests that use the rest client
to interact with a running cluster.
* ESClientYamlSuiteTestCase is the superclass of all tests that use the
rest client to run the yaml tests. These tests are shared across all
official clients, thus the `ClientYamlSuite` part of the name.
2016-07-25 17:02:40 -04:00
Martijn van Groningen a9ab095b8c Moved all mustache classes into one package.
No need for multiple packages inside a small module.
2016-07-25 18:50:12 +02:00
Martijn van Groningen a784055db1 Cleaned up the tests in lang-mustache.
Messy tests with mustache were either moved to core, moved to a rest test or remained untouched if they actually tested mustache.
Also removed tests that were redundant.
2016-07-25 17:57:39 +02:00
Tanguy Leroux f745c96949 Clean up more messy tests
After  many tests that used Groovy scripts (for good or bad reason) in their tests have been moved in the lang-groovy module and the issue  has been created to track these messy tests in order to clean them up.

This commit moves more tests back in core, removes the dependency on Groovy, changes the scripts in order to use the mocked script engine, and change the tests  to integration tests.
2016-07-25 17:02:49 +02:00
Jason Tedor 2d1b0587dd Introduce Netty 4
This commit adds transport-netty4, a transport and HTTP implementation
based on Netty 4.

Relates 
2016-07-22 22:26:35 -04:00
javanna 36becd9bfe Merge branch 'master' into feature/async_rest_client 2016-07-22 22:22:03 +02:00
Ali Beyad 2b9cfff90f Fixes CORS handling so that it uses the defaults
Fixes CORS handling so that it uses the defaults for http.cors.allow-methods
and http.cors.allow-headers if none are specified in the config.

Closes 
2016-07-22 12:25:28 -04:00
javanna d13a3d3761 Reindex from remote: add fallback in case content-type header is not set
We better read the header, but who knows what can happen, maybe headers are filtered out for some reasons and we don't want to run into an NPE, then we fallback to auto-detection.
2016-07-22 16:46:17 +02:00
Simon Willnauer 6adfeb4dde [TEST] Add Netty3RestIT to actually run REST tests on netty3 2016-07-22 16:29:14 +02:00
javanna db8beeba3b Merge branch 'master' into feature/async_rest_client 2016-07-22 15:51:03 +02:00
Tal Levy 19e7b1c737 fix: no other processors should be executed after on_failure is called in a compound processor () 2016-07-21 14:27:04 -07:00
Isabel Drost-Fromm c8d3098d3e Move github review comments to TODOs 2016-07-21 15:44:47 +02:00
Isabel Drost-Fromm 2d57fd10b1 Merge branch 'master' into feature/rank-eval 2016-07-21 14:54:49 +02:00
Tal Levy f7cd86ef6d rethrow script compilation exceptions into ingest configuration exceptions ()
* rethrow script compilation exceptions into ingest configuration exceptions
* update readProcessor to rethrow any exception as an ElasticsearchException
2016-07-20 10:37:56 -07:00
Nik Everett 3a82c613e4 Migrate query registration from push to pull
Remove `ParseField` constants used for names where there are no deprecated
names and just use the `String` version of the registration method instead.

This is step 2 in cleaning up the plugin interface for extending
search time actions. Aggregations are next.

This is breaking for plugins because those that register a new query should
now implement `SearchPlugin` rather than `onModule(SearchModule)`.
2016-07-20 12:33:51 -04:00
Nik Everett fc4b439635 Remove AggregationStreams and friends
* Remove outdated aggregation registration method
* Remove AggregationStreams
* Adds StreamInput#readNamedWriteableList and
StreamOutput#writeNamedWriteableList convenience methods. We strive to
make the reading and writing from the streams terse so they are easier
to scan visually.
* Remove PipelineAggregatorStreams
* Remove stream info from InternalAggreation.Type
* Remove InternalAggregation#type
* Remove Streamable from PipelineAggregator
* Remove Streamable from MultiBucketsAggregation.Bucket
2016-07-20 09:46:04 -04:00
javanna bce54cf38d reindex from remote to read content-type header rather than guessing content type based on content 2016-07-19 15:16:45 +02:00
javanna 54fa997545 Reindex from remote: remove async client in favour of using RestClient performRequest async method 2016-07-19 15:16:45 +02:00
javanna 1fbec71243 Rest client: introduce async performRequest method and use async client under the hood for sync requests too
The new method accepts the usual parameters (method, endpoint, params, entity and headers) plus a response listener and an async response consumer. Shortcut methods are also added that don't require params, entity and the async response consumer optional.

There are a few relevant api changes as a consequence of the move to async client that affect sync methods:
- Response doesn't implement Closeable anymore, responses don't need to be closed
- performRequest throws Exception rather than just IOException, as that is the the exception that we get from the FutureCallback#failed method in the async http client
- ssl configuration is a bit simpler, one only needs to call setSSLStrategy from a custom HttpClientConfigCallback, that doesn't end up overridng any other default around connection pooling (it used to happen with the sync client and make ssl configuration more complex)

Relates to 
2016-07-19 15:15:58 +02:00
javanna 118a14fbe3 Build: upgrade httpcore version to 4.4.5
Closes 
2016-07-19 15:11:40 +02:00
Nik Everett d573541f66 Support requests_per_second=-1 to mean no throttling in reindex
This is entirely on the REST level, Float.POSITIVE_INFINITY is still
how you get no throttling over the transport api.

Closes 
2016-07-18 13:05:06 -04:00
Simon Willnauer 8394544548 Add a dedicated client/transport project for transport-client ()
The `client/transport` project adds a new jar build project that
pulls in all dependencies and configures all required modules.

Preinstalled modules are:
 * transport-netty
 * lang-mustache
 * reindex
 * percolator

The `TransportClient` classes are still in core
while `TransportClient.Builder` has only a protected construcutor
such that users are redirected to use the new `TransportClientBuilder`
from the new jar.

Closes 
2016-07-18 15:42:24 +02:00
Martijn van Groningen e0ebf5da1c Template cleanup:
* Removed `Template` class and unified script & template parsing logic. Templates are scripts, so they should be defined as a script. Unless there will be separate template infrastructure, templates should share as much code as possible with scripts.
* Removed ScriptParseException in favour for ElasticsearchParseException
* Moved TemplateQueryBuilder to lang-mustache module because this query is hard coded to work with mustache only
2016-07-18 10:16:01 +02:00
Ali Beyad dddbe4acad Fix MatrixAggregationRestIT test 2016-07-15 19:41:47 -04:00
Ali Beyad 19d0dbcd17 Removes waiting for yellow cluster health upon index ()
creation in the REST tests, as we no longer need it due
to index creation now waiting for active shard copies
before returning (by default, it waits for the primary of
each shard, which is the same as ensuring yellow health).

Relates 
2016-07-15 17:18:34 -04:00
Ali Beyad 6acb8b31fc Removes ensureYellow() calls after index creation in the ()
integration tests, as they are no longer needed with
index creation now waiting for shards to be started before
returning from the index creation call (by default, it waits
for the primary of each shard to be started before returning,
which is what ensureYellow() was ensuring anyway).

Closes 
Relates 
2016-07-15 15:37:35 -04:00
Martijn van Groningen d0069f0fbb Provide access to ThreadContext in ingest plugins
Also introduced a `Processor.Parameters` class that is holder for several services processors rely on,
the  IngestPlugin#getProcessors(...) method has been changed to accept `Processor.Parameters` instead
of each service seperately.
2016-07-15 08:16:15 +02:00
Jason Tedor 31c648eee8 Rename transport-netty to transport-netty3
This commit renames the Netty 3 transport module from transport-netty to
transport-netty3. This is to make room for a Netty 4 transport module,
transport-netty4.

Relates 
2016-07-14 22:03:14 -04:00
Ryan Ernst b5b0f04e34 Merge pull request from rjernst/client_plugin
Add flag for plugins/modules which can be used in the transpory client
2016-07-14 08:56:46 -07:00
Simon Willnauer 5616251f22 Remove `node.mode` and `node.local` settings ()
Today `node.mode` and `node.local` serve almost the same purpose, they
are a shortcut for `discovery.type` and `transport.type`. If `node.local: true`
or `node.mode: local` is set elasticsearch will start in _local_ mode which means
only nodes within the same JVM are discovered and a non-network based transport
is used. The _local_ mode it only really used in tests or if nodes are embedded.
For both, embedding and tests explicit configuration via `discovery.type` and `transport.type`
should be preferred.

This change removes all the usage of these settings and by-default doesn't
configure a default transport implemenation since netty is now a module. Yet, to make
the user expericence flawless, plugins or modules can set a `http.type.default` and
`transport.type.default`. Plugins set this via `PluginService#additionalSettings()`
which enforces _set-once_ which prevents node startup if set multiple times. This means
that our distributions will just startup with netty transport since it's packaged as a
module unless `transport.type` or `http.transport.type` is explicitly set.

This change also found a bunch of bugs since several NamedWriteables were not registered if a
transport client is used. Now that we don't rely on the `node.mode` leniency which is inherited
instead of using explicit settings, `TransportClient` uses `AssertingLocalTransport` which detects these problems since it serializes all messages.

Closes 
2016-07-14 13:21:10 +02:00
Ryan Ernst 9b63fc37c8 Make module client jars go to org.elasticsearch.plugin groupId 2016-07-14 02:58:27 -07:00
Ryan Ernst c36850f114 Add client flag for percolator module build 2016-07-14 02:41:58 -07:00
Ryan Ernst 20b038d976 Build: Add flag for plugins/modules which can be used in the transport client
This changes adds a flag which can be set in the esplugin closure in
build.gradle for plugins and modules which contain pieces that must be
published to maven, for use in the transport client. The jar/pom and
source/javadoc jars are moved to a new name that has the suffix
"-client".

I enabled this for the two modules that I know definitely need this;
there may be more. One open question is which groupId to use for the
generated pom.

closes 
2016-07-14 02:07:52 -07:00
Tal Levy ed768b101f show ignored errors in verbose simulate result ()
Closes .
2016-07-13 13:32:10 -07:00
Tal Levy 8fd01554bc update foreach processor to only support one applied processor. ()
Closes .
2016-07-13 13:13:00 -07:00
Nik Everett 2422b969c1 Migrate matrix_stats to NamedWriteable
This is the last consumer of the old style register method so I removed
the method.
2016-07-13 10:48:20 -04:00
Martijn van Groningen 2c3165d080 Removed deprecated 1.x script and template syntax
Closes 
2016-07-13 15:07:36 +02:00
Simon Willnauer c463083537 minor cleanups and an additional BogusPlugin for HttpSmokeTestCase 2016-07-12 17:55:05 +02:00
Simon Willnauer 9cb247287f consolidate security code in on place an allow test based on the jar dependency to opt out of netty internal property setting assertion 2016-07-12 17:41:21 +02:00
Simon Willnauer 4fb79707bd Fix remaining tests that either need access to the netty module or require explict configuration
Some tests still start http implicitly or miss configuring the transport clients correctly.
This commit fixes all remaining tests and adds a depdenceny to `transport-netty` from
`qa/smoke-test-http` and `modules/reindex` since they need an http server running on the nodes.

This also moves all required permissions for netty into it's module and out of core.
2016-07-12 16:29:57 +02:00
Ryan Ernst 93aebbef0f Merge branch 'master' into modularize_netty 2016-07-11 23:49:00 -07:00
Nik Everett 8263873783 Switch search extension from push to pull
Switches most search behavior extensions from push (`onModule(SearchModule)`)
to pull (`implements SearchPlugin`). This effort in general gives plugin
authors a much cleaner view of how to extend Elasticsearch and starts to
set up portions of Elasticsearch as "the plugin API". This commit in
particular does that for search-time behavior like customized suggesters,
highlighters, score functions, and significance heuristics.

It also switches most such customization to being done at search module
construction time which is much, much easier to reason about from a testing
perspective. It also helps significantly in the process of de-guice-ing
Elasticsearch's startup.

There are at least two major search time extensions that aren't covered in
this commit that will simply have to wait for the next commit on the topic
because this one has already grown large: custom aggregations and custom
queries. These will likely live in the same SearchPlugin interface as well.
2016-07-11 18:49:05 -04:00
Simon Willnauer 048e4416e7 Move netty transport and http into a module
This moves all netty code and it's dependency into a module.
2016-07-11 22:21:29 +02:00
Martijn van Groningen ff5527f037 percolator: Forbid the usage or `range` queries with a range based on the current time
If there are percolator queries containing `range` queries with ranges based on the current time then this can lead to incorrect results if the `percolate` query gets cached.  These ranges are changing each time the `percolate` query gets executed and if this query gets cached then the results will be based on how the range was at the time when the `percolate` query got cached.

The ExtractQueryTermsService has been renamed `QueryAnalyzer` and now only deals with analyzing the query (extracting terms and deciding if the entire query is a verified match) . The `PercolatorFieldMapper` is responsible for adding the right fields based on the analysis the `QueryAnalyzer` has performed, because this is highly dependent on the field mappings. Also the `PercolatorFieldMapper` is responsible for creating the percolate query.
2016-07-08 14:20:56 +02:00