Commit Graph

27002 Commits

Author SHA1 Message Date
Martijn van Groningen 211d50f7b8 [INGEST] Lazy load the geoip databases.
Load the geoip database the first time a pipeline gets created that has a geoip processor.
This saves memory (measured ~150MB for the city db) in cases when the plugin is installed, but not used.
2017-02-24 08:52:27 +01:00
Jim Ferenczi 57b5d1d29b disable BWC tests for the highlighters, need a new 5.x build to make it work 2017-02-24 08:50:39 +01:00
Jim Ferenczi 63bdd01eb7 Expose WordDelimiterGraphTokenFilter (#23327)
This change exposes the new Lucene graph based word delimiter token filter in the analysis filters.
Unlike the `word_delimiter` this token filter named `word_delimiter_graph` correctly handles multi terms expansion at query time.

Closes #23104
2017-02-24 00:53:38 +01:00
Tim Brooks 0e802961f1 Test that buildCredentials returns correct clazz (#23334)
This is fallout from #23297. That commit wrapped
`InstanceProfileCredentialsProvider` to ensure that the `getCredentials`
and `refresh` methods had privileged access. However, it looks like
there was a test ensuring that `buildCredentials` returned the correct
clazz type. This commit adjusts that test to check that the correct
wrapper is returned.
2017-02-23 17:33:15 -06:00
Shai Erera eeac6d27f2 Add BreakIteratorBoundaryScanner support for FVH (#23248)
This commit adds a boundary_scanner property to the search highlight
request so the user can specify different boundary scanners:

* `chars` (default,  current behavior)
* `word` Use a WordBreakIterator
* `sentence` Use a SentenceBreakIterator

This commit also adds "boundary_scanner_locale" to define which locale
should be used when scanning the text.
2017-02-23 23:32:22 +01:00
Ali Beyad 25a9a7ee3a Prioritize listing index-N blobs over index.latest in reading snapshots (#23333)
There are two ways to determine the latest index-N blob that contains
the truth of the contents of the repository: (1) list all index-N blobs
and figure out what the latest value of N is, and (2) read the
index.latest blob, which contains the latest value of N explicitely.
Note that the index.latest blob is not written atomically and can be
re-written, as opposed to the index-N blobs which are never re-written
(to create an updated index blob, index-{N+1} is written).

Previously, the latest index-N was determined by first trying to read
the index.latest blob and if that blob was missing (it was deleted
before being re-written and in between deleting it and re-writing it,
the system crashed), then all index-N blobs were listed to pick the
highest N value.

For non-read-only repositories, this could produce race conditions with
the file system.  In particular, it is possible that the index.latest
blob is being read in order to serve a read request (e.g. get snapshots)
and while doing so, an attempt is made to delete the index.latest blob
and re-write it in order to finalize a snapshot operation.  On some file
systems (e.g. Windows), it is forbidden to delete a file while it is
open for reading by another process/thread.

This commit changes the priority so that figuring out the latest index-N
blob is first done by listing all index-N blobs and determining the
latest N value.  If that values because the repository does not
support listing blobs (e.g. the URL repository), then the index.latest
blob is read.  This is safe because in read-only repositories that do
not support listing blobs, the index.latest blob is never deleted and
then re-written, so the aforementioned issue does not arise.
2017-02-23 15:44:12 -05:00
Ryan Ernst 0b4834f7da Test: Fix hdfs test fixture setup on windows
The test setup for hdfs is a little complicated for windows, needing to
check if the hdfs fixture can be run at all. This was unfortunately not
updated when the integ tests were reorganized into separate runner and
cluster setups.
2017-02-23 11:20:41 -08:00
Jason Tedor f85a7aed37 Keep the pipeline handler queue small initially
This commit sets the intial size of the pipeline handler queue small to
prevent waste if pipelined requests are never sent. Since the queue will
grow quickly if pipeline requests are indeed set, this should not be
problematic.

Relates #23335
2017-02-23 14:17:46 -05:00
sabi0 09b3c7f270 Do not create String instances in 'Strings' methods accepting StringBuilder (#22907) 2017-02-23 10:57:34 -08:00
Christoph Büscher 12b143e871 Tests: fix AwsS3ServiceImplTests 2017-02-23 19:06:35 +01:00
Christoph Büscher 8b1b152e91 Remove abstract InternalMetricsAggregation class (#23326)
This class doesn't seem to do much other than to group together
certain types of aggregations.
2017-02-23 18:03:40 +01:00
Tanguy Leroux 7e3c06c55d Add BulkRequest support to High Level Rest client (#23312)
This commit adds support for BulkRequest execution in the High Level Rest client.
2017-02-23 16:37:26 +01:00
Tim Brooks a4afc22df6 Wrap getCredentials() in a doPrivileged() block (#23297)
This commit fixes an issue that was missed in #22534.
`AWSCredentialsProvider.getCredentials()` appears to potentially open a
socket connect. This operation needed to be wrapped in `doPrivileged()`.

This should fix issue #23271.
2017-02-23 08:59:42 -06:00
Jason Tedor 3e69c38dbd Respect promises on pipelined responses
When pipelined responses are sent to the pipeline handler for writing,
they are not necessarily written immediately. They must be held in a
priority queue until all responses preceding the given response are
written. This means that when write is invoked on the handler, the
promise that is attached to the write invocation will not necessarily be
the promise associated with the responses that are written while the
queue is drained. To address this, the promise associated with a
pipelined response must be held with the response and then used when the
channel context is actually written to. This was introduced when
ensuring that the releasing promise is always chained through on write
calls lest the releasing promise never be invoked. This leads to many
failing test cases, so no new test cases are needed here.

Relates #23317
2017-02-23 09:32:43 -05:00
Jason Tedor e579629b16 Align REST specs for HEAD requests
Previous changes aligned HEAD requests to be consistent with GET
requests to the same endpoint. This commit aligns the REST spec for the
impacted endpoints.

Relates #23313
2017-02-23 08:55:13 -05:00
Simon Willnauer 2f3f9b9961 Remove unnecessary result sorting in SearchPhaseController (#23321)
In oder to use lucene's utilities to merge top docs the results
need to be passed in a dense array where the index corresponds to the shard index in
the result list. Yet, we were sorting results before merging them just to order them
in the incoming order again for the above mentioned reason. This change removes the
obsolet sort and prevents unnecessary materializing of results.
2017-02-23 13:48:54 +01:00
Simon Willnauer 771fd1f4ea Fix SamplerAggregatorTests to have stable and predictable docIds
Closes #23315
2017-02-23 08:08:38 +01:00
Ryan Ernst de8049fd2a Tests: Ensure multi node integ tests wait on first node
When a rest integ test has multiple nodes, each node is supposed to not
start configuring itself until the first node has been started, so that
the unicast host information can be written. However, this was never
explicitly setup to occur, and we were just very lucky with the current
gradle version and stability of the code always produced a task graph
that had node0 starting first. With the recent refactorings to integ
tests, the order has changed. This commit fixes the ordering by adding
an explicit dependency between the first node and the other nodes.
2017-02-22 20:54:58 -08:00
Jason Tedor 6ca90a61a6 Relocate a comment in HttpPipeliningHandler
This commit moves a comment in HttpPipeliningHandler as it makes more
sense for this comment to be where the field that it is explaining is
declared.
2017-02-22 20:51:18 -05:00
Jason Tedor 30f723d2b0 Add comments to HttpPipeliningHandler
This commit adds some comments explaining the design of
HttpPipeliningHandler.
2017-02-22 20:47:34 -05:00
Lee Hinman 6c9b89b882 [TEST] Fix incorrect test cluster name in cluster health doc tests 2017-02-22 17:18:11 -07:00
Ryan Ernst 74ecd34fd7 Build: Change location in zip of license and notice inclusion for plugins (#23316)
This commit moves the LICENSE.txt and NOTICE.txt files for each plugin
to be alongside the other plugin files, inside the elasticsearch subdir.
This ensures those files are installed alongside the plugin.
2017-02-22 16:13:50 -08:00
Ryan Ernst 18f57c05cf Script: Fix value of `ctx._now` to be current epoch time in milliseconds (#23175)
In update scripts, `ctx._now` uses the same milliseconds value used by the
rest of the system to calculate deltas. However, that time is not
actually epoch milliseconds, as it is derived from `System.nanoTime()`.
This change reworks the estimated time thread in ThreadPool which this
time is based on to make available both the relative time, as well as
absolute milliseconds (epoch) which may be used with calendar system. It
also renames the EstimatedTimeThread to a more apt CachedTimeThread.

closes #23169
2017-02-22 15:11:02 -08:00
Ryan Ernst 175bda64a0 Build: Rework integ test setup and shutdown to ensure stop runs when desired (#23304)
Gradle's finalizedBy on tasks only ensures one task runs after another,
but not immediately after. This is problematic for our integration tests
since it allows multiple project's integ test clusters to be
simultaneously. While this has not been a problem thus far (gradle 2.13
happened to keep the finalizedBy tasks close enough that no clusters
were running in parallel), with gradle 3.3 the task graph generation has
changed, and numerous clusters may be running simultaneously, causing
memory pressure, and thus generally slower tests, or even failure if the
system has a limited amount of memory (eg in a vagrant host).

This commit reworks how integ tests are configured. It adds an
`integTestCluster` extension to gradle which is equivalent to the current
`integTest.cluster` and moves the rest test runner task to
`integTestRunner`.  The `integTest` task is then just a dummy task,
which depends on the cluster runner task, as well as the cluster stop
task. This means running `integTest` in one project will both run the
rest tests, and shut down the cluster, before running `integTest` in
another project.
2017-02-22 12:43:15 -08:00
Lee Hinman 77d641216a Handle long overflow when adding paths' totals
From #23093, we fixed the issue where a filesystem can be so large that it
overflows and returns a negative number. However, there is another issue when
adding a path as a sub-path to another `FsInfo.Path` object, when adding the
totals the values can still overflow.

This adds the same safety to return `Long.MAX_VALUE` instead of the negative
number, as well as a test exercising the logic.
2017-02-22 13:04:34 -07:00
Yannick Welsch 0f88f21535 Don't set local node on cluster state used for node join validation (#23311)
When a node wants to join a cluster, it sends a join request to the master. The master then sends a join validation request to the node. This checks that the node can deserialize the current cluster state that exists on the master and that it can thus handle all the indices that are currently in the cluster (see #21830).

The current code can trip an assertion as it does not take the cluster state as is but sets itself as the local node on the cluster state. This can result in an inconsistent DiscoveryNodes object as the local node is not yet part of the cluster state and a node with same id but different address can still exist in the cluster state. Also another node with the same address but different id can exist in the cluster state if multiple nodes are run on the same machine and ports have been swapped after node crashes/restarts.
2017-02-22 20:27:27 +01:00
Jason Tedor 708d11f54a Ensure that releasing listener is called
When sending a response to a client, we attach a releasing listener to
the channel promise. If the client disappears before the response is
sent, the releasing listener was never notified. The reason the
listeners were never notified was due to a mistaken invocation of write
and flush on the channel which has two overrides: one that takes an
existing promise, and one that does not and instead creates a new
promise. When the client disappears, it is this latter promise that is
notified, which does not contain the releasing listener. This commit
addreses this issue by invoking the override that passes our channel
promise through.

Relates #23310
2017-02-22 13:54:17 -05:00
Lee Hinman 6f1ed8a3d1
[TEST] Add additional logging to IndicesStoreIntegrationIT.testIndexCleanup 2017-02-22 10:11:05 -07:00
Luca Cavanna 495b24655b Update indices settings api to support CBOR and SMILE format (#23309)
Also expand testing on the different ways to provide index settings and remove dead code around ability to provide settings as query string parameters

Closes #23242
2017-02-22 17:51:10 +01:00
javanna 594f00c582 Remove content type auto-detection from search templates
Now that search templates always get converted to json, we don't need to try and auto-detect their content-type, which anyways didn't work as expected before given that only json was really working.
2017-02-22 16:20:53 +01:00
javanna f2acf466aa Convert script/template objects to json format
Elasticsearch accepts multiple content-type formats, hence scripts can be stored/provided in json, yaml, cbor or smile. Yet the format that should be used internally is json. This is a problem mainly around search templates, as they only support json out of the four content-types, so instead of maintaining the content-type of the request we should rather convert the scripts/templates to json.

 Binary formats were not previously supported. If you stored a template in yaml format, you'd get back an error "No encoder found for MIME type [application/yaml]" when trying to execute it. With this commit the request content-type is independent from the template, which always gets converted to json internally. That is transparent to users and doesn't affect the content type of the response obtained when executing the template.
2017-02-22 16:20:53 +01:00
javanna 9391c6ffa9 Replace CustomMustacheFactory constant with same constant from Script (CONTENT_TYPE_OPTION) 2017-02-22 16:20:53 +01:00
Simon Willnauer 5c1924ad19 Remove BWC layer for number of reduce phases (#23303)
Both PRs below have been backported to 5.4 such that we can enable
BWC tests of this feature as well as remove version dependend serialization
for search request / responses.

Relates to #23288
Relates to #23253
2017-02-22 15:03:09 +01:00
Tanguy Leroux 09734e7469 Add UpdateRequest support to High Level Rest client (#23266)
This commit adds support for UpdateRequest to the High Level Rest client
2017-02-22 11:54:54 +01:00
Christopher Best eeaa0ccec2 Update getting-started.asciidoc (#23296) 2017-02-22 11:06:27 +01:00
Alexander Reelsen 6781c4320c Documentation: Consoleify cat shards/recovery API docs (#23116)
Relates #23001
2017-02-22 09:18:10 +01:00
mms-programming d31e41547a Handle BlobPath's trailing separator case (#23091) 2017-02-22 09:04:55 +01:00
Areek Zillur 148be11f26 Make document write requests immutable (#23038)
* Make document write requests immutable

Previously, write requests were mutated at the
transport level to update request version, version type
and sequence no before replication.
Now that all write requests go through the shard bulk
transport action, we can use the primary response stored
in item level bulk requests to pass the updated version,
seqence no. to replicas.

* incorporate feedback

* minor cleanup

* Add bwc test to ensure correct index version propagates to replica

* Fix bwc for propagating write operation versions

* Add assertion on replica request version type

* fix tests using internal version type for replica op

* Fix assertions to assert version type in replica and recovery

* add bwc tests for version checks in concurrent indexing

* incorporate feedback
2017-02-21 17:41:22 -05:00
Nik Everett 38d25a0369 Fix Painless's implementation of interfaces returning primitives (#23298)
Fixes Painless to properly implement scripts that return primitives
and void. Adds some simple tests that we emit sane opcodes and some
other tests that we implement primitives as expected.

Mostly this is just a fix following up from #22983 but there is one
thing I did really worth talking about, I think. So, before this script
Painless scripts could only ever return Object and they did would always
return null for paths that didn't return any values. Now that they
can return primitives the question is "what should Painless return
from paths that don't return any values?" And I answered that with
"whatever the JLS default value is". So 0/0L/0f/0d/false.
2017-02-21 17:10:55 -05:00
Simon Willnauer ca38e88148 Remote assertion that relies on all shards being successful
The assertion that if there are buffered aggs at least one incremental
reduce phase should have happened doens't hold if there are shard failure.
This commit removes this assertion.

Relates to #23288
2017-02-21 22:41:49 +01:00
Martijn van Groningen 81d53470e7 percolator: add support for term extraction for MultiPhraseQuery 2017-02-21 21:10:55 +01:00
Nik Everett 9105672969 Allow painless to implement more interfaces (#22983)
Generalizes three previously hard coded things in painless into
generic concepts:

1. The "main method" is no longer hardcoded to:
```
public abstract Object execute(Map<String, Object> params,
        Scorer scorer, LeafDocLookup doc, Object value);
```
Instead Painless's compiler takes an interface and implements it. It looks like:
```
public interface SomeScript {
    // Argument names we expose to Painless scripts
    String[] ARGUMENTS = new String[] {"a", "b"};
    // Method implemented by Painless script. Must be named execute but can have any parameters or return any value.
    Object execute(String a, int b);
    // Is the "a" argument used by the script?
    boolean uses$a();
}
SomeScript script = scriptEngine.compile(SomeScript.class, null, "the_script_here", emptyMap());
Object result = script.execute("a", 1);
```

`PainlessScriptEngine` now compiles all scripts to the new
`GenericElasticsearchScript` interface by default for compatibility
with the rest of Elasticsearch until it is able to use this new
ability.

2. `_score` and `ctx` are no longer hardcoded to be extracted from
`#score` and `params` respectively. Instead Painless's default
implementation of Elasticsearch scripts uses the `uses$_score` and
`uses$ctx` methods to determine if it is used and gives them
dummy values if they are not used.

3. Throwing the `ScriptException` is now handled by the Painless
script itself. That way Painless doesn't have to leak the metadata
that is required to build the fancy stack trace. And all painless scripts
get the fancy stack trace.
2017-02-21 14:08:57 -05:00
Jack Conradson fac2d954e3 Fix certain bad casts in Painless due to boxing/unboxing. (#23282) 2017-02-21 10:23:27 -08:00
Nik Everett 7475175957 Adds unit test for sampler aggregation (#23243)
* Adds unit test for sampler aggregation

Relates to #22278
2017-02-21 12:51:47 -05:00
Jim Ferenczi 0ff6356b7e Revert "Never reduce the same agg twice"
This change reverts 5e4ba4a60e
Incremental reduction of aggs should also work with a single aggregation now that InternalTopHits.equals
 is fixed.
2017-02-21 18:48:28 +01:00
Simon Willnauer ce625ebdcc Expose `batched_reduce_size` via `_search` (#23288)
In #23253 we added an the ability to incrementally reduce search results.
This change exposes the parameter to control the batch since and therefore
the memory consumption of a large search request.
2017-02-21 18:36:59 +01:00
Jim Ferenczi 1ba9770037 Fix comparaison of double in InternalTopHits
InternalTopHits uses "==" to compare hit scores and fails when score is NaN.
This commit changes the comparaison to always use Double.compare.

Relates #23253
2017-02-21 18:18:44 +01:00
Simon Willnauer 5e4ba4a60e Never reduce the same agg twice
Some randomization caused reduction of the same agg multiple times
which causes issues on some aggregations.

Relates to #23253
2017-02-21 17:55:44 +01:00
Simon Willnauer 489f38918d Fix incremental reduce randomization in base tests cases
We can and should randomly reduce down to a single result before
we passing the aggs to the final reduce. This commit changes the logic
to do that and ensures we don't trip the assertions the previous imple tripped.

Relates to #23253
2017-02-21 17:13:46 +01:00
Nik Everett 74c33823ab Comment 2017-02-21 10:43:29 -05:00